Posted on

Team Status Report for 11/20

This week we implemented the audio portion of our project. Now when a sign is made, the letter will be spoken by the computer. And the audio output is at the rate that we set in the requirements at the beginning of the semester. We also implemented the suggestion we got during interim demos about creating another model to differentiate between letters that are commonly mistaken for others (R, U, V, W and M, N, S, T, E). However, this did not noticeably improve the functionality. We will continue analyzing the data from those easily confused letters to see if we can improve the accuracy. Adding contact sensors may be the best solution.

The Arduino Nano BLE arrived and we are working on integrating Bluetooth into our system so the glove can work wirelessly. There have been some issues uploading scripts to Arduino Nano BLE and will hopefully be resolved next week.

The reading for the flex sensor on the ring finger of the glove also stopped varying (it gives roughly 256 degrees no matter how much you bend it), so we replaced the flex sensor. There is also some calibration we should do in the Arduino script to improve the conversion from voltage to degrees. Now that we have repaired the glove, we will need to recollect data.

We are on schedule since the audio integration is now complete. In the final weeks, we will continue refining the system by changing the hyper-parameters of our model.  We will also create plots showing performance to add to our final presentation, paper, and video.

Posted on

Stephanie’s Status Report for 11/20

This week I worked on integrating the letter-specific models with our old classifier. The original classifier is responsible for identifying the input data as one of the 26 letters. If the predicted letter is in the list of similar letters that the original classifier easily gets confused on, notably R, U, V, W and M, N, S, T, E, the data will then be passed to one of the smaller models for further classification. This gave slightly better results and was able to make more correct classification.

I performed further data analysis to see if there’s any other way I can improve the model. I did find some data contained invalid values (negative values from flex sensors) that my old outlier removal function did not pick up, hence I was able to refine that to get better data. The sensor values for U and V are rather similar so it could be hard to have a classifier to distinguish between these two. R’s range of values is quite different from the rest of the letters for the middle finger, hence to improve future models for this specific group of letters, I may only train with data of important features, e.g., the flex sensor values.

During this, I also found the ring finger’s flex sensor value for W (finger is straight) is quite similar to that of R,U, and V (finger is bent). Upon further testing with the glove, I found that the flex sensor on the ring finger is not working as intended and is only giving about the same value no matter the bending angle, so we are looking to replace that sensor.

I believe we are still on schedule. In the next week, I’ll be collecting new data after the flex sensor has been fixed.  I’ll also be doing some real time testing and experimenting with new ideas of the smaller model (e.g., training on only important features, changing hyperparameter to produce models with less accuracy on the collected data but more generalizability). After that, we’ll be working on our final presentation and report.

Posted on

Sophia’s Status Report for 11/20

This week I repaired the glove and worked on connecting with the Arduino Nano BLE over Bluetooth.

The flex sensor on the ring finger of the glove stopped reading values. There was no variation in reading when the sensor was bent, so I replaced that flex sensor. We also noticed there was some calibration in the Arduino script we could have completed to make the transformation from voltage to angle of bending more accurate. Since we had to repair the glove, we will do the calibration before collecting the next set of data.

The Arduino Nano 33 Sense with BLE arrived. I tried for a considerable amount of time to upload a basic sketch to it, but for some reason, it won’t upload. I’ll continue trying to work with the new Nano next week. I also looked deeper into how to connect the Arduino Nano 33 BLE to the PC over Bluetooth. It turns out BLE (Bluetooth Low Energy) is not compatible with classic Bluetooth and is most easy to interface with over a mobile device. There are smartphone apps with the sole purpose of reading from Arduino Nano BLE’s or other BLE devices. It is a bit more difficult to connect and read its serial output from a PC, but not impossible. One of the drawbacks of connecting this way is that the transmission rate is slower. Once we get the transmission working we can test to see just how much slower the data streams are and if that will affect whether we reach our requirements.

Posted on

Rachel’s Status Report for 11/20

I spent most of this week trying to figure out what was going wrong with the audio output for our system. When I made a gesture, it would consistently say “A” repeatedly and nothing else would be outputted until the stream of “A”s was finished outputting. I thought that this was because the glove was continually making predictions and the audio outputs were overlapping each other. However, after trying lots of different things, I figured out that the audio files themselves were corrupted. I’m not really sure what happened, but I generated the audio files for each letter again and they’re fine now, so the audio integration is complete. Currently, our system is able to make approximately 20 predictions per second, but will only output the audio if it gets 7 of the same predictions in a row. This number was chosen to achieve our specification 2 gestures per second while leaving some room for transitions.

I also modified the script to utilize smaller models for the letters R, U, V, and W as well as M, N, S, T, and E. When our main model outputs any of those letters, the data gets put through another model for further classification. These smaller models only select from the group of letters that are easily confused. We found that this gives ever so slightly better results and are still in the process of doing data analysis to determine which data values to train on and/or if these letters are even distinguishable based on our sensor data.

I believe we are still on track since the functionality of our glove is complete. However, we did find that one of the flex sensors was not outputting useful data (it consistently outputs values around 256 degrees no matter the amount of bend), so we will need to replace that sensor and recollect data. After that, though, all that is left is to prepare our final presentation, make the final video, and add + make changes to our design review report for the final report. After Thanksgiving, I will collect data with our fixed glove and do all the final report/video/presentation things!

Posted on

Team’s Status Report for 11/13

This week we had our interim demo presentations. For this presentation, we showed our glove classifying in real-time the ASL alphabet and speaking out the letter. There were some issues with latency as the speech output of the letter would continue to output even when a different letter was being signed. There were also some issues differentiating between similar signs. We received a lot of great feedback for fixing these problems. To fix the latency we will try to improve the handshaking protocol. To improve classification, we can build a model around the letters that are easily confused and use this more specific model anytime the general model classifies a letter that falls into these groups of similar signs.

The professors also recommended we improve our product to work wirelessly so that it is more usable/portable. As per their feedback, we have ordered a Bluetooth-enabled Arduino Nano to replace our current Nano.

Additionally, we are working on collecting data from a wide variety of people. Each team member is taking turns bringing the glove home to collect data off of roommates. We will also bring our glove to Daniel Seita, an ASL user at CMU to train our model.

Finally, we will begin working on the final presentation and report.

Posted on

Sophia’s Status Report for 11/13

This week we had our interim demos. From the feedback, we have decided to try and make our glove wireless with our remaining time to make it more usable. I ordered a new Arduino Nano with BLE and also some batteries as power sources. Next week, I will need to take the glove and modify it so that the battery can be secured on the glove and correctly connect to the Arduino Nano. I will also need to work on pairing the Arduino Nano and the computer running the script. I also plan to add touch sensors to help differentiate between letters e, m, n, s, and t. This could require us to collect a new set of data, however, after discussing with the others, it seems we could also add the data from the touch sensors as conditionals into the script.

We are on schedule. We adjusted our old schedule to reflect our new plans. The changes are basically that we are extending data collection and making modifications based on feedback.

Posted on

Stephanie’s Status Report for 11/13

This week, I mostly worked on data analysis with real time testing, data collection, and developing models specific for certain letters. Rachel and I worked together on writing the script for recording accuracy for each letter in real time and I used the results we got to make a confusion matrix to pinpoint which letters tend to get more misclassified than others. I have also collected data from my friends.

After the interim demo on Thursday with class TAs,  we got some helpful feedback. One of them was to add models for classifying similar letters. Right now, we are using one model to classify all 26 letters. The poses for this group of letters: r,u,v,w, are similar to each other, and the same goes for m,n,s,t,e. Hence we have decided to make two models specifically for this two group of letters so that our glove can distinguish between the similar letters better. I have already developed two models based on the data collected so far, however having more data would be even better to help refine the models, so I’ll be updating the models as we get more data.

Next week, I’ll be working on integrating these news models with our current classification code next week, and analyzing this new method’s performance.  As a team, we will also begin to work on our final presentation slides. I would say we are on track with our new schedule. All we have left is to implement our new classification method and get audio to work without lag.

Posted on

Rachel’s Status Report for 11/13

This week, I mostly worked on data collection and adapting the scripts to incorporate audio with our script. I found another modules (gtts and playsound) that are able to create and play audio files relatively quickly (without noticeable delay from a user perspective), so we will be using that instead of pyttsx3, which had a really long delay. I added in some handshaking signals between the arduino and python programs, which slowed down the prediction and output rate to be about 0.27 gestures per second, which is significantly below our target of 2 gestures per second. In changing the arduino script back, I noticed that I was sending new line characters, which was being ignored by the script but that line that was sent could have been better used by sending actual data. After fixing that, our glove can make about 17 predictions per second. I am currently working on incorporating the audio properly, so that there isn’t a lag between streamed in data and the outputted audio– for reasons unknown to me at the moment, the handshaking signals I was passing around before are not working.  Since the changes we plan to make in the next couple of weeks do not involve changes in what the data looks like, I also had my house mates collect data for us to train on.

This week, I plan on fully integrating the audio and getting more people to collect data. I will also begin to work on the final presentation slides as well as the final report. I would say we are on track since all that remains is collecting data and training our final model (we are near the end!). We also have ordered a bluetooth arduino nano, which we will have to switch out for our current arduino– this will also require some changes in the scripts that we have been using, but it shouldn’t become a blocker for us.

Posted on

Sophia’s Status Report 11/6

This week I arranged for meetings with Daniel, an ASL user at CMU, and with Danielle, an ASL interpreter. From Daniel, we learned that for fingerspelling, double letters are usually signed with a single fluid movement rather than signing the letters twice. This is something we might want to keep in mind when trying to parse together the words. Next week we plan to meet with him in person and collect data from him to train our model.

Danielle had a lot of feedback poking holes in the use case we defined for our product. She pointed out that the ASL alphabet is used just for proper nouns, and so you can’t really communicate in ASL with just the alphabet.  Furthermore, ASL has five different aspects: palm orientation, placement, location, movement, non-manual signals (body language and facial expression, tone of voice). The glove can’t really pick up the non-manual signals so at best, the glove can translate, but not interpret. She explained to us that a lot of grammar is actually based on non-manual signals in ASL. She also pointed out that ASL users don’t usually like to have things obstructing their hand movement. She shared that she doesn’t like to sign, or feels she signs awkwardly when she wears a ring on her finger. With this input, we should look into a way to measure the resistance the glove gives.

We are on schedule, but we had to readjust the schedule since adjustments to our product required us to recollect training data.

Posted on

Team’s Status Report for 11/6

This week, we implemented our new data collection process (collecting a window size of three snapshots) and collected data from each of our team members. We also integrated the real time testing with the glove so the software system can record the accuracies of each letter tested. We found that letters with movements (J and Z) perform much better and the classifier recognize it more frequently. We also found that the classifier recognized dissimilar letters better but it does get confused on similar letters (e.g., M and N). In order for the classifier to discern between them better, we will need to get more data. Overall, using a window of three for our data points have improved accuracy compared to our older model. It’s also recognizing about 3-4 gestures per second with the new data collection process. This rate is more suitable for actual usage since the average rate of signing is about 2 signs per second.

We also met up with an ASL user and an interpreter. Both gave useful feedback pertaining ethics, use cases, and capability of the glove for our project.

In terms of schedule, we are on the right track. Next week we will be meeting with the ASL users that we have contacted to collect data from them. Ideally, we will be getting people of different hand sizes for data collection. We will also start on refining our classifier to better recognize similar letters and implementing the audio output.