Author: rachelt1

This week, I mainly focused on collecting more real-time data to test our glove’s performance under different conditions as well as creating the slides in preparation for our final presentation. I tested the glove’s performance by changing the number of consecutive predictions that the glove must see before outputting the signal as well as seeing our glove’s performance under different signing rates (speed of user). In doing so, I realized that the amount of consecutive predictions we require and how much buffer we allow between signs are both going to have to be dependent on our target user– if our target user is a novice learner, we would probably allow for more buffer time while if our target user is someone who is well-versed in the ASL letters, this buffer time should be reduced so that they can sign smoothly. I also created other plots (such as points of two entirely overlapping signs) to show in our presentation why our glove performs more poorly on some of the letters.

In collecting data for these graphs, I also found some errors in the serial communication code that would occasionally cause the script to hang, so I also fixed that this past week. This coming week, I will work with the rest of my team to complete the final report as well as create a video for our final demo. The glove also needs to be fixed and the sensors secured, so I will also help with the job of securing the sensors so that the data we get from them remain consistent (which was another problem we found throughout the semester). After securing the glove, I will re-collect data from several people, so that we can train our final model. Since everything we have left is relatively outlined and what we expected to be doing this upcoming week, I would say we are right on track!

I spent most of this week trying to figure out what was going wrong with the audio output for our system. When I made a gesture, it would consistently say “A” repeatedly and nothing else would be outputted until the stream of “A”s was finished outputting. I thought that this was because the glove was continually making predictions and the audio outputs were overlapping each other. However, after trying lots of different things, I figured out that the audio files themselves were corrupted. I’m not really sure what happened, but I generated the audio files for each letter again and they’re fine now, so the audio integration is complete. Currently, our system is able to make approximately 20 predictions per second, but will only output the audio if it gets 7 of the same predictions in a row. This number was chosen to achieve our specification 2 gestures per second while leaving some room for transitions.

I also modified the script to utilize smaller models for the letters R, U, V, and W as well as M, N, S, T, and E. When our main model outputs any of those letters, the data gets put through another model for further classification. These smaller models only select from the group of letters that are easily confused. We found that this gives ever so slightly better results and are still in the process of doing data analysis to determine which data values to train on and/or if these letters are even distinguishable based on our sensor data.

I believe we are still on track since the functionality of our glove is complete. However, we did find that one of the flex sensors was not outputting useful data (it consistently outputs values around 256 degrees no matter the amount of bend), so we will need to replace that sensor and recollect data. After that, though, all that is left is to prepare our final presentation, make the final video, and add + make changes to our design review report for the final report. After Thanksgiving, I will collect data with our fixed glove and do all the final report/video/presentation things!

This week, I worked on some of the things we had identified to help improve our glove’s performance. I sewed the flex sensors down (so that it is connect to the glove at more than jus a few select intervals). Before, the glove would bend in between connections when our fingers were too straight and the degree of that bend was also contributing to inconsistencies between our training data and our real-time data. Sewing down the glove helped with that a lot and now we are seeing that the glove behaves as expected. The glove is now confused on letters that we know are similar but not on ones that we know should be different. For example, u and w are incredibly distinct letters in the ASL alphabet, but where commonly mis-classified in our previous iteration, but u and r were not even though they are significantly more similar. I believe this is due to the bending issue discussed earlier– now our glove gets confused on r and u but not on u and w. We also discussed using a window of time to see if we could capture the letters that involve motion more accurately. I adjusted the python and arduino scripts to do that as well. We found a few benefits from a window of time after collecting data and testing with a window size of 3 consecutive time instances: the window of time is better at capturing the gestures requiring movement and is also better at capturing the transitions between letters (it knows that those are not to be classified for the most part).

We also met with an ASL user as well as an interpreter to discuss possible improvements to our design and the ethics of our project. One important point that we discussed in both meetings is that we should re-word the scope and use case of our project and add what this can be extended to as “future work.”

I would say we are still on track. We discussed our schedule as a group and determined that our initial schedule with most of our data collection earlier on and iterating later didn’t make sense because iterating on our glove would require getting new data. We have modified our schedule so that we are still in the iterating phase and the bulk of the data collection will happen after we settle on what we think is the best implementation we can achieve in the time we have remaining. This week, I will modify the python and arduino scripts to output audio. I experimented with this a bit this week and found that there are weird syncing issues if only the python script is modified– I believe I need to send the arduino a signal to wait on sending data while the audio is outputting. We will hopefully also decide on our final design and gather data from various people.

This week during lab, our group did a lot of work together. We collected data from each of us and found a similar accuracy between the new data and the data that I had collect earlier on just myself, so we can have confidence that with more variety of user data, we will be able to generalize our glove. I mostly spent my time outside of lab writing the various scripts we will use for collecting data, adding to existing data, and using the glove and have real-time classifications. Currently, our system can make approximately 8.947 classifications per second.

There are certain letters that consistently classify poorly– letters that require movement (J, Z) and letters that are similar to others (N, T, M, X, etc.). It makes sense for similar letters to be classified incorrectly more often because they would give similar sensor data to other letters. For J and Z, we hypothesized that the poor classification is because our data collection procedure does not necessarily collect data from the same point in each movement (we are using a button press to tell the system to collect data). To investigate, I plotted some graphs of the letter J (a gesture requiring movement) and P (a gesture that is consistently correctly classified). Interestingly, the issue seems to be that certain measurements from the gyroscope and accelerometer were too consistent, so when we were using the glove real-time, the differing measurements were throwing the classification off. Here is an example of one of those measurements (accelerometer x):

I also collected new data experimenting with the IMU on the back of the palm rather than on the wrist. The hope is that because the back of the palm will have more movement, our classification will pick up the movement from the letters J and Z better.

I would say we are on schedule. This week, I will work to gather data from various people (ideally with varying hand sizes) and search for a better text to speech package since pyttsx3 has a very noticeable delay, which would not allow a very smooth user experience. I will also work with my team to figure out a solution to determine when a sign should be classified. Currently, our plans to utilize a classification probability threshold as well as a “transition” class would not work to account for pauses in signing (perhaps to think of the next letter to sign).

I did not get as much work done as I had planned this week due to a heavy workload from other classes. I had originally planned to do some data analysis on our real and generated data as well as come up with/test multiple methods of detecting what is a real gesture.

I spent most of this week doing some analysis on the data we had collect compared to the fake data we had generated to try to understand why our real data performed so much better than our fake data. I did this by plotting several combinations of three axes (x, y and size) for the letter L. I chose to do the letter L because the sign for this letter a good variety of finger angles. To begin, we had more fake data than real data to train our models on, which is interesting considering our real data trained a model with higher accuracy. Based on my plots, I believe the reason for better results with the real data is that we have more consistent consistent correlation between the finger angles and the IMU data. For example, the plot I created comparing Middle Finger vs. AccelerometerZ vs. GyroscopeY showed a very scattered plot that does not imply any correlation for the fake data. The real data, on the other hand has a much more clear pattern (eg. bigger AccelerometerZ values correlates to a larger GyroscopeY value). Below are those two comparisons. This makes sense because when generating the fake data, we were unsure how to model the IMU data since we weren’t able to find good documentation on the values they would output based on certain movements or orientations.

I also collected some extra data, since I am in possession of the glove, for a “space” character as well as some random gestures so that the system does not try to identify letters while the user is transitioning between letters.

I would say we are on schedule since we had taken much less time than anticipated for the glove construction. The next steps are to integrate the glove with the software system so that it can do real-time interpretation of the data collected from the glove, so I will be working on this with Stephanie this upcoming week.

This week, I presented our Design Review to the class and started writing the Design Review written report. I also got the glove that Sophia built to communicate with the python program and made sure that the data is read in correctly. Throughout the process, I found that some of the connections were slightly loose, so for letters that require movement (such as the letter “j”), sometimes the connection would become undone giving -63.0 as the angle read in from the flex sensor. This is something we will need to improve upon. We could also make the sensors fit more tightly to the glove so that it is more sensitive to bending. I also collected data for Stephanie to test the ML models on. However, since we were not able to find time to meet this week, this data is only from me making the gestures while varying my poses slightly, so we cannot be sure that this ML model will be generalizable just yet– this would require us to get data from more people.

Since putting together the glove and implementing the serial communication took less time than expected, we are actually ahead of schedule. This is good because there are still some things we need to fix with the current system (eg. wire connections and recognition of gesture beginning and end), which may take longer than the time we had left for software+hardware integration and adjustments.

For collecting this preliminary set of data, I used a key press to indicate that I am done making the gesture, but we do not want this to be part of the user experience in the final product, so next week I will work on figuring out how the system will recognize that a new gesture has arrived. I will also work on developing the data normalization algorithm.

This week, I mainly focused on putting together the design review slides, making sure that confusions from our proposal presentation will be clarified. Stephanie worked on doing some tests with generated data to identify what model might work best for our use case. As expected, some models performed better than others, so I did research on each model so that we can explain during the design review why some models perform better. I also figured the serial communication between the arduino and our python program, so once our glove is built, we can jump right into training and tuning our ML model.

I would say we are behind schedule because our parts did not come in until Friday. We can only hope that our generated data mimics the real data well enough that our predictions for which model will perform the best is still accurate. If not, that would set us back even more. However, since we have all of our parts (aside from the PCB, which isn’t a critical component of this project), I believe we can make up for lost time this week.

Next week, I will work on figuring how we are going to get the model to determine when a gesture starts and ends by doing more research on past projects. Some methods that could work are having a “nothing” pose that separates each sign, having a button to indicate the start of a gesture (not ideal), or waiting some amount of time after receiving data. I am also going to give our design review presentation.