This week during lab, our group did a lot of work together. We collected data from each of us and found a similar accuracy between the new data and the data that I had collect earlier on just myself, so we can have confidence that with more variety of user data, we will be able to generalize our glove. I mostly spent my time outside of lab writing the various scripts we will use for collecting data, adding to existing data, and using the glove and have real-time classifications. Currently, our system can make approximately 8.947 classifications per second.
There are certain letters that consistently classify poorly– letters that require movement (J, Z) and letters that are similar to others (N, T, M, X, etc.). It makes sense for similar letters to be classified incorrectly more often because they would give similar sensor data to other letters. For J and Z, we hypothesized that the poor classification is because our data collection procedure does not necessarily collect data from the same point in each movement (we are using a button press to tell the system to collect data). To investigate, I plotted some graphs of the letter J (a gesture requiring movement) and P (a gesture that is consistently correctly classified). Interestingly, the issue seems to be that certain measurements from the gyroscope and accelerometer were too consistent, so when we were using the glove real-time, the differing measurements were throwing the classification off. Here is an example of one of those measurements (accelerometer x):
I also collected new data experimenting with the IMU on the back of the palm rather than on the wrist. The hope is that because the back of the palm will have more movement, our classification will pick up the movement from the letters J and Z better.
I would say we are on schedule. This week, I will work to gather data from various people (ideally with varying hand sizes) and search for a better text to speech package since pyttsx3 has a very noticeable delay, which would not allow a very smooth user experience. I will also work with my team to figure out a solution to determine when a sign should be classified. Currently, our plans to utilize a classification probability threshold as well as a “transition” class would not work to account for pauses in signing (perhaps to think of the next letter to sign).