Posted on

Team’s Status Report for 10/30

This week, we collected data from each of our team members and integrated the glove with the software system so that the classification can be done in real-time. We found that some letters that we expect to have high accuracy performed poorly in real time. Namely, the letters with movement (J and Z) did not do well. We also found that different letters performed poorly for each of our group members.

After our meeting with Byron and Funmbi, we had a bunch of things to try out. To see if our issue was with the data we had collected or perhaps with the placement of the IMU, we did some data analysis on our existing data as well as moved the IMU to the back of the palm from the wrist. We found that the the gyroscope and accelerometer data for the letters with movement are surprisingly not variable– this means that when we were testing real time, the incoming data was likely different from the training data and thus resulted in poor classification. The data from the IMU on the back of the hand has a 98% accuracy from just the data collected from Rachel; we will be testing it in real time this coming week.

We also found that our system currently can classify about 8.947 gestures per second, but this number will change when we incorporate the audio output. This rate is also too high for actual usage since people cannot sign that fast.

We are also in contact with a couple of ASL users who the office of disabilities connected us with.

We are still on schedule. This week we will work on parsing the letters (not continually classify them). We are also going to take data from a variety of people with different hand sizes, ideally. We will also experiment with capturing data over a time interval to see if that yields better results. We will also be improving the construction of the glove by sewing down the flex sensors more (so that they are more fitting to the glove) and doing a deeper dive into our data and models to understand why they perform the way they do. We will also hopefully be able to meet with the ASL users we are in contact with.

Posted on

Rachel’s Status Report for 10/30

This week during lab, our group did a lot of work together. We collected data from each of us and found a similar accuracy between the new data and the data that I had collect earlier on just myself, so we can have confidence that with more variety of user data, we will be able to generalize our glove. I mostly spent my time outside of lab writing the various scripts we will use for collecting data, adding to existing data, and using the glove and have real-time classifications. Currently, our system can make approximately 8.947 classifications per second.

There are certain letters that consistently classify poorly– letters that require movement (J, Z) and letters that are similar to others (N, T, M, X, etc.). It makes sense for similar letters to be classified incorrectly more often because they would give similar sensor data to other letters. For J and Z, we hypothesized that the poor classification is because our data collection procedure does not necessarily collect data from the same point in each movement (we are using a button press to tell the system to collect data). To investigate, I plotted some graphs of the letter J (a gesture requiring movement) and P (a gesture that is consistently correctly classified). Interestingly, the issue seems to be that certain measurements from the gyroscope and accelerometer were too consistent, so when we were using the glove real-time, the differing measurements were throwing the classification off. Here is an example of one of those measurements (accelerometer x):

I also collected new data experimenting with the IMU on the back of the palm rather than on the wrist. The hope is that because the back of the palm will have more movement, our classification will pick up the movement from the letters J and Z better.

I would say we are on schedule. This week, I will work to gather data from various people (ideally with varying hand sizes) and search for a better text to speech package since pyttsx3 has a very noticeable delay, which would not allow a very smooth user experience. I will also work with my team to figure out a solution to determine when a sign should be classified. Currently, our plans to utilize a classification probability threshold as well as a “transition” class would not work to account for pauses in signing (perhaps to think of the next letter to sign).

Posted on

Stephanie’s Status Report for 10/30

In this week, I worked on researching on why the neural network’s accuracy increased with the added labels of “space” and “transitions”. To compare the accuracies of different data sets, I compiled three different sets of data: one with all the labels, one with all the letters and the space labels, and one with only letter labels. To see if the differences in data did affect the overall accuracy, I performed hyperparameter tuning when training each data set’s model to find the best performing accuracies. The results showed that best tuned models were able to reach 93%+ accuracy for all the data sets.  (The accuracy also varies by runs due to random initialization in the model)

In conclusion, the added data labels didn’t actually improve the accuracy of the model. Since neural networks can have various amount of hidden layers and activation function, it’s likely that we just did not find the optimal parameters of the model for a particular dataset.

Furthermore, we found that our first set of data were mostly taken with the hand facing one direction. This may significantly impair our model performance. This was further proved when I tested the model trained on first set of data. On our two newest sets of data, the model only showed 4% and 21% accuracy. Hence it’s likely that we will not be using this set of data in the future.

We also adjusted the IMU on the glove this week and just got a new set of data. Preliminary tests on this dataset showed high accuracy (98%), but model trained on second data set only had a 50% accuracy on this set, which signifies this is a large change to the overall data. We’ll be doing more real time testing next week. I’ll also be working on making some data comparison with our older data to see why the old model did not perform as well in real time testing.

I would say we are a little behind schedule in terms of gathering data as we are still contacting people who use ASL. But we’re doing well with data training/testing and model analyzing.

 

 

Posted on

Sophia’s Status Report for 10/30

This week I reached out to the office of disabilities at CMU to contact actual ASL users. There is a post-doc at CMU which uses ASL who has agreed to talk with us and provide insight into our project. There is also an ASL interpreter interested in helping out with our project.

In addition to reaching out to ASL users, I wrote up a data collection procedure that we can give to people to follow to collect data more autonomously.

After our discussion with Byron, we identified the next steps we need to take to improve our prototype. Right now we are going to focus on improving the classification accuracy. I moved the IMU to the back of the glove in a makeshift way. There are a lot of wires, so it’s quite messy, but it will allow us to see if the change in IMU position helps differentiate between signs.

Open photo

This upcoming week I need to restitch some of the flex sensors because sometimes when a pose is made, the flex sensors don’t always bend in a consistent way due to the limited stitches. The embroidery thread has finally arrived so I will stitch over each of the flex sensor in this upcoming week.

I think we are on schedule, but I’m not confident we will be able to  parse letters into words. However, this was not a functionality we originally defined in our requirements.

Posted on

Sophia’s Status Report for 10/23

This week I focused on maintenance for the glove. Some of the connections between the flex sensors and the board hosting the Arduino Nano came loose due to poor crimping job so I redid the connection. I also ordered embroidery thread to better secure the flex sensors. The stitches keeping the flex sensors in place currently are coming loose.

I also finalized the gerber, BOM and PLC files required to order the connective PCB. I want to order the PCB from jlcpcb, but I want to hold off on ordering a little longe. The connective PCB design that I plan to order is the same size as the perf board, so there is not much added advantage of having this PCB. I did some research on possibly making the PCB smaller and in the next week I plan to make a second design and then order the connective and new design in one batch to save on shipping costs.

In addition to creating a new PCB design, I plan to reach out to the office of disabilities to contact some fluent ASL users and interpreters who we can test our device on.

I think we are on schedule, but I’m not sure how smoothly replacing the PCB will be if we decide to use it.

 

Posted on

Team’s Status Report for 10/23

This week, we focused on maintaining the glove as well as moving toward real-time usage of the glove. Some of the connections on our first iteration of the glove came loose as we made gestures that required more movement, so we went back and secured those connections.

We did some analysis on the fake data that we had generated and the real data we collected from the gloves to gain some clarity as to why our models with the real data outperformed out models with the fake data despite having a smaller dataset. The conclusion is that there is a much stronger correlation between the fingers and IMU data in the real data, which makes sense since we were having a hard time modeling the IMU data when we were attempting to mimic the data.

We also added a “random” and “space” category to the classes that our model can output. The “space” class meant to help us eventually allow users to spell words and have those full words outputted as audio (rather than just the individual letters). The “random” class was added to help suppress the audio output as the user transitions between signs. These additions will allow us to move onto using our glove real-time, which is our next step.

We are on schedule and plan on putting the glove together with our software system to use in real-time this week. We are also going to collect data from people with several hand sizes to use to see how well we can get our glove to generalize.

Posted on

Stephanie’s Status Report for 10/23

There has been a change of plan from what I wanted to do from last two weeks. My original plan was to collect more data for model training, however, our glove needed more fabrication work to ensure the sensors are well attached. We also plan to enhance the glove’s data collection process. Our first set of data (done by Rachel) had to be collected by pressing buttons to determine the time duration of when the data will be read in. So for this week, we are trying to integrate real-time data collection and interpretation. More data is collected for hand gestures in-between each letter gesture (these gestures are mostly random since they are just transitions from one letter to another). A ‘space’ letter is added in case the models can not categorize the ‘random’ gestures well. First round of testing shows promising result. With the random forest model, which has been the model with highest accuracy so far, the accuracy for recognizing these two new labels are quite high.

I also found that with these two new labels added, the neural network accuracy have increased by 10%. This is an interesting finding and I plan to look further into why this is the case. Before this, I have done much tweaking to the model and validating with different hyperparameters, but the network’s accuracy seemed to be capped around 80%, however, with this new dataset, its accuracy reached around 88% and that is something I would like to find out why to help our future classification works.

I would say we are still on schedule since we planned a lot of time for the software implementation. As for next week, I’ll be looking more into the models and working on make our data collection work more smoothly, and if possible, starting to collect data from others.

Posted on

Rachel’s Status Report for 10/23

I did not get as much work done as I had planned this week due to a heavy workload from other classes. I had originally planned to do some data analysis on our real and generated data as well as come up with/test multiple methods of detecting what is a real gesture.

I spent most of this week doing some analysis on the data we had collect compared to the fake data we had generated to try to understand why our real data performed so much better than our fake data. I did this by plotting several combinations of three axes (x, y and size) for the letter L. I chose to do the letter L because the sign for this letter a good variety of finger angles. To begin, we had more fake data than real data to train our models on, which is interesting considering our real data trained a model with higher accuracy. Based on my plots, I believe the reason for better results with the real data is that we have more consistent consistent correlation between the finger angles and the IMU data. For example, the plot I created comparing Middle Finger vs. AccelerometerZ vs. GyroscopeY showed a very scattered plot that does not imply any correlation for the fake data. The real data, on the other hand has a much more clear pattern (eg. bigger AccelerometerZ values correlates to a larger GyroscopeY value). Below are those two comparisons. This makes sense because when generating the fake data, we were unsure how to model the IMU data since we weren’t able to find good documentation on the values they would output based on certain movements or orientations.

I also collected some extra data, since I am in possession of the glove, for a “space” character as well as some random gestures so that the system does not try to identify letters while the user is transitioning between letters.

I would say we are on schedule since we had taken much less time than anticipated for the glove construction. The next steps are to integrate the glove with the software system so that it can do real-time interpretation of the data collected from the glove, so I will be working on this with Stephanie this upcoming week.

Posted on

Team’s Status Report for 10/9

This week, the team worked on finishing up the glove fabrication, writing the design review report, and testing on real data. Sophia worked on building the glove prototype with the system built on a protoboard so that we can start testing earlier as we wait on the PCB order. While the construction is not yet perfect by the team’s standard, we were able to get a preliminary set of data. Rachel took on the code implementation for communicating between the Arduino and the computer and recorded a set of ASL letters with an ample amount of samples for each letter. She also made some suggestions for the glove to collect data better, which the team will work on in the next week. This first set of data was then passed onto Stephanie, who did some preliminary training and testing using the best-performing models that were trained on the generated data. The results look quite promising and showed high accuracy metrics that meet our requirements. Overall, we are ahead of the schedule as the individual tasks all took less time than expected.

Next week, we will be spending the early half to finish up the design review report and other times for refining the glove system and getting data. Since we will be meeting up during lab times, we will work together on stabilizing the glove parts and getting data from Sophia and Stephanie. If time permits, we will get data from others and finalizing the ML models to use, which can put us ahead of schedule.

Posted on

Stephanie’s Status Report for 10/9

In this week, Sophia finished building the glove and Rachel was able to get our first set of real data. Since I have already performed validation tests on the models I have used with the generated fake data, I decided to use these tuned models on the real data after preprocessing them. Surprisingly, the results were overall better than that of the fake data. This shows that our fake data was not well generated. One possibility is that we included too much variance in generating sensor values. However, though the accuracy metrics were quite different, the trends remain the same. Random forest classifier achieved highest accuracy while perceptron had the lowest. I also did some extra tuning with neural net, but there wasn’t any significant improvement in accuracies, likely because our data isn’t high dimensional. One thing I would like to add is that this set of real data is only from Rachel, so there could be a possibility of overfitting which explains the high accuracy metrics.

In terms of schedule, we are actually ahead. We were able to get data from both type of sensors. We do need to work on getting consistent data and ensure the craftsmanship of the glove since Rachel mentioned some parts came undone. We will need to make sure that the sensors on the glove are stabilized before moving on to collect data from others.

Next week, I’ll be working on fixing the glove with the team and gathering more training data, starting from Sophia and me. If time and resources permit, we will try to find others who can sign for us. We will also working on finishing up the design report.