In this week, I worked on researching on why the neural network’s accuracy increased with the added labels of “space” and “transitions”. To compare the accuracies of different data sets, I compiled three different sets of data: one with all the labels, one with all the letters and the space labels, and one with only letter labels. To see if the differences in data did affect the overall accuracy, I performed hyperparameter tuning when training each data set’s model to find the best performing accuracies. The results showed that best tuned models were able to reach 93%+ accuracy for all the data sets. (The accuracy also varies by runs due to random initialization in the model)
In conclusion, the added data labels didn’t actually improve the accuracy of the model. Since neural networks can have various amount of hidden layers and activation function, it’s likely that we just did not find the optimal parameters of the model for a particular dataset.
Furthermore, we found that our first set of data were mostly taken with the hand facing one direction. This may significantly impair our model performance. This was further proved when I tested the model trained on first set of data. On our two newest sets of data, the model only showed 4% and 21% accuracy. Hence it’s likely that we will not be using this set of data in the future.
We also adjusted the IMU on the glove this week and just got a new set of data. Preliminary tests on this dataset showed high accuracy (98%), but model trained on second data set only had a 50% accuracy on this set, which signifies this is a large change to the overall data. We’ll be doing more real time testing next week. I’ll also be working on making some data comparison with our older data to see why the old model did not perform as well in real time testing.
I would say we are a little behind schedule in terms of gathering data as we are still contacting people who use ASL. But we’re doing well with data training/testing and model analyzing.