The most significant challenge we are facing in our project is the ML algorithm’s detection of our chosen set of words. Currently we have okay accuracy, but prediction is inconsistent, in that depending on the user the accuracy changes, or the model is able to recognize the word, but not enough times in a row to meet the criteria for the classification heuristic. In addition, it is common for the double-handed words in ASL to incorporate some type of motion. This is significant because currently, our model relies on prediction based on a single frame, and not a windowing of the frames. This is probably affecting accuracy, so we are looking into mitigating this by incorporating some type of sampling of the frame data, but the tradeoff to this may be a decrease in latency.
In terms of changes to the existing design of the system, we have decided to nix the idea of using the 26 letters in the BSL alphabet. This change was necessary because in the BSL alphabet, for a significant portion of the letters, the determining factor is based on touch, and our glove doesn’t have any touch sensors. So instead, we found a 2017 research paper that found the top 50 most commonly used ASL words and split them into five categories: pronoun, noun, verb, adjective, and adverb. From each of these categories, we picked 2-3 double-handed words to create a set of 10 words.
No changes have been made to our schedule.