This week, my main focus was user testing and also examining the accuracy of our static and dynamic models.
In regard to user testing, I made a google form survey that asks our testers to rate different usability features of the website as well as how helpful they felt it was in teaching them ASL. I also made a step by step guide for users to follow when we conduct the testing, which we will use to see how intuitive it is for users to complete the steps and to make sure they test various different actions (i.e. making sure each user tries to correctly and incorrectly sign on the platform to see the results). Finally, as a TA for the ASL StuCo this semester, I reached out to students who are either semi-experienced or experienced in ASL to conduct our tests. We will also be reaching out to a few people who are brand new to ASL in order to get a beginner’s perspective on our platform.
As for the models, I have been trying different combinations of training epochs and prediction threshold values (where the model only outputs a prediction if it is over a certain number i.e. 90%) to determine the best weights for the model to make it more accurate. In these tests, I have been able to identify certain signs that consistently have trouble over the combinations as well as some environmental factors like contrasting backgrounds that can influence the results. Because of this work and feedback during our weekly meeting, I will continue trying these combinations in a more intentional way, where I will record data related to the accuracy of the models based on epochs and/or threshold values in order to graph the tradeoffs associated with our system. The final accuracy data collection and graphs themselves will be recorded at the end of next week in order to account for any training shifts we make this week based on some of the identified signs with consistently inaccurate predictions.
Our project is on schedule at this point, however our model accuracy is not quite at the 97% we set out for it to be at the beginning of the semester. Since we planned to be adjusting and tuning the model up to the very end of the semester, this is not too big of a deal but we are going to start shifting focus primarily to testing as well as the final presentation/demo/report. Thus, while we are on schedule, our final implementation may not be as robust as we had planned for it to be.
Next week, I will be conducting user tests along with my teammates, focusing on factors such as hand dominance, hand size, lighting, distance from camera, and potentially contrasting backgrounds.I will also be examining the dynamic models more in depth to identify signs that are having less successful detections. Additionally, I will be recording information on accuracy vs threshold value and accuracy vs epochs used in training, then using that information to make tradeoff curves that we can hopefully include in our final presentation.