Category: Hinna’s Status Reports

This week, I finished working on the final presentation with my group, where I added some walkthroughs of the webapp (attached to this post). I also started working on the final poster where I wrote out our product pitch, some of the system description, and worked with my group members to do the rest. Additionally, I began writing the final report, making adjustments to the introduction, user requirements, and project management based on changes we made since the design review.

Our project is on schedule as of now, where we mostly need to finish working on solution tradeoffs, user testing, and  final deliverables. Over the next week, I will work with my team to finish user testing, the final poster, the final video, and finalize plans for the final demo. I will also work on analyzing some of our tradeoffs (i.e. epochs vs accuracy for each of the models).

Over this past week, I created tradeoff graphs based on metrics we found for model accuracy, which we graphed against the number of epochs used in training and testing respectively. In these graphs, we identified that the dynamic models are performing very well (93%+ accurate) which most likely has to do with the fact that we had to create our own training data for them. On the other hand, the 1-finger and open-hand models were performing pretty poorly (60-70% accurate). So, along with my teammates, I made more training data for those models to see if adding that would help improve their accuracy.

Additionally, as the dynamic models are now integrated into the webapp, I examined how well they were doing, testing them personally at various angles, distances (within 2 feet), and using both hands in order to see how accurate they were. I found that when doing the signs quickly, within one second, the prediction was not accurate but when doing it more slowly, the accuracy improved. This finding was also reflected in some of our user test results where we had 2 users test the platform on Friday.

Finally, I have been working with my teammates on the final presentation, where I have updated our schedule and project management tasks, altered our Solution Approach diagram to account for the number of neural networks we have, adjusted our user requirements based on changes made since the design presentation (i.e. our distance requirement lowered and our model accuracy requirement increased), adjusted the testing/verification charts, and finally included the tradeoff curves for testing & training accuracy vs the number of epochs.

Our project overall seems to be on schedule with a few caveats. One is that we are head of schedule in terms of integration as we finished that last week, so our initial plan of integrating until the very end of the semester is no longer the case. However, our model accuracy is not quite where it needs to be for every subset of signs, so given that we only have about a week left, the fact that we might not be able to get them all to our desired accuracy of 97% makes it feel like we are a little behind. Additionally, we held user tests this past week and only 2 users signed up (our total goal is 10 users), which means our testing is behind schedule.

As for next week, my main focuses will be getting more user tests done, finalizing the tradeoff curves in the case where our model accuracies are improved through the addition of more training data, and working on the final report, demo, and video.

Throughout this week, I was feeling fairly sick, so I did not accomplish as much as I intended to. However, I was able to continue working on creating testing/training data (specifically focusing on our 15 dynamic signs) as well as continue working locally with our machine learning models to see which groupings of signs are performing the worst (currently this is the fist model). Additionally, we did a virtual dry run through of our interim demo, which we couldn’t do since our whole group was sick, and based on TA/professor feedback, I started looking into normalizing the inputs we get to our model in order to make sure that slightly varying the distance or hand position doesn’t affect the predictions. I will continue this research along with my other group members this week to determine if the normalization will be feasible given the amount of time we have left.

Additionally, given that there are only 3 weeks left in the semester, I began planning for user testing by researching the best webapp user satisfaction questions to ask, reaching out to potential testers (other students at CMU), and creating a testing schedule. Given that we initially planned to start user testing last week, this portion of the project is behind schedule.

Overall, we are slightly behind schedule due to our group-wide illnesses this past week and the fact that our models are not working for every letter, resulting in us pushing back user testing. In order to get back on track, we have scheduled two additional meetings this week (outside our normal class time and previously set weekly meets), so that we can dedicate more focused work time to our project.

Next week, we hope to have the static models fully done and the dynamic models significantly progressing (with at least 7/15 having 80%+ prediction rates). Additionally, we plan to start implementing a quiz/testing menu option on the webapp, where users can be quizzed on a random subset of signs. Finally, we plan to have a user testing schedule planned out and ready to go so we can get some feedback in these final weeks.

This week, I personally worked on my portions of creating the ASL testing/training data, adjusting the instructional material (videos and text) to make sure it fits with how our model is reading hand data (i.e. making sure hands are angled so that all/most fingers are visible in frame), and locally examining the web app + ML models.

In regard to examining the web app, I have been brainstorming with Valeria on some of the suggestions that were given to us on our previous week meeting, where we are trying to decide the best way to store user statistics, include a random quiz mode for users to use with the ability for them to select which categories to be tested on, and format/display user profile information. As for the machine learning models, I have been locally working with them to see which is performing the best (currently the 1 finger model is) and to try to determine holes in how they have been trained (i.e. the sign for ‘d’ requires a certain tilt for it to be accurately detected). After identifying some of these signs, I have been working with Aishwarya to figure out solutions for theses issues in the models.

Our schedule is technically on schedule but is in danger of being behind, especially because my other two group members tested positive for Covid this past week. To account for this, we are pushing some tasks back a week on our schedule (such as integration) and doing our best to collaborate virtually.

In the next week, we plan to successfully execute an interim demo with most if not all of the static sign models working, along with a webapp that can teach users the ASL terms we have chosen after they make an account on the platform.