This week, I worked on our final presentation and continued to debug our pipeline testing results. We will continue to work on improving the accuracy this week as well as begin our poster and final report.
Nia’s Status Report 4/18
This past week and the week before, I mainly worked on completing the frontend and integrating it with the pico and backend. Specifically, I created an APIs to deliver the pico’s readings and return the model’s evaluation and confidence value. I also helped Kat with evaluating the ML model and contributed to the team’s overall data collection. We currently have a KNN model that evaluates signs with 94% accuracy. We are continuing to improve the model and debug the results that it returns when testing the entire pipeline.
This week, we began working on the final presentation where we will deliver our results from this past semester.
Through this project, I learned how to implement a Flask API backend as well as the difference/tradeoffs between ML models, specifically KNN and CNN. I mainly did research online and looked at code from past classes and projects to acquire this new knowledge
Nia’s Status Report 4/4
This week, I completed the data collection script and finalized the Practice page. We began testing but ran into issues with reading from the same port that runs the script. We will research solutions and begin collection data this week. We are a bit behind and may not be able to fine tune our model as extensively, but we will be able to evaluate signs by the end of next week. I will also work on the API endpoints to post to the frontend once we can confidently collect data.
Nia’s Status Report 3/28
This week, I updated the frontend to not receive feedback from the user about whether or not they signed correctly when the model identifies the sign as incorrect. This is to prevent our model from possibly receiving incorrect information and as a result, decreasing the accuracy of the model.
I also began writing the data collections script that we will use to semi-automate our data collection process. Specifically, this script will allow us to automatically save .csv files with proper naming mechanisms as well as pause data collection at different points and resume from where we left off (for example, complete 20 trials of “A”, end the terminal process, start the terminal process, continue at trial 21 of “A”.
I addition, I helped brainstorm how we incorporate the touch sensors on the glove such as positioning and wiring.
Next week, I plan on completing the data collection script so that once we have the hardware and processing complete, we will begin collecting data.
Team Status Report 3/21
Now that we know the sensors can receive data, we’re going to move quickly towards assembling the glove so we can train the ML model. We’d like to have it assembled soon so we can have something ready for the interim demo in a little over a week. We’re considering adding capacitive touch sensors to the glove (which we’ve already ordered for testing) to make sure we can accurately represent all of the signs.
The most significant risk that could jeopardize the product is how well the sensors hold up during testing and data collection. We will prioritize keeping the sensors in place and secure during data collection to ensure we collect accurate data across all trials. If the sensors move significantly during data collection, we run the risk of our model incorrectly assigning classes.
Nia’s Status Report 3/21
This week, I attended the ethics lecture and discussed with my team along with others how we can be mindful of ethical tradeoffs when creating new technology for society. Along with that, I confirmed the set up of our feature vectors and the machine learning pipeline we plan to implement. Next week, we will assemble the glove and I will test the model’s evaluation API that connects to our frontend interface.
Nia’s Status Report 3/14
This week, I fully implemented the Practice page within the existing React/Vite localhost framework. Currently, the practice mode cycles through all 26 ASL letter signs (A–Z), evaluating each response as correct or incorrect. For testing purposes, a 1-in-4 probability of an incorrect answer is simulated. Upon completing all 26 characters, the user is presented with a results summary screen displaying their overall performance.
For the machine learning subsystem, I built and validated the data generation and training pipeline The pipeline currently generates a synthetic dataset and feeds it through a neural network classifier. I initially had an issue where the output layer was hardcoded to 5 classes while the dataset contained 36, causing the model to perform at random chance with ~5% accuracy. After fixing the output layer to dynamically match the number of classes in the dataset, the model achieved 100% accuracy on the synthetic data. This is expected as the synthetic data is generated from a distinct random base vector with minimal noise. This result confirms that the pipeline is functioning correctly and is ready to be evaluated against real sensor data.
Next week, I plan to design and implement the transcription mode. I will also begin forming the actual training dataset from collected data points. Once an initial set of real samples is gathered, I will apply data augmentation techniques to artificially expand the training set. This is a standard approach for sensor datasets where collection time is limited and will allow us to increase dataset size significantly without requiring more physical recording sessions. If time permits, I will also set up a REST API endpoint that exposes the model’s predictions to the frontend.
Nia’s Status Report 3/7
This week, I worked on our design report. I worked on the introduction, design requirements, system implementation for the processing/machine learning subsystem and the software/user interface subsystem, testing/verification, and some portions of project management. As of now, our main concern is begin able to collect data for the machine learning training. We won’t be able to do so until we have built a MVP of our hardware system. I also created a design of our intended UI for teach mode using Figma Make
I plan to implement the frontend during the next couple of weeks. Once we have a working hardware system, I will assist with data collection and processing.
Nia’s Status Report 2/21
This week, I worked on our design presentation slides and began creating the file structure for our code. We will be adding code for the frontend and machine learning portions as we wait for our parts to arrive next week.
I will also begin designing the UI on figma. We plan to have a test mode to begin where the user is shown a letter/number and they are to sign the letter and receive feedback (correct/incorrect).
Nia’s Status Report 2/14
This week, I worked on ordering parts for our prototype. Specifically, I ordered the gloves and began research on what monitor we want to use to display our UI. I also created our GitHub and began planning our ML model system design.
Our progress is a bit behind schedule as we hope to begin collecting data before spring break. We are still waiting for our sensors to arrive and then we can begin test such as number of sensors per finger.