Category: Ricky’s Status Report

This week, I did some brief work in finalizing some metrics for the ML model. This included some rigorous testing into the real-time performance and the test-time performance. I was also able to develop confusion matrices on some data to get a better picture for how the model is working. I am also pretty confident in my final ML model design with the addition of some more words into the second model to allow for more refined predictions. I also briefly tested the model with the PCB as made by Ria. However, the performance of the model is inconsistent, which I believe is due to the fact that the PCB is currently being secured by tape. I will hope to continue testing once the PCB is more reliably secured, but we always have the original protoboard as a backup plan which I will propose if needed. I also spent a decent chunk of time working on the poster. This tied nicely with the metrics that I found. I also wrote out the start of the final report which will hopefully have us finishing before demo day.

My progress is on schedule with the slight caveat that I am waiting for the PCB design to be finalized. I foresee that minimal adjustments will need to be made once that is attached but we will have to wait and see.

Next week, I hope to quickly finalize our demo day product. Finish the report and video and present on demo day.

This week, my main task was debugging our issues with the machine learning model. During the carnival, I trained some preliminary models, but their performance was not that good in real time. To compensate for this issue, I tested a variety of architectures as well as new activation functions. I also manually selected the new words that I determined were more easily mixed up and created a smaller simpler model for just those words. This allowed me to develop a two-model approach to classification that leveraged the original model’s strength at half of the vocab and the more narrow model’s strength for the rest. I also developed a model that handled time series data by stitching in the previous five frames of data as one input into the NN model. Armed with the variety of models, I entered this weekend with the plan of testing them all. I was able to determine that the two-model approach performed the best, being able to determine 10/11 of the words consistently. This was not a 100% rigorous set of tests, but it gave me enough metrics to assist with our final presentation slides, so I was content with that performance as of now.

My progress is on schedule.

Next week, I will narrow down which model I will use for our final demo with a more rigorous and formal testing of the two approaches. I will also assist in the development of the poster by adding additional pictures from the ML side. I will also assist with the video and the report.

Addressing the additional question, I definitely needed to relearn how to work with ARDUINOs and learn about how to interface between ARDUINO and Python. Since I handled the bulk of the Bluetooth and the machine learning side, I did need to handle a lot of the communication between the two platforms. This meant that I had to consult many pages of ARDUINO documentation as well as perform extensive research into compatible Python libraries. For learning strategies, most of it was on reading documentation and reading StackOverflow. I did watch a few videos of other example projects for assistance as well.

Most of the week, I was focused on collecting data and everything data-related. At the start of the week, we initially tried to start collecting data. However, there were some issues with the left glove. This ranged from some of the wire connections being loose to various parts of the flex sensor being caught in the string. I assisted in hashing those out and added my opinion about additional changes. While this happened, I adjusted the scripts to handle the two streams of data from the gloves. Then, once everything was set, we collected data for both me and Somya. We collected for the whole BSL alphabet which included 26 letters and an additional sign for the resting state. With around 1200 points for each letter, we were able to collect a total dataset of size 64800. I also handled cleaning up the data and visualizing it before we train a model with it next week.

Ideally, we would’ve also collected data for Ria, but due to time constraints, we were unable to. I believe that the best course of action is to proceed with this slightly smaller dataset and collect more if the performance is not good enough. With that in mind, I am on schedule.

Next week, I will develop a new ML model with the data we collected, evaluate performance, and make any additional changes as needed. I do foresee potential classification issues with several letters of the BSL alphabet, but I will hopefully discuss that with my partners on Monday. We might need to curate our vocabulary more rigorously or find another solution if the ML model does not reach our desired accuracy metrics. I will hopefully be performing verification tests next week to ensure both the accuracy and speed of the ML model and the overall classification system.

This week I did a whole restructuring of our codebase. Originally, we had the Bluetooth and USB scripts written in the same file. But for ease of use, I decided to refactor the code so that the pattern in many parts of the code was separated. I also spent the bulk of the week fixing our Bluetooth system. When running the bleak package, I realized that its performance was very inconsistent for my Windows laptop. Further research identified that Bleak was not fully developed for Windows. As a result, I was forced to look into other Bluetooth packages. I eventually found the simplepyble package. I was able to integrate it and test that it properly connects to our gloves and ARDUINO. However, further tests showed that it was incompatible with the pyttsx3 package. This was because the prior demands a multi-threaded setting while the latter demands a single-threaded setting. I eventually found another speech package “win32com.client” using the Dispatch object. This worked! I integrated the Bluetooth, Speaker, and ML systems and did some benchmarking for performance. Quick tests identified that the time from sign to speech for USB is around 0.6 seconds and Bluetooth is 0.82 seconds. There are some quick hyperparameter checks that we can change to improve time performance at the risk of accuracy which we will look into later. I also helped assemble the circuit for the second glove. Please look below for our public repository for the codebase or future reference: TStar111/EchoSign (github.com)

The schedule is right on schedule.

Next week, the bulk of the time will be spent on collecting data. If time permits, I will start training and benchmarking the ML model.

 

At the start of the week, I handled the training of the model with the dataset that we collected from the previous week. I was able to train a simple NN architecture to a high level of test accuracy. (See below for training visualization). Afterward, I was able to go back and test real-time performance. I didn’t test to a rigorous degree but I was able to replicate a high degree of performance on the glove as well. I was also able to quickly check the speed of the ML classification step. This step identified that the inference step takes an infinitesimal amount of time. This allows me to meet my design requirements of an accuracy above 90% and inference time of less than 25ms. I also took some time to add an additional classification heuristic on top of the ML model. This was implemented using an additional wrapper on the ML model, which checks for repeated occurrences of the same letter before firm prediction. I also added speaker capabilities from the pyttsx3 library in Python. I also spent a significant portion of the week debugging Bluetooth capabilities. We faced specific limitations due to our laptop not being Linux, but we were eventually able to identify the Bleak library as a reliable source for Bluetooth capability.

My progress is on schedule, though I anticipate that I will have less time next week due to upcoming exams.

Next week, I hope to assist with the development of the second glove as needed. I hope to also establish Bluetooth communication between the two gloves and the laptop at the same time. I will also hope to figure out what we plan on doing with the speaker. If time permits, I hope to start collecting data from the two-glove setup next week.

This week, I had a hand in multiple elements of our project. I was primarily responsible for the software end things. I finalized the data collection script on the ARDUINO side. I also wrote several scripts to clean up our dataset once we were able to collect the data. I wrote some scripts that allowed for the plotting of the dataset for us to understand how the signings are different from each other. I was also able to test and revise my python scripts that received the ARDUINO data which allowed for successful data collection. I also assisted in glove fabrication. I helped debug issues with the ARDUINO including high source voltage and verifying the ADC. I also assisted Somya in attaching the sensors to the glove. Then, once the glove was ready we spent a decent chunk of time collecting around 30,000 datapoints across the three of us.

My progress is on schedule given the overall pushback of the project.

Over the weekend, I hope to train both NN and SVM using the collected data over the weekend. The plan is to have them trained by Tuesday/Wednesday and present the results to my teammates.

This week, I primarily worked on the design document. I was responsible for large chunks of the text regarding the machine learning training/testing and the data collection and labeling. I touched upon the use-case requirements, some design trade studies, and big parts of the system implementation and testing. Additionally, I also finalized what would be appropriate for EDA. Specifically, I want to be able to graph the average values of each sensor for each letter and also calculate relevant summary statistics like mean, median, variation, etc.

The software development is on schedule but the actual data collection component is a bit behind. I will just wait for the glove to be developed to catch up. I will keep working on the software until then.

Next week, I will hopefully help assemble the glove and data collection. I will also implement the EDA stated above as well as finish off the last two ML model training modes.

This week, I primarily focused on implementing the script that will train an NN architecture and save the model for future inference use. This implementation has a variety of features including variable choices of input data, cross-validation raining to identify the optimal stepsize, and usage of the wandb package to track training performance through the epochs. I have also implemented the SVM model trainer. This was pretty straightforward with a heavy dependence on the sklearn package. I also tuned up the data collection for an automated collection of 26 letters in one shot. I performed some research into pre-training data analysis based on my “Estimation, Detection, and Learning” class. Based on the research, I added some potential action items to prepare a clean dataset. 

I am on schedule as of right now.

Next week, I hope to put significant work into the design document. In addition, I hope to flesh out a robust testing module, finalize dataset analysis, and begin collecting data with the glove (dependent on the glove’s progress).