Posted on

Somya’s Status Report for 3/30/2024

This past week, I made some changes to the glove and helped figure out some bugs we were having with the bluetooth. One thing we’re noticing in our sensor data is that we get some discrepancies depending on who is using the glove. This is to be expected, as we all have different hand sizes as well as slight variation in the way we make each sign. I’m trying to come up with ways we can make the data more consistent besides post-processing cleanup. In our weekly meeting, we discussed adding a calibration phase as well as a normalization of the data which should definitely help but I still think securing the sensors at additional points than what they are now will also make a difference. I had a few stacked midterms this past week so while my progress is still on schedule, I didn’t make as much progress as I would have liked. This upcoming week, however, I should be able to dedicate a lot more time to capstone, especially with the interim demo around the corner. 

 

More specifically, this upcoming week I would like to add the haptic feedback code to our data collection scripts. Our current plan for MVP is to have the LED on the Arduino blink when the speaker (either on the computer or the external speaker) outputs the signed letter/word and more importantly, that it outputs it correctly. I think we should color code the output based on the success of the transmission: red for didn’t go through, yellow for possible but might want to resign, and green for successful transmission. I also want to order some vibrating motors because for our final prototype we want to have this type of feedback so the user doesn’t have to constantly look down at their wrist. Finally, I want to bring up changing/adding to what position we deem to be “at rest”. Right now, we just have the user holding up their unflexed hand as at rest, and the model is pretty good at recognizing this state, but this isn’t really practical—people’s at rest is typically with their hands at their side or folded in their lap, or moving around but not actually signing anything. The model sort of falls apart with this notion of at rest, and I think adding this to our training data will make our device more robust. 



Posted on

Team Status Report for 3/30/2024

The most significant risk that we face right now is the timeline for collecting the data and the robustness of the ML model. We hope to start collecting data next week which gives us ample time to collect more data if the initial dataset needs to be increased. Ricky also has plans for testing multiple different architectures and hyperparameters if the performance is insufficient. The other risk involves just keeping the readings consistent. This risk is being mitigated through some newly implemented calibration techniques and the introduction of a PCB.

The major change to the design is that we will as of now stick to the laptop as the main speaker component. This is due to our choice of ARDUINO which limits the speaker capability from the Arduino chip. We will circle back to this idea if time permits after testing and model tuning.

We will be proceeding with the original schedule. Ricky has adjusted a bit of his timeline to reflect some of the delays in Bluetooth integration and Ria has added the PCB creation timeline but it is relatively similar to the original schedule.

Posted on

Ricky’s Status Report for 3/30/2024

This week I did a whole restructuring of our codebase. Originally, we had the Bluetooth and USB scripts written in the same file. But for ease of use, I decided to refactor the code so that the pattern in many parts of the code was separated. I also spent the bulk of the week fixing our Bluetooth system. When running the bleak package, I realized that its performance was very inconsistent for my Windows laptop. Further research identified that Bleak was not fully developed for Windows. As a result, I was forced to look into other Bluetooth packages. I eventually found the simplepyble package. I was able to integrate it and test that it properly connects to our gloves and ARDUINO. However, further tests showed that it was incompatible with the pyttsx3 package. This was because the prior demands a multi-threaded setting while the latter demands a single-threaded setting. I eventually found another speech package “win32com.client” using the Dispatch object. This worked! I integrated the Bluetooth, Speaker, and ML systems and did some benchmarking for performance. Quick tests identified that the time from sign to speech for USB is around 0.6 seconds and Bluetooth is 0.82 seconds. There are some quick hyperparameter checks that we can change to improve time performance at the risk of accuracy which we will look into later. I also helped assemble the circuit for the second glove. Please look below for our public repository for the codebase or future reference: TStar111/EchoSign (github.com)

The schedule is right on schedule.

Next week, the bulk of the time will be spent on collecting data. If time permits, I will start training and benchmarking the ML model.

 

Posted on

Ria’s Status Report for 3/30

This week, I started by deciding if the speaker was a feasible endeavor for our team. Ricky mentioned a library called Talkie that does text to speech on an Arduino. I thought about how we can use that library on the glove and output speech through a speaker after sending that audio data through an amplifier circuit that I built. Unforuntately, when trying to run the test script I wrote, I found that Talkie does not support the BLE board we have. I assumed since the BLE board has the same architecture as the other boards Talkie does support, it would be worth testing it out, but the source code cases on the board in use and BLE was not on that list. Instead of modifying the source code for the library, we decided that for MVP, it was the best use of our time to keep the speech output coming from the computer.

My next task for this week was figuring out how to power the board. The LiPo 3.7V batteries are the ideal size for our portable product, but the BLE board takes in 4.5V-21V into Vin. I thought about if a larger bulkier battery could work but could not find a battery that was both cheap and was lightweight enought to be mounted on a wearable device. I decided to look into a step up dc dc converter instead. The capacitors on them are a bit bulky, but they will do the trick and the MT3608 chip can output a voltage in the range we need. I found the schematic for this board, added it to our custom pcb, ordered the part, and plan to test the wireless powering of our board on Monday.

Posted on

Somya’s Status Report for 3/23/2024

I finished fabrication of the second glove as well as work on debugging the transmission of the bluetooth data from the Arduino to the laptop that is running the Python script used to collect the data and compress into a csv file. In addition, I brainstormed various ways we can remove the laptop as being a required component of our final demo. 

My progress is on schedule, but we are slightly behind on the testing of the double glove due to waiting for the second Arduino BLE compute unit to arrive. Once it does arrive however, we should have all the moving parts in place to begin testing integration of data from both gloves immediately. In the meantime we are collectively working on other features like the speaker and haptic feedback, as well as cleaning up of noisy data to improve the ML model.

This next week, I hope to finish the circuit with both the speaker and haptic feedback, as well as be fully finished collecting data for the BSL double-handed alphabet so I can see what issues the synchronization of the two input streams brings and start debugging that. 


Posted on

Ria’s Status Report for 3/23/2024

This week I focused on creating the PCB layout for our circuit thus far. I started off by drafting the circuit with each flex sensor and op amp. Then we had to finalize which board we are using which we did altogether (determining if Bluetooth would meet our latency needs). We finally decided to stick with the Arduino Nano BLE Sense Rev2 and I added in those headers into the schematic. Finally, I added headers for a battery, and headers for the speaker and haptic motor. 

The next thing I did this week was solder five more flex sensors to red white wire pairs so that they can be ready to attach to the second glove. Now that our first glove works, we decided to parallelize the tasks needed to be done to duplicate it. The next thing I’ll focus on for the following week is getting the speaker and haptic motor driver to work on one glove. 



Posted on

Ricky’s Status Report for 3/23/2024

At the start of the week, I handled the training of the model with the dataset that we collected from the previous week. I was able to train a simple NN architecture to a high level of test accuracy. (See below for training visualization). Afterward, I was able to go back and test real-time performance. I didn’t test to a rigorous degree but I was able to replicate a high degree of performance on the glove as well. I was also able to quickly check the speed of the ML classification step. This step identified that the inference step takes an infinitesimal amount of time. This allows me to meet my design requirements of an accuracy above 90% and inference time of less than 25ms. I also took some time to add an additional classification heuristic on top of the ML model. This was implemented using an additional wrapper on the ML model, which checks for repeated occurrences of the same letter before firm prediction. I also added speaker capabilities from the pyttsx3 library in Python. I also spent a significant portion of the week debugging Bluetooth capabilities. We faced specific limitations due to our laptop not being Linux, but we were eventually able to identify the Bleak library as a reliable source for Bluetooth capability.

My progress is on schedule, though I anticipate that I will have less time next week due to upcoming exams.

Next week, I hope to assist with the development of the second glove as needed. I hope to also establish Bluetooth communication between the two gloves and the laptop at the same time. I will also hope to figure out what we plan on doing with the speaker. If time permits, I hope to start collecting data from the two-glove setup next week.

Posted on

Team Status Report for 3/23/2024

One significant risk that we face right now is the development of the synchronization algorithm/heuristic. We were able to establish a Bluetooth connection from a laptop to the gloves via the Bleak library in Python. We are currently doing some preliminary research into connecting two Bluetooth connections at once to the laptop. We have a contingency plan of connecting the gloves and sending one stream of data to the laptop if our original plan seems infeasible. We are also exploring different strategies in generating audio output from the glove itself. We seem to be slightly blocked by our choice of ARDUINO so our contingency plan is maintaining speaker functionality from the laptop. 

There have been no major updates to the design as of right now. We will closely monitor the situation with the speaker and Bluetooth in the upcoming week to see if any adjustments need to be made.

We are on schedule as we have started working on the construction of the second glove. We have essentially finished prototype 1 as well, making it ready for presentation during the interim demo.

Posted on

Somya’s Status Report for 3/16/2024

What did you personally accomplish this week on the project? Give files or

photos that demonstrate your progress. Prove to the reader that you put sufficient

effort into the project over the course of the week (12+ hours).

This week I attached all the sensors onto the glove and worked on implementing/testing the circuit. After performing unit tests and discussing an expanded range in our weekly meeting, we determined that op-amp use would be the way to go, and through use of op-amps we managed to expand the voltage output from 0.8V to 2V. 

 

Is your progress on schedule or behind? If you are behind, what actions will be

taken to catch up to the project schedule?

My progress is on  schedule. 

 

What deliverables do you hope to complete in the next week? 

 

Now that we are in the data collection phase, I want to do some more research into how we can clean up the data with noise reduction algorithms. In addition, I would like do some more research on the Bluetooth data transmission so we can more on that in parallel along with the data collection, as well as fabrication of glove 2

Posted on

Ricky’s Status Report for 3/16/2024

This week, I had a hand in multiple elements of our project. I was primarily responsible for the software end things. I finalized the data collection script on the ARDUINO side. I also wrote several scripts to clean up our dataset once we were able to collect the data. I wrote some scripts that allowed for the plotting of the dataset for us to understand how the signings are different from each other. I was also able to test and revise my python scripts that received the ARDUINO data which allowed for successful data collection. I also assisted in glove fabrication. I helped debug issues with the ARDUINO including high source voltage and verifying the ADC. I also assisted Somya in attaching the sensors to the glove. Then, once the glove was ready we spent a decent chunk of time collecting around 30,000 datapoints across the three of us.

My progress is on schedule given the overall pushback of the project.

Over the weekend, I hope to train both NN and SVM using the collected data over the weekend. The plan is to have them trained by Tuesday/Wednesday and present the results to my teammates.