Posted on

Ricky’s Status Report for 2/24/2024

This week, I primarily focused on implementing the script that will train an NN architecture and save the model for future inference use. This implementation has a variety of features including variable choices of input data, cross-validation raining to identify the optimal stepsize, and usage of the wandb package to track training performance through the epochs. I have also implemented the SVM model trainer. This was pretty straightforward with a heavy dependence on the sklearn package. I also tuned up the data collection for an automated collection of 26 letters in one shot. I performed some research into pre-training data analysis based on my “Estimation, Detection, and Learning” class. Based on the research, I added some potential action items to prepare a clean dataset. 

I am on schedule as of right now.

Next week, I hope to put significant work into the design document. In addition, I hope to flesh out a robust testing module, finalize dataset analysis, and begin collecting data with the glove (dependent on the glove’s progress).

Posted on

Team Status Report for 2/24/2024

Right now the most significant risks revolve around getting our parts in time so we can begin construction of the glove and collecting data. We were unfortunately unable to place our orders in time for this week so we will hope to get our parts next week and start assembly of the glove. Meanwhile, development of the major software components (communication, ML) has started.

No major changes have been made to the system. We did buy multiple computing units but we will not choose one of the two until we can test their performance. 

There is a slight pushback in the timeframe to make the first glove but everything else is on schedule.

Posted on

Somya’s Status Report for 2/24/2024

I did some more research on how we integrate the sensors onto the glove as well as what specific resistor values to initially try in the voltage divider circuit with the flex sensors. The midpoint resistance of the flex sensors is 10,000 ohms, so depending on the voltage of the power supply I will select a range of divider resistors accordingly. In addition, I looked into the various tradeoffs choosing these sensors implies, as I definitely want to touch on that in our Design Report, in addition to the testing and verification protocols for the data readings that we will obtain from the sensors, i.e. what voltage outputs to reject/keep. 

My progress is on schedule.  

By Monday, I should have a circuit schematic of the sensor integration pinout. In addition, I hope to get most of my assigned parts for the Design Report.



Posted on

Week of 2/18/2024

This week I focused on finalizing the design for our software interfacing and ordering parts.

I made an observation that Bluetooth may not satisfy our latency requirements which Ricky helped confirm based on a past project failing to incorporate Bluetooth for this reason. This is why I proposed using WiFi instead. Because we have BLE built into our board, we realized that we would need a different board preferably with a built in IMU with a triaxial accelerometer. We found the Arduino 33 Nano IoT which satisfied all of our requirements but it had less compute power. We value compute power so we decided to start with the Sense board and switch to this if it does not work out so we ordered both.

The next thing I did was write some code to connect to the board and retrieve data as well as format that data. I wrote it for the local and remote hosts. We still have to test this since we don’t have our boards yet.

Posted on

Somya’s Status Report for 2/17/2024

In addition to working on the Design Review slides and updating the Gantt chart, I was able to order the flex sensors, so hopefully we can start actually building the glove this upcoming week. Initially I wanted to use Sensor Symbol’s SpectraFlex sensors, but after calculating the cost for 10-12 sensors decided this would put too big of a dent in our budget. As such, we decided to go with Sensor Symbol’s original flex sensors (active sensor length 95mm to span the length of an average-sized finger). The tradeoff demonstrated by this choice is between cost and performance/aesthetics. In addition, I did some research on the word choice we want to demonstrate for our MVP. One of the comments we got from our initial proposal presentation was that the decision to have 26 double-handed words seemed arbitrary. To address this, I found a 2017 paper that identified the top 200 most commonly used words in the ASL vocabulary. They selected 56 ASL words from five word categories (pronoun, noun, verb, adjective, and adverb). 29 words from this subset are double-handed and 27 are one-handed. So, we might take inspiration from this paper and have our MVP vocabulary be based on signed word frequency. 

In terms of progress, my progress is on schedule. 

In terms of this week’s deliverables, I expect all our parts to arrive shortly, so in preparation for that I want to flesh out a circuit diagram for the pinout of the flex sensors with the computing unit and necessary resistors/op-amps. I want to do some research into which op-amps I should select for the best voltage output from the sensors (the op-amp acts as an impedance buffer). I also want to start thinking about the Arduino code we will use to conglomerate the data from the sensors and send them to the computer, as well as how we will test sensor output, e.g. determine a noise threshold to reject/accept voltage outputs based on how noisy they are. 



Posted on

Team Status Report for 2/17/2024

The most significant risk right now is waiting for parts to arrive promptly as well as the potential risks of relying on Bluetooth for communication. We can’t do anything about the parts, so we will proceed by working on as many software components as possible. To mitigate the dependence on Bluetooth we bought an additional chip that allows for wifi connectivity which will serve as an alternative if Bluetooth doesn’t work out.

There were several changes to the requirements. In our design, we decided to use a laptop to handle the machine learning prediction and speaker output. We moved away from using a smaller computing unit because we believed that the main goal of our project should be on the wireless and doubly nature of our gloves. We also have decided to use the British sign language alphabet as our goal set. This is because it provides a standardized set of 26 double-handed gestures that can be used with each other. We also reduced accuracy goals to 85%. This is to reflect the increased complexity of the double-gloved design as well as past projects’ results (Gesture Glove achieved 75% real-time accuracy with a single glove). We also are looking into reducing latency requirements due to the potential slowdown caused by Bluetooth transmission. We haven’t nailed down an exact number right now but are doing research into it. No major cost updates as we order our initial parts.

We are on schedule. No major changes to schedule.

Part A (by Somya)

Our product will enhance the welfare and safety of the deaf community because of its usability quotient. By having a discrete pair of gloves that can translate sign language to speech, they will facilitate communication between the deaf and non-deaf population. As such, the product will minimize the need for separate structures to be established for deaf people so that they can go about their day-to-day tasks in a more convenient manner. The better the communication between the deaf and the non-deaf community is, the more integrated and less alienated the former population will be. In terms of safety, our product could be really useful for the deaf to communicate in an emergency situation that could arise in any location where there may not be anyone that understands sign language. In such situations, timely communication is of utmost importance, and our product will be designed with this use case in mind. 

Part B (by Ria) 

Our product will be impactful for the deaf community and potentially those who are hard of hearing. Current society is progressing towards providing various disabled communities with tools that they can use to seamlessly navigate daily life. Our goal is to extend this mission to users who speak sign language wanting to communicate with people who don’t. This is a step towards establishing better conversions between the deaf community and those who can hear. 

Not only does this product attempt to solve a complex problem, but it also raises awareness about the nuanced challenges that the deaf communities face every day. By learning more about their needs and struggles, we are aiming to spark conversations – pun intended – about how we can go even further. This product can be useful in education, video games, and even during emergency situations. We hope that this project adds a few cobblestones to the road being paved towards a more inclusive society. 

Part C (by Ricky)

Our product is not meant to be marketed as something to be developed for significant profit. I claim that there are not a lot of economic factors that concern us because our main purpose isn’t to make a product cheaper or easier to produce. We would urge potential investors against trying to profit from our product because it is meant to bridge the communication gap between deaf individuals and non-deaf individuals. Inevitably, there will be a cost associated with purchasing our product but we hope that it will remain limited to the cost of parts. Due to its lightweight design we hope that the distribution of the product will be relatively seamless and costless. Basically, our product is not intended to be profitable. Instead, it is meant to help a disadvantaged community.

Posted on

Ria’s Status Report for 2/17/2024

I spent majority of this week ironing out the design review slides and its content trying to weigh all pros and cons of every decision we made and making sure we didn’t leave any major loopholes.

The first thing I focused on was the board we are using. Our current communication protocol called for bluetooth but since Gesture Glove found that bluetooth resulted in them not meeting timing requirements, we thought the IoT Nano board might be a good option – but it has less compute. Since we are more concerned with functionality first, we will make an iteration with the BLE Sense board then switch to IoT when we get to wireless communication.

I also tried to spend ample time on breaking down each subtask into smaller subtasks as much as possible. This was a joint effort with my team mates which involved creating more action items for our schedule and refining the block diagram as well as creating a data flow chart.

 

Posted on

Ricky’s Status Report for 2/17/2024

My main task for this week was to prepare slides for the design presentation. About this, I added more input into which ML models we would use, how much data we would need, how to collect the data, and the general data flow. I also set up GitHub and added the general file structure for the different scripts we would need. This includes gathering data, training the model, testing the model, and the general runtime routine. I also wrote the first draft of the script that is responsible for collecting the data via Bluetooth. I also revamped our entire Gantt chart to reflect the changes to our development cycle (prototypes). I am continuing to think about how we will store the saved models so that different models can be loaded based on the demo goals (NN, SVM, vocabulary)

My progress is on schedule.

Next week, I hope to draft up the scripts for training the various ML models. I hope to also finalize where we will store the models and how to reuse them when they have been thoroughly trained. I will also set up the testing routine to evaluate the performance of the models. I will also explore automatic testing for optimal hyperparameters. I will also assist in developing the physical glove as well as add my input to the way the data should be formatted from the glove

Posted on

Ricky’s Status Report for 2/10/2024

This week, I primarily focused on researching how the ML model will be incorporated into the design. The first major issue I looked into was the logistics of collecting the training data. Recall that we will be collecting our dataset to train the model. Past similar projects don’t offer too much information about this. Instead, I came up with the idea of using a Python script to automate collecting sensor readings while another person signs a specific word. Then, after exporting to an Excel file, we can go back and systematically label all the data with the specific word label. This is probably the plan for now and will be adjusted based on its performance when we get to that stage. I also considered additional processing on the input before the ML. Past projects seemed to preprocess the flex sensor output to a predetermined range of angle values. I believe that this is something worth looking into and will add that to our design, and have the raw input as a backup plan. Processing outputs is something I also took inspiration from previous projects. They used a general heuristic of requiring repeated patterns of the same predicted word before speaker output. I believe that this idea will also help us cut down false positives when the signer is in a transition state. In summary, I was able to narrow down the complexity and logistics of using the ML model for prediction.

My progress is on schedule. I am confident that I will wrap up the planning and design logistics soon.

Next week, I hope to present my ML findings to my partners and start working on the design slides that are focused on the ML model. I will also look into writing a Python script for data collection based on the glove compute that we decide on. I will also need to look into Bluetooth modules for Python.

Posted on

Ria’s Status Report for 2/10/2024

I spent this week doing a ton of research on what compute to use for our design as well as some lower level ideas for the software implementation. These were all things I presented in the team meetings so that we could further our design into a more complete product.

I first looked at the macro level of the project and found gloves that we could iterate on. I found some very basic ones on amazon that were cotton and we decided to use those for now.

Then I researched a few designs for how the compute would be set up. The full fleshed idea would ideally have two boards, one on each hand doing all compute and all interactions with the world. But since this is a really complex design, we decided to break our project up into rapid prototypes with the first one being a single working glove communicating to a computer via bluetooth. Thinking through the processes involved and breaking down the problem into subcomponents helped me better visualize the future of the class and product.

I proposed a few boards: Jetson Nano, Arduino Nano 33, and the Pi Zero. With the information each of us provided to the conversations helped us prepare for next week when we do finalize a design.