Category: Team’s Status Reports

This week, we collected data from each of our team members and integrated the glove with the software system so that the classification can be done in real-time. We found that some letters that we expect to have high accuracy performed poorly in real time. Namely, the letters with movement (J and Z) did not do well. We also found that different letters performed poorly for each of our group members.

After our meeting with Byron and Funmbi, we had a bunch of things to try out. To see if our issue was with the data we had collected or perhaps with the placement of the IMU, we did some data analysis on our existing data as well as moved the IMU to the back of the palm from the wrist. We found that the the gyroscope and accelerometer data for the letters with movement are surprisingly not variable– this means that when we were testing real time, the incoming data was likely different from the training data and thus resulted in poor classification. The data from the IMU on the back of the hand has a 98% accuracy from just the data collected from Rachel; we will be testing it in real time this coming week.

We also found that our system currently can classify about 8.947 gestures per second, but this number will change when we incorporate the audio output. This rate is also too high for actual usage since people cannot sign that fast.

We are also in contact with a couple of ASL users who the office of disabilities connected us with.

We are still on schedule. This week we will work on parsing the letters (not continually classify them). We are also going to take data from a variety of people with different hand sizes, ideally. We will also experiment with capturing data over a time interval to see if that yields better results. We will also be improving the construction of the glove by sewing down the flex sensors more (so that they are more fitting to the glove) and doing a deeper dive into our data and models to understand why they perform the way they do. We will also hopefully be able to meet with the ASL users we are in contact with.

This week, we focused on maintaining the glove as well as moving toward real-time usage of the glove. Some of the connections on our first iteration of the glove came loose as we made gestures that required more movement, so we went back and secured those connections.

We did some analysis on the fake data that we had generated and the real data we collected from the gloves to gain some clarity as to why our models with the real data outperformed out models with the fake data despite having a smaller dataset. The conclusion is that there is a much stronger correlation between the fingers and IMU data in the real data, which makes sense since we were having a hard time modeling the IMU data when we were attempting to mimic the data.

We also added a “random” and “space” category to the classes that our model can output. The “space” class meant to help us eventually allow users to spell words and have those full words outputted as audio (rather than just the individual letters). The “random” class was added to help suppress the audio output as the user transitions between signs. These additions will allow us to move onto using our glove real-time, which is our next step.

We are on schedule and plan on putting the glove together with our software system to use in real-time this week. We are also going to collect data from people with several hand sizes to use to see how well we can get our glove to generalize.

This week we worked on designing the PCB as well as fake data for choosing an ML model. Sophia took lead in designing the PCB since she has the most experience with building circuits and is currently taking a course specifically for PCB design. There were some obstacles in getting the PCB correctly designed, but Sophia was able to find ways around it. Stephanie and Rachel both worked on creating fake data; Stephanie and Rachel researched the sensors and created fake data based off of that using two slightly different approaches. We decided that making these datasets separately would be a good idea so that we have a little more variance in the data and can more confidently choose the model for our use case. For more details on the PCB design or fake model generation, please read our individual status reports.

For the next week, we are planning on having the PCB completely designed and sent to manufacture. We also will test and settle the ML model for our project– Stephanie has already done some preliminary testing and all of our candidate models seem to be sufficient, but more robust testing is necessary. From the software perspective, we are right on schedule with determining the ML model. However, from the circuits perspective, it seems that our order for the parts was not seen until recently and we also do not know how long the PCB board will take to print, so these things could set us back a bit. In addition to solidifying the ML model, while we wait for the parts to arrive, we can work on preparing our design presentation for the following week.

This week, the team worked on planning out a schedule for our project as well as cut down on the scope of our project. Initially, we had wanted to have our glove determine all the letters of the alphabet in ASL as well as a few common phrases, but after closer evaluation and more thorough planning, we determined that we don’t have enough time for such a large scope project.

Another change to our project is that our glove will no longer be wireless. Having a glove that can’t reliably transmit data to compute would be detrimental to our project and having the glove wired to our computer would not be changing the fundamental function of the glove.

We also spent a lot of time thinking about the requirements for our products, originally from an implementation perspective then shifting to a user-experience perspective. After shifting perspectives, we had to redo our research so that our decisions on the requirements were educated.

For next week, we plan on ordering parts (ordering extras for back up in case something breaks) as well as determining the ML model we will use to classify each gesture and creating a dummy dataset to identify which model fits our use case the best. We will also be giving our proposal presentation during this week.