Category: Team Status Reports

The two remaining tasks before demo are wireless integration and to record more trials so the accuracy improves from 92% to 95%. Kat has been focusing on wrapping up the wireless integration since the demo, but we know the glove functions well with the cable connection. It’s possible the wireless implementation will add latency to the data classification, but the delay would be minimal due to the small amount of data we’re sending.  More data collection will only improve the model, so there are few risks with continuing.

Software unit testing: Tested the UI/UX with mock results for the entire character set and added buttons (shuffle set, reset, show signs) to improve the user experience.

Hardware unit testing (Teadora): confirmed connectivity/voltage from flex sensors into ADS with a multimeter. Confirmed I2C connectivity (with Kat) by receiving data on the console through cabled connection. Tested multiple conductive pad placements and attachment methods, and confirmed functionality using the suggested libraries for the breakout board and “touched” function.

Overall system test: Tested the full pipeline by collecting data from the pico when the frontend sends a request, sending the data to the model for evaluation, and posting the model’s evaluation to the frontend. We are finding that the accuracy drops significantly when testing the entire system which is what we are currently working to improve.

Currently the biggest challenges are with tuning the ML classification model. During the past week we’ve recorded trials to have training data for the model, but we aren’t currently meeting our live accuracy classification requirements. While we expect this will improve with more trials, it also seems likely that there’s bugs in the model-user interface integration.

The hardware integration is finished: the final board is soldered, the wiring is neat and correct, and the battery connection was attached. We’re still using the cable to power the system for right now, since we’re prioritizing getting the classification accuracy higher before integrating the wireless mode.

We adjusted the features the ML model is being trained on to more accurately distinguish the signs, which means we need to rerecord some of our data. Kat and Nia observed that the IMU rotation (pitch, roll, and heading) were helpful in distinguishing some signs. We also removed the translation mode as it would require exceptional model evaluation which we don’t have yet.

The current risks are related to the time it takes to train the ML model. Touch sensors won’t affect the model since it will just be added as an additional feature so they will likely improve it and the ml model shouldn’t take more than a week. Training should take max 5 mins once we have data, so the main holdup really is getting data. We can improve the model post MVP by tuning training parameters

This past week we assembled a glove to make sure the flex sensor readings worked on the hand.

We added capacitive touch sensors to the system so we could better detect when fingers are touching each other or the palm, which is necessary to distinguish signs like U/V, M/N, etc. Teadora selected capacitive touch sensors instead of force sensing resistors and details the reasons for the decision in this week’s status report. There wasn’t a significant parts cost ($15 at most for conductive tape and a breakout board), but we will need to test the touch sensors and make sure the readings can transfer easily to the ML model. We will display our in progress glove with flex sensors and IMU at the interim demo.

Now that we know the sensors can receive data, we’re going to move quickly towards assembling the glove so we can train the ML model. We’d like to have it assembled soon so we can have something ready for the interim demo in a little over a week. We’re considering adding capacitive touch sensors to the glove (which we’ve already ordered for testing) to make sure we can accurately represent all of the signs.

The most significant risk that could jeopardize the product is how well the sensors  hold up during testing and data collection. We will prioritize keeping the sensors in place and secure during data collection to ensure we collect accurate data across all trials. If the sensors move significantly during data collection, we run the risk of our model incorrectly assigning classes.

This week we submitted the design report, which required making more decisions about the power supply part of the hardware subsystem. There seems to be some discussion of which battery we will use. Teadora’s proposal is a standard 9V battery with a step down converter to power the RPico and IMU at 5V. Katherine’s proposal involves a 3.7V lithium ion battery to power the RPi Pico and IMU. 

This upcoming week we need to test and potentially make some revisions to our design now that our parts have arrived. We will test the parts with what we have so far and determine if we need to make adjustments or not. We also need to order a part to add ADC pins to our RPi Pico, but Katherine has already found the part/s that we can use for that. 

Additional questions:

Section A was written by Katherine

Global factors to be considered for our project are for people without access to computers or technology. This is our main global consideration as our product is geared towards people who want to learn, however it would be hard for someone who does not have a computer since our product would work with a computer in pairing. As of right now, our design is dependent on a local machine to run our algorithm and website. As our product is a prototype we are expecting that it would be significantly more accessible with our algorithm and website uploaded to AWS Cloud. Therefore it would be as simple as having the URL, a computer and the glove. However, as our project specifically interfaces with a computer to display results it would change our product greatly to make it accessible to anyone who does not have one.

 Section B was written by Teadora 

The cultural factors we are considering are beliefs around disability and interpersonal connection. Historically, people with disabilities have been excluded from public life, whether intentionally or through a lack of accessible options. When we were researching existing ASL sensing gloves, a lot of the options focused on recognizing signs and reading them out loud to reduce the need for an interpreter. These design decisions reinforce a cultural belief that speaking out loud is a better form of communication. However, they don’t respond to the reality that sign languages are their own complete languages, separate from spoken language. As a note from the Virginian community college Germana states, “The sentence “I see a big orange cat” would be signed as follows:CAT, ORANGE, BIG, I SEE [1]. In situations where quick and accurate communication with someone who uses ASL is needed, the existing gloves won’t suffice, and in some ways continue to exclude the deaf community from public life. Our design is based in the belief that learning ASL is a better solution because it connects with an existing language and culture. Our project emphasizes specific users: people who are interested in learning ASL, perhaps to communicate with friends or because they’re losing some hearing themselves. Another cultural factor is that ASL has started to increase in cultural capital. It’s seen as impressive for someone to know ASL, and this is emphasized by the inclusion of ASL interpreters in popular entertainment, like concerts and TV shows. Our design recognizes that communicating in ASL is increasingly necessary and desirable, not just to the deaf community but to the broader American public. 

 Section C was written by Nia

The environmental factors for our device mainly relate to the materials used in the hardware and the energy consumption of the device. Because our design uses electronic components such as the Raspberry Pi Pico, an IMU sensor, wiring, and a battery power supply, it contributes to the broader issue of electronic waste. Electronic devices often contain metals, plastics, and batteries that can be harmful to the environment if they are not disposed of properly. When selecting components for our prototype, we considered using available and reusable parts so that they can be repurposed in other projects rather than immediately discarded. 

Another environmental consideration involves the type of battery used to power the system. The team discussed using either a standard 9V battery with or a 3.3V lithium-ion battery. While 9V batteries are easy to obtain, they are often disposable and may contribute more waste if replaced frequently. Rechargeable lithium-ion batteries, on the other hand, can be reused many times and generally produce less battery waste over the lifetime of the device. However, lithium-ion batteries must also be handled and recycled properly because they contain materials that can be hazardous if they end up in landfills [2].

[1] “Provided by ASL Grammar Guide The Academic Center for Excellence 1 ASL Grammar Guide,” 2023. Available: https://germanna.edu/sites/default/files/2023-07/ASL%20Grammar%20Guide%20%28edit%207-24-23%29.pdf

[2] Vermont Department of Environmental Conservation. Lithium-Based Battery Management Fact Sheet. 2020. Available: https://dec.vermont.gov/sites/dec/files/wmp/SolidWaste/Documents/lithium-basedBatteryManagementFactSheet.pdf

 

The biggest risk we are facing right now is our parts not arriving on time. We are behind schedule if they do not come this week as our plan was to assemble the glove and begin testing. However, without hardware we are still making progress working on the communication between Raspberry Pi and software as well as working on the user-interface. This week we will finish the Design document and assemble the glove if our parts arrive. We will also continue implementing the software portions that we do not need hardware for.

 

This week we also formalized parts of the design for the design report, like picking which protocols we’ll use for MVP and final. The glove is based on Raspberry Pi and will use C for the controls. The IMU will use I2C to communicate with the Raspberry Pi. The flex resistors will communicate with the Raspberry Pi through its GPIO, possibly with capacitors to filter out noise. For MVP, the glove will communicate with the computer via USB, but we’d like to do wireless communication for final. We’re considering adjusting the weight requirements based on the batteries we’ve been able to find: if we can’t find a light enough battery to make the glove easy to use, we’ll need to switch to powering the glove with a USB cable or similar, but make the cable long so it’s still ergonomic.

Like last week, there are no significant risks at this stage. We’re in the process of ordering parts, trying to balance getting enough parts early while not blowing our budget if the parts aren’t right. However, we have a pretty good idea of which specific parts we need at this point. We need to look into software choices soon, including how to receive data and use it to build the model.

Part A was written by Nia

Our project aims to improve welfare by providing education to all for learning ASL and increasing fluency. We’re taking into account safety by ensuring batteries that won’t overheat during use.

Part B was written by Teadora

The product solution we’re designing meets needs for social factors. ASL (and other sign languages) are a large part of the deaf community, and it’s common for signed languages to be a child’s first language, especially if people in their family are hard of hearing. Learning sign language can also be useful for older adults since hearing loss can occur later in life, especially as a complication of other health concerns. In recent years, it’s become more common to see sign interpreters in everyday events, like concerts, government press briefings, and other events with live speakers. While sign language interpreting is a complicated skill that takes many years to master, learning sign language is an important way to connect with the deaf community and create accessible environments. Our solution will help people learn ASL and facilitate communication between different cultural groups.

https://www.asha.org/siteassets/ais/ais-comorbidities-and-hearing-loss.pdf

https://signhealth.org.uk/resources/learn-about-deafness/deaf-or-deaf/

 

Part C was written by Katherine

The economic factors are important as there are many versions of our product that are similar being built so it is very important we create a cheap and easily accessible one. Our production should be high quality and also cheaper than other products. As for distribution we want it to be easily accessible which we are demonstrating by creating ourselves with parts we order ourselves.

There are no significant risks at this stage. We need to order parts soon and get moving quickly so we don’t experience delays. We’ll mitigate this risk by ordering from preferred suppliers and ordering enough materials in case something breaks. The things that could jeopardize us are not ordering the right parts or not ordering them soon enough. Therefore we are being very intentional about picking our parts while intended to order parts this upcoming week. 

The plans that we are ready to execute are to order the flex sensors and IMU when we get access, and to decide on the specific model of Rasberry-Pi we are going to use. The features we are looking for are, bluetooth, wifi and lightweight. There are no changes that need to be made or that have been made for the existing design system. We are still using the same design that we presented on our proposal slides.