Category: Team Status Reports

Team’s Status Report 11/16

1. What are the most significant risks that could jeopardize the success of the project?

• Jetson Latency: Once the model hit our accuracy threshold of 70%, we had planned to continue doing improvement as time allowed. However, upon system integration, we have recognized that the Jetson cannot handle a larger model due to latency concerns. This is fine; however, does limit the potential of our project while running on this Jetson model.
• Campus network connectivity: We identified that the Adafruit Feather connects reliably on most WiFi networks but encounters issues on CMU’s network due to its security. We still need to tackle this issue, as we are currently relying on a wired connection between the Jetson and braclet.

2. Were any changes made to the existing design of the system?

• We have decided to remove surprise and disgust from the emotion detection model as replicating a disgusted or surprised face has proven to be too much of a challenge. We had considered this at the beginning of the project given that they emotions were the 2 least important; however, it was only with user testing that we recognized that they are too inaccurate.

3. Provide an updated schedule if changes have occurred:

We remain on schedule and were able to overcome any challenges allowing us to be ready for a successful demo.

Documentation:

• We don’t have pictures of our full system running; however, this can be seen in our demos this week. We will include pictures next week when we have cleaned up the system and integration.

 

Model output coming from jetson to both Adafruit and Webserver:

Model output running on Jetson (shown via SSH):

1. What are the most significant risks that could jeopardize the success of the project?

• Power and Component Integration: For the embedded bracelet, we identified a power issue with the NeoPixel which caused unexpected behavior. Although we ordered a part to address this, any further power discrepancies could delay testing.
• Integration with Jetson We have not received our NVIDIA Jetson yet and still need to complete integration between the three subsystems.

2. Were any changes made to the existing design of the system?

• Power Adjustment for NeoPixel: To resolve the NeoPixel’s voltage requirements, we ordered a new component, modifying the power design for stable operation.
• Real-time API and UI Enhancements: We designed a REST API for real-time data transmission and updates, which adds a dynamic element to the website interface. Upcoming UI enhancements, including a timer and confidence bar, will improve the usability and clarity of the feedback display.

3. Provide an updated schedule if changes have occurred:

• The team remains on schedule with component integration. Kapil will continue with the 3D printed enclosure and communication setup while awaiting the NeoPixel component. Noah will transition to Jetson-based model testing and optimization. Mason’s API testing and UI improvements will align with the next integration phase, ensuring that system components function cohesively.

Photos and Documentation:

• Initial bracelet circuit, highlighting successful haptic motor tests and NeoPixel power issues.

• Images of the real-time labeling function, which demonstrates the model’s current performance and areas needing improvement.

Part A: Global Factors by Mason

The EmotiSense bracelet is developed to help neurodivergent individuals, especially those with autism, better recognize and respond to emotional signals. Autism affects people across the globe, creating challenges in social interactions that are not bound by geographic or cultural lines. The bracelet’s real-time feedback, through simple visual and haptic signals, supports users in understanding the emotions of those they interact with. This tool is particularly valuable because it translates complex emotional cues into clear, intuitive signals, making social interactions more accessible for users.

EmotiSense is designed with global accessibility in mind. It uses components like the Adafruit Feather microcontroller and programming environments such as Python and OpenCV, which are globally available and widely supported. This ensures that the technology can be implemented and maintained anywhere in the world, including in places with limited access to specialized educational resources or psychological support. By improving emotional communication, EmotiSense aims to enhance everyday social interactions for neurodivergent individuals, fostering greater inclusion and improving life quality across diverse communities.

Part B: Cultural Factors by Kapil

Across cultures, emotions are expressed and more importantly interpreted in different ways. For instance, in some cultures, emotional expressions are more subdued, while in others, they are more pronounced. Despite this, we want EmotiSense to recognize emotions without introducing biases on cultural differences.  To achieve this, we are designing the machine learning model to focus on universal emotional cues, rather than specific cultural markers.

One approach we are taking to ensure that the model does not learn differences in emotions across cultures or races is by converting RGB images to grayscale. This eliminates any potential for the model to detect skin tone or other race-related features, which could introduce unintended biases in emotion recognition. By focusing purely on the structural and movement-based aspects of facial expressions, EmotiSense remains a culturally neutral tool that enhances emotional understanding while preventing the reinforcement of stereotypes or cultural biases.

Another way to prevent cultural or racial biases in the EmotiSense emotion recognition model is by ensuring that the training dataset is diverse and well-balanced across different cultural, ethnic, and racial groups. This way we can reduce the likelihood that the model will learn or favor emotional expressions from a specific group.

Part C: Environmental Factors by Noah

While environmental factors aren’t a primary concern in our design of EmotiSense, ensuring that we design a product that is both energy-efficient and sustainable is crucial. Since our system involves real-time emotion recognition, the bracelet and the Jetson need to run efficiently for extended periods without excessive energy consumption. We’re focusing on optimizing battery life to ensure that the bracelet can last for at least four hours of continuous use. To complete this task while maintaining a lightweight bracelet, the design naturally requires us to maintain energy efficiency.

Additionally, a primary concern would be ensuring that our machine-learning model does not utilize excessive energy in its prediction. By maintaining a lightweight model running on a pretty efficient machine, the Nvidia Jetson, we minimize our reliance on computations relying on lengthy computation.

1. What are the most significant risks that could jeopardize the success of the project?

• Facial Recognition and Image Preprocessing: Noah is working on developing the facial recognition and image preprocessing components from scratch, which presents a risk if model performance doesn’t meet expectations or integration issues arise with Jetson or other hardware. However, Noah is mitigating this by researching existing models (RFB-320 and VGG13) and refining preprocessing techniques.
• Website Deployment and User Experience Testing: Mason’s deployment of the website is slightly delayed due to a busy week. If this stretches out, it could impact our ability to conduct timely user experience testing, but Mason plans to have it completed before the break to refocus on the UI and Jetson integration.
• Component Delivery Delays: There is a risk that delays in ordering and receiving key hardware components (such as the microcontroller and potentially a new camera for Noah’s work) could affect the project timeline. We are mitigating this by ensuring orders are placed promptly and making use of simulation tools or placeholder hardware in the meantime.

2. Were any changes made to the existing design of the system?

• Haptic Feedback System: Based on feedback from our design presentation, we decided to move away from binary on/off vibration feedback in the bracelet. Instead, the haptic feedback will now dynamically adjust its intensity. This required the addition of a microcontroller, but it improves the overall functionality of the bracelet.
• Facial Recognition Model: Noah is creating a custom facial recognition model instead of using a pre-built model, as our project goals shifted to developing this from scratch. This adjustment will give us more flexibility and control over the system’s performance, but also adds additional development time.
• Website User System and Database: Mason has made progress on the user system and basic UI elements but is slightly behind due to other commitments. No structural changes have been made to the overall website design, and deployment is still on track.

3. Provide an updated schedule if changes have occurred:

• Bracelet Component and PCB Changes: The decision to remove the PCB from the bracelet and instead use a 3D printed enclosure has been made. This simplifies the next steps and focuses more on the mechanical assembly of the bracelet.
• Website Deployment: Mason’s deployment of the web app is scheduled to be completed before the October break, and the UI/Jetson configuration work will continue afterward.

Photos and Documentation:

• The team is awaiting final feedback from TA and faculty on the circuit design for the bracelet before ordering components.
• Noah’s bounding box and preprocessing work on facial recognition will need further refinement, but initial results are available for review.

1. What are the most significant risks that could jeopardize the success of the
   project? How are these risks being managed? What contingency plans are ready?

The most significant risk at this stage is developing an emotion recognition model that is accurate enough to be useful. Top-of-the-line models currently are nearing the 80% accuracy rate which is somewhat low. Coming within reach of this metric will be crucial to ensuring our use cases are met. This has become a larger concern now that we will be making our own, custom model. This risk is being managed by ensuring that we have backup, pre-existing models that can be averaged with our model in case our base accuracy is too low.

Additionally, user testing has become a concern for us as we do not want to trigger the need for review by the IRB. This prevents us from doing user testing targeted at autistic individuals; however, we can still conduct them on the general population.

2. Were any changes made to the existing design of the system (requirements,
block diagram, system spec, etc)? Why was this change necessary, what costs
does the change incur, and how will these costs be mitigated going forward?

The main change was cited above. We will be making our own computer vision and emotional recognition model. This is primarily with the goal to show our prowess within the area of computer vision and not leveraging too many existing components. This constricts Noah’s time somewhat; however, Mason will be picking up the camera and website integration so that Noah can fully focus on the model.

3. Provide an updated schedule if changes have occurred.

No Schedule changes at this time.

4. This is also the place to put some photos of your progress or to brag about a
the component you got working.

The physical updates are included in our design proposal.

Additionally, here is the dataset we will use to train the first version of the computer vision model (https://www.kaggle.com/datasets/jonathanoheix/face-expression-recognition-dataset).

Part A (Noah Champagne):

Due to the intended audience of EmotiSense being a particularly marginalized group of people, it is our utmost concern that the product can deliver true and accurate readings of emotions. We know that EmotiSense can greatly increase neurodivergent people’s ability to engage socially, and want to help facilitate that benefit to the best of our abilities. But, we care deeply about maintaining the safety of our users and have duly considered that a false reading of a situation could discourage or bring about harm to them. As such we have endeavored to create a system that only offers emotional readings once a very high confidence level has been reached. This will help to ensure the health of those using the product who can confidently use our system.

Additionally, our product serves to increase the welfare of a marginalized group who suffers to a larger extent than the general population. Providing them with this product helps to close the gap in conversational understanding between those with neurodivergence and those without it. This will help to provide this group with more equity in terms of welfare.

Part B (Kapil Krishna):

EmotiSense is highly sensitive to the needs of neurodivergent communities. We want to emphasize the importance of being sensitive to the way neurodivergent communities interact and support one another. The design of EmotiSense aims to be sensitive to this fact and simply enhances self-awareness and social interactions, not combat neurodivergence itself. Additionally. EmotiSense aims to provide support for those who interact with neurodivergent individuals in providing a tool and strategy to facilitate better communication without compromising autonomy. Lastly, we observe that there has been increasing advocacy for technology to empower those with disabilities. EmotiSense aims to align with this trend in reducing social barriers and creating more inclusivity and awareness. With regard to economic organizations, we notice that affordability and availability of the device are key in creating the most impact possible.

Part C (Mason Stark):

Although our primary aims fit better into the previous two categories, Emotisense can still suit significant use cases for economic endeavors. One such use case is customer satisfaction. Many businesses will try to get customer satisfaction data by polling customers online, or even by physical customer satisfaction polling systems (think of those boxes with frown and smile faces in airport bathrooms). However these systems are oftentimes under utilized, and subject to significant sampling biases. Being able to get the emotions of customers in real time could better assist businesses in uncovering accurate satisfaction data. When businesses have access to accurate customer satisfaction data, they can leverage that data to improve their business functionality and profitability. Some specific businesses where emotisense could be deployed include “order at the counter” restaurants, banks, and customer service desks.

1. What are the most significant risks that could jeopardize the success of the
   project? How are these risks being managed? What contingency plans are ready?

Primary risks include Model accuracy, environment control, and speed/user experience. We are planning to try to manage accuracy by experimenting with a few models and by using a composition of models if needed. The environment control is something that we will working on when prototyping the hardware and our physical system. We make a protocol for keeping the environment consistent. User experience and speed will be monitored as we get into the later stages of development.

2. Were any changes made to the existing design of the system (requirements,
block diagram, system spec, etc)? Why was this change necessary, what costs
does the change incur, and how will these costs be mitigated going forward?

No changes at this moment.

3. Provide an updated schedule if changes have occurred.

No Schedule changes at this time.

4.This is also the place to put some photos of your progress or to brag about a
component you got working.

We don’t have many physical updates at this time but they certainly will be included in the next report. The majority of our work this week surrounded prep for and rehearsing our presentation.