Posted on

Noah’s Status Report for 10/5

My goals before the October break are to make major strides in the feature recognition component and image preprocessing component of the project. Having changed out project goals to make our model from scratch, I am putting additional efforts into making sure our facial recognition component is fully fleshed out. Here are some of the things that I did this week that constitute my effort toward this goal:

  • Creating a facial recognition architecture to run quickly
    • Draws a bounding box around the face of the conversational partner
    • Currently working on potentially isolating the eyebrows, lips, and eyes of the partner as these are seen to be the most indicative of the emotion
      • Likely to try this programmed-in feature recognition as well as getting the model to recognize these patterns by itself.
  • Connected the camera to my PC which with the help of OpenCV enabled me to process video in real-time
    • Allows for the initial attempt to draw the bounding box, although it was not very reliable as I believe my image pre-processing is incorrect with the size and resolution of the images
      • This is my next goal for early this week
  • Look into both the RFB-320 facial recognition model and VGG13 emotional recognition model
    • I believe I will be creating variants of these models

Goals for next week:

  • Potentially submit an order for a more reliable Logitech camera as this one has been giving me issues
  • Keep working on the facial recognition and preprocessing to ensure the emotion recognition model can grow properly on top of that
Posted on

Mason’s Status Report for 10/5

Before the start of break I hope to have the website deployed so I can focus on the UI design and Jetson configuration afterward. This week I spent time working on the web app’s front page and user system. I honestly didn’t accomplish a ton this week, as I had a very busy schedule with other classes and assignments. I’m on track for finishing the deployment this week and being able to switch over my work to focus on user experience after break.

Specific tasks completed:

  • Worked on user model for database.
  • Added basic UI elements for output display
  • Worked on rest API format integration with AJAX

Tasks for this week/after break:

  • TCP Jetson Communication
  • EC2 Deployment
  • Frontend Enhancement
  • Latency testing and user experience testing.
Posted on

Team Status Report for 10/5

1. What are the most significant risks that could jeopardize the success of the project?

  • Facial Recognition and Image Preprocessing: Noah is working on developing the facial recognition and image preprocessing components from scratch, which presents a risk if model performance doesn’t meet expectations or integration issues arise with Jetson or other hardware. However, Noah is mitigating this by researching existing models (RFB-320 and VGG13) and refining preprocessing techniques.
  • Website Deployment and User Experience Testing: Mason’s deployment of the website is slightly delayed due to a busy week. If this stretches out, it could impact our ability to conduct timely user experience testing, but Mason plans to have it completed before the break to refocus on the UI and Jetson integration.
  • Component Delivery Delays: There is a risk that delays in ordering and receiving key hardware components (such as the microcontroller and potentially a new camera for Noah’s work) could affect the project timeline. We are mitigating this by ensuring orders are placed promptly and making use of simulation tools or placeholder hardware in the meantime.

2. Were any changes made to the existing design of the system?

  • Haptic Feedback System: Based on feedback from our design presentation, we decided to move away from binary on/off vibration feedback in the bracelet. Instead, the haptic feedback will now dynamically adjust its intensity. This required the addition of a microcontroller, but it improves the overall functionality of the bracelet.
  • Facial Recognition Model: Noah is creating a custom facial recognition model instead of using a pre-built model, as our project goals shifted to developing this from scratch. This adjustment will give us more flexibility and control over the system’s performance, but also adds additional development time.
  • Website User System and Database: Mason has made progress on the user system and basic UI elements but is slightly behind due to other commitments. No structural changes have been made to the overall website design, and deployment is still on track.

3. Provide an updated schedule if changes have occurred:

  • Bracelet Component and PCB Changes: The decision to remove the PCB from the bracelet and instead use a 3D printed enclosure has been made. This simplifies the next steps and focuses more on the mechanical assembly of the bracelet.
  • Website Deployment: Mason’s deployment of the web app is scheduled to be completed before the October break, and the UI/Jetson configuration work will continue afterward.

Photos and Documentation:

  • The team is awaiting final feedback from TA and faculty on the circuit design for the bracelet before ordering components.
  • Noah’s bounding box and preprocessing work on facial recognition will need further refinement, but initial results are available for review.
Posted on

Kapil’s Status Report for 9/28

1. What did you personally accomplish this week on the project?

  • I attended two team meetings and participated in a work session on Monday, as well as a TA + professor meeting on Wednesday, where I received valuable feedback on my progress and design considerations.
  • I spent most of the week working on a prototype design for the bracelet components of our system. Initially, I aimed for a simple design connecting components directly to the Jetson Nano, but I encountered challenges that made me realize the need for a microcontroller to avoid compatibility issues.

  • https://www.adafruit.com/product/1643
  • https://www.adafruit.com/product/2305
  • https://www.adafruit.com/product/1201
  • I explored several microcontroller options, including the STM32, ESP32, and Adafruit Feather. After discussing with the group TA, I chose the Adafruit Feather due to its seamless integration with the Adafruit components I am using for the bracelet.

  • I created two circuit diagrams: one for a wired connection and one for a wireless setup, ensuring both options are viable based on the needs of the project.

  • Next steps include finalizing the circuit and placing a purchase order for the components. I also plan to start the PCB design in parallel with these tasks.

2. Is your progress on schedule or behind?

  • My progress is on schedule. While there was a slight delay in finalizing the microcontroller decision, I quickly addressed it, and now I’m moving forward with both circuit and PCB design, which are key deliverables for the upcoming weeks.

3. What deliverables do you hope to complete in the next week?

  • Finalize the circuit design for the bracelet and place an order for the required components.
  • Begin the PCB design process, working through layout and pin configurations for seamless integration with the bracelet’s components.
  • Prepare to test both wired and wireless configurations once the parts arrive.
Posted on

Mason’s Status Report for 09/28

I spend this week working on the development and construction of our web application. This involved migrating an existing application I had built in a previous class. I spent a lot of my time figuring out how to transform my previous application from my old version, and getting it work again. Here are some of the specific items I finished:

  • Coordinated the database migration with SQLite
    • Ran into errors associated with the database coordination and creation of new model structures
    • Removed old models and forms from application, worked on new post form
      • post form to be used in TCP communication with jetson
  • Inclusion of AJAX in application
  • Added to design presentation
    • Diagramed Website and AJAX coordination with Jetson
    • Outlined user testing for website experience
  • Gave input on model design considerations
  • Followed up on the 2 team meetings I was sick and missed
  • Looked into TCP communication with Jetson

Goals for next week:

  • Complete POST request model and AJAX coordination
  • Deploy website on AWS and purchase a domain
  • Begin prototyping UI and get feedback on UI from teammates
Posted on

Noah’s Status Report for 9/28

For me, this week was mainly focused on fleshing out the computer vision component of our project. Originally, we had planned to utilize an existing model and made modifications to that model as well as integrating it into our system. However, after further discussion and research, we decided we would learn more by creating our own model. This would allow us to showcase our prowess, while also achieving viable results. Here is a bulleted list of the tasks completed:

  • Conducted research on existing computer vision models
    • Noted the methods varying projects attempted and what level of success was achieved with each
    • Chose to utilize a convolutional neural network within Pytorch
      • This will allow us to run our model using CUDA software on the Jetson so that processing is in real-time
  • Created the final design presentation
    • Conducted research on existing facial recognition models to understand the limits of our project in terms of latency and accuracy
    • Set an accuracy goal of 70-75%
  • Lead discussion with my teammates regarding my choices for the model allowing for feedback
  • Updated my part of the schedule to reflect the start of my work on our computer vision component
  • Attended 2 team meetings to work together with teammates
  • Found 2 existing datasets that we might use for our machine-learning model

I’d say that we are on schedule.

Goals for next week:

  • Submit orders for our hardware components to ensure they come on time
  • Get more feedback from our peers during the design presentation and make reasonable changes if any problems are noticed
Posted on

Team Status Report for 9/28

  1. What are the most significant risks that could jeopardize the success of the
    project? How are these risks being managed? What contingency plans are ready?

The most significant risk at this stage is developing an emotion recognition model that is accurate enough to be useful. Top-of-the-line models currently are nearing the 80% accuracy rate which is somewhat low. Coming within reach of this metric will be crucial to ensuring our use cases are met. This has become a larger concern now that we will be making our own, custom model. This risk is being managed by ensuring that we have backup, pre-existing models that can be averaged with our model in case our base accuracy is too low.

Additionally, user testing has become a concern for us as we do not want to trigger the need for review by the IRB. This prevents us from doing user testing targeted at autistic individuals; however, we can still conduct them on the general population.

2. Were any changes made to the existing design of the system (requirements,block diagram, system spec, etc)? Why was this change necessary, what costsdoes the change incur, and how will these costs be mitigated going forward?

The main change was cited above. We will be making our own computer vision and emotional recognition model. This is primarily with the goal to show our prowess within the area of computer vision and not leveraging too many existing components. This constricts Noah’s time somewhat; however, Mason will be picking up the camera and website integration so that Noah can fully focus on the model.

3. Provide an updated schedule if changes have occurred.

No Schedule changes at this time.

4. This is also the place to put some photos of your progress or to brag about athe component you got working.

The physical updates are included in our design proposal.

Additionally, here is the dataset we will use to train the first version of the computer vision model (https://www.kaggle.com/datasets/jonathanoheix/face-expression-recognition-dataset).

Part A (Noah Champagne):

Due to the intended audience of EmotiSense being a particularly marginalized group of people, it is our utmost concern that the product can deliver true and accurate readings of emotions. We know that EmotiSense can greatly increase neurodivergent people’s ability to engage socially, and want to help facilitate that benefit to the best of our abilities. But, we care deeply about maintaining the safety of our users and have duly considered that a false reading of a situation could discourage or bring about harm to them. As such we have endeavored to create a system that only offers emotional readings once a very high confidence level has been reached. This will help to ensure the health of those using the product who can confidently use our system.

Additionally, our product serves to increase the welfare of a marginalized group who suffers to a larger extent than the general population. Providing them with this product helps to close the gap in conversational understanding between those with neurodivergence and those without it. This will help to provide this group with more equity in terms of welfare.

Part B (Kapil Krishna):

EmotiSense is highly sensitive to the needs of neurodivergent communities. We want to emphasize the importance of being sensitive to the way neurodivergent communities interact and support one another. The design of EmotiSense aims to be sensitive to this fact and simply enhances self-awareness and social interactions, not combat neurodivergence itself. Additionally. EmotiSense aims to provide support for those who interact with neurodivergent individuals in providing a tool and strategy to facilitate better communication without compromising autonomy. Lastly, we observe that there has been increasing advocacy for technology to empower those with disabilities. EmotiSense aims to align with this trend in reducing social barriers and creating more inclusivity and awareness. With regard to economic organizations, we notice that affordability and availability of the device are key in creating the most impact possible.

Part C (Mason Stark):

Although our primary aims fit better into the previous two categories, Emotisense can still suit significant use cases for economic endeavors. One such use case is customer satisfaction. Many businesses will try to get customer satisfaction data by polling customers online, or even by physical customer satisfaction polling systems (think of those boxes with frown and smile faces in airport bathrooms). However these systems are oftentimes under utilized, and subject to significant sampling biases. Being able to get the emotions of customers in real time could better assist businesses in uncovering accurate satisfaction data. When businesses have access to accurate customer satisfaction data, they can leverage that data to improve their business functionality and profitability. Some specific businesses where emotisense could be deployed include “order at the counter” restaurants, banks, and customer service desks.

Posted on

Mason’s Status Report for 9/21

The team’s focus this week was primarily delivering the presentation. We also made significant devisions about deadlines and goals for the coming weeks. Here’s some of what I worked on:

  • Making the proposal presentation and preparing the delivery
    • Researched industry standards for latency and speed with REST APIs
    • Researched wired communication speeds
    • Identified Model accuracy guidance
  • Met up with and brainstormed together with teammates
  • Build off of TA and instructor feedback
  • Gave detailed feedback on other presentations
  • Researched AJAX implementations

Upcoming goals for the week:

  • Finalize and order any hardware needed
  • Work on website, get early version running locally
  • Get more TA and professor feedback during the week
  • Identify our wired communication protocol from camera to Jetson
Posted on

Kapil’s Status Report for 9/21

Progress

  • This week, I began the preliminary design for the PCB, focusing on the use case requirements and what hardware is best for meeting these requirements. I also narrowed down on how the individuals components of the embedded device will communicate with each other. I also communicated budget requirements with the team to ensure we are choosing the best hardware devices for the application.
  • I reviewed design tools like SolidWorks to assess which would best suit our project. I also researched the pin configurations and layout considerations needed for our facial recognition system.
  • Additionally, I attended all team meetings and participated in discussions regarding hardware latency and accuracy goals.
  • I also supported the team during our proposal presentation by giving input on how hardware and design choices will impact latency and overall performance.

Schedule

  • Currently, my progress is on schedule. The PCB design phase will official begin next week according to the Gantt chart but I’m feeling more confident given the initial research I’ve conducted this past week. However, I anticipate that the next phase, involving component orders and integration testing, will need close attention to stay on track.

Upcoming Deliverables

  • Finalize the component choices for the PCB.
  • Complete the initial PCB layout and design considerations.
  • Submit component orders once the design and budget are confirmed.
Posted on

Team Status Report for 9/21

  1. What are the most significant risks that could jeopardize the success of theproject? How are these risks being managed? What contingency plans are ready?

Primary risks include Model accuracy, environment control, and speed/user experience. We are planning to try to manage accuracy by experimenting with a few models and by using a composition of models if needed. The environment control is something that we will working on when prototyping the hardware and our physical system. We make a protocol for keeping the environment consistent. User experience and speed will be monitored as we get into the later stages of development.

2. Were any changes made to the existing design of the system (requirements,block diagram, system spec, etc)? Why was this change necessary, what costsdoes the change incur, and how will these costs be mitigated going forward?

No changes at this moment.

3. Provide an updated schedule if changes have occurred.

No Schedule changes at this time.

4.This is also the place to put some photos of your progress or to brag about acomponent you got working.

We don’t have many physical updates at this time but they certainly will be included in the next report. The majority of our work this week surrounded prep for and rehearsing our presentation.