Posted on

Noah’s Status Report for 11/16

This week, I focused on getting the computer vision model running on the Jetson and integrating the webcam instead of my local computer’s native components. It went pretty well, and here is where I am at now.

  • Used SSH to load my model and the facial recognition model onto the Jetson
  • Configured the Jetson with a virtual environment that would allow my model to run. This was the complicated part of integration.
    • The Jetson’s native software is slightly old, so finding compatible packages is quite the challenge.
    • Sourced the appropriate Pytorch, Torchvision, and OpenCV packages
  • Transitioned the model to run on the Jetson GPUs
    • This requires a lot of configuration on the Jetson including downloading the correct CUDA drivers and ensuring compatibility with our chosen version of Pytorch
  • Worked on the output of the model so that it would send requests to both the web server and bracelet with proper formatting.

I am ahead of schedule, and my part of the project is done for the most part. I’ve hit MVP and will be making slight improvements where I see fit.

Goals for next week:

  • Calibrate the model such that it assumes a neutral state when it is not confident in the detected emotion.
  • Add averaging of the detected emotions over the past 3 seconds which should increase the confidence in our predictions.
  • Add an additional output to indicate if a face is even present.
  • Look into compiling the CV model onto onyx – a Jetson specific way of running models – so that there will be lower latency.
Posted on

Mason’s Status Report for 11/16

This week, I focused on integration tasks to prepare the system for the demo. Here’s what I accomplished:

This week’s progress:

  • Jetson and Adafruit UART Integration
    • Worked with Kapil to implement UART communication between the Jetson and Adafruit
    • Helped write, run, and debug the code required to communicate via the Jetson’s UART pins.
    • Created and executed test scripts for this communication, eventually achieving functionality with the output of the model.
  • Model Deployment on Jetson
    • Resolved compatibility challenges related to leveraging the Jetson’s GPU for efficient model execution. 
    • Successfully installed the necessary packages and verified the model running in real-time using a webcam.
  • System Integration
    • Made changes to the model to integrate it with the API and UART communication, ensuring smooth output transmission.
    • Finalized the Jetson setup for the demo: the model now runs locally on the Jetson and transmits outputs as specified in the project write-up.

Goals for Next Week:

  • Collaborate with Kapil on Bluetooth integration between the Jetson and the bracelet/Adafruit.
  • Work with Noah to improve the model’s efficiency and reduce latency.
  • Conduct tests for API latency to ensure real-time responsiveness.
  • Begin user and user experience testing to evaluate the system’s performance and usability.

I’d say we’re on track and I feel good about the state of our project going into the demo on Monday.

UART (bracelet) working alongside API (iPad):

I forgot to capture photos of the model on Jetson/whole system in action.

Posted on

Kapil’s Status Report for 11/16

1. What did you personally accomplish this week on the project?

  • Received the NeoPixel and soldered header pins on.
  • Connected the NeoPixel to the Adafruit Feather.
    • Initially, it did not work, but after debugging a physical connection issue, I successfully resolved the problem and got it operational.
  • Attempted to connect Adafruit Feather to the WiFi
    • Unfortunately, due to persistent connectivity issues, I was unable to establish a connection. To work around this, I identified a program that allows me to push code to the Feather via a wired cable, enabling continued progress without relying on WiFi connectivity.
  • Attended two group meetings focusing on integrating the Jetson.
    • First meeting: encountered issues with securing an SSH connection
    • Second meeting: Successfully established the entire pipeline. The Jetson now communicates with both the iPad display and the embedded bracelet via UART, achieving the intended functionality for the demo.

2. Is your progress on schedule or behind?

  • My progress is on schedule. Despite the WiFi connectivity setbacks, I was able to get the NeoPixel working and successfully integrated the Jetson pipeline. The system is now demo-ready with UART communication functioning as intended.

3. What deliverables do you hope to complete in the next week?

  • Post-demo, my focus will shift to implementing Bluetooth Low Energy (BLE) to replace UART for wireless communication.
  • Conduct timing analysis to ensure the system meets its real-time performance requirements
  • Begin finalizing the 3D-printed enclosure and organizing the circuit for a more polished appearance.

4. Tests conducted and planned for verification and validation:

  • Verification Tests (Subsystem-level):
    • The embedded bracelet must achieve a latency of 150ms. As outlined in the Design Report, I will be using an oscilloscope for this. I will have two latencies one for UART (wired communication), and one for BLE (wireless)
      • testing BLE latency and comparing it to the UART baseline to ensure that BLE does not compromise system responsiveness
    • For the user testing portion, I have two goals
      • Feedback Recognition Accuracy: At least 90% of participants should correctly identify the type of feedback (haptic or visual) associated with specific emotions within 3 seconds of actuation.
      • Error Rate: The system must maintain an error rate of less than 10% for incorrectly signaling emotions, ensuring reliability.
Posted on

Team’s Status Report 11/9

Team’s Status Report 11/9

1. What are the most significant risks that could jeopardize the success of the project?

  • Model Accuracy: While the model achieved 70% accuracy, meeting our threshold, real-time performance with facial recognition is inconsistent. It may require additional training data to improve reliability with variable lighting conditions.
  • Campus network connectivity: We identified that the Adafruit Feather connects reliably on most WiFi networks but encounters issues on CMU’s network due to its security. We will need to tackle this in order to get the Jetson and Adafruit communication working.

2. Were any changes made to the existing design of the system?

  • AJAX Polling and API inclusion for Real-time Updates: We implemented AJAX polling on the website, allowing for continuous updates from the Jetson API. This feature significantly enhances user experience by dynamically displaying real-time data on the website interface.
  • Jetson Ethernet: We have decided to use ethernet for the jetson to connect it to the cmu network.

3. Provide an updated schedule if changes have occurred:

The team is close to being on schedule, though some areas require focused attention this week to stay on track for the upcoming demo. We need to get everything up and running in order to transition fully to testing and enhancement following the demo.

Photos and Documentation:

  • Jetson-to-website integration showcasing successful data transmission and dynamic updates. emotisense@ubuntu is the ssh into the Jetson. The top right terminal is the jetson, the left and bottom right are the website UI and  request history respectively. I also have a video of running a test file on the jetson but video embedded doesn’t work for this post.
Posted on

Noah’s Status Report for 11/9

This week was mostly focused on getting prepared for integration of my CV component into the larger system as a whole. Here are some of the tasks we completed this week:

  • Spent some more time doing a randomized grid search to determine the best hyperparameters for our model.
    • Made the model slightly better up to 70% accuracy which was our goal; however, it doesn’t translate super well to real-time facial recognition.
    • Might need to work on some calibration or use a different dataset
  • Conducted research on the capabilities of the Jetson we chose and how to load our model onto that Jetson so that it would properly utilize the GPU’s
    • I have a pretty good idea of how this is done and will work on it this week once we are done configuring the SSH on the Jetson.

I am on schedule and ready to continue working on system integration this upcoming week!

Goals for next week:

  • Start testing my model using a different dataset which closer mimics the resolution we can get from our webcam.
    • If this isn’t promising, we might advice adding some calibration to each individual person’s face.
  • Download the model to the Jetson and use the webcam / Jetson to run the model instead of my computer
    • This will fulfill my portion of the system integration
    • I would like to transmit the outfit from my model to Mason’s website in order to ensure that we are getting reasonable metrics.

 

Posted on

Mason’s Status Report for 11/9

This week, I aimed to complete various components of our project, focusing on API integration and Jetson setup. Here’s a summary of the tasks I successfully completed:

  • Implemented and tested the website API
    • Finalized the API for real-time data transmission from the Jetson to the website, allowing dynamic updates.
  • Downloaded the NVIDIA SDK and flashed the OS on the Jetson
    • This setup is now complete, enabling us to move forward with device integration and testing.
  • Collaborated with Kapil on Bracelet and Jetson integration
    • We worked together to connect the Bracelet with the Jetson. We were able to get the Adafruit on the network so that the jetson will be able to communicate with it
  • Tested Jetson-to-Website integration
    • Ran scripts on the Jetson that called the API with various test values to validate the website’s response and observed successful updates reflecting test inputs.
  • Enhanced website with AJAX polling
    • Added AJAX polling to enable continuous updates from the API, so the website now dynamically reflects real-time changes in the data.

Overall, we are close to being on schedule, though there is still considerable work to do this week to stay on track for the upcoming demo.

Goals for Next Week:

  • Assist Noah with running the model on the Jetson
    • Work together to optimize the model setup and ensure it operates effectively on the Jetson platform.
  • Finalize the Jetson-Bracelet integration
    • Complete the necessary configurations and testing to enable seamless communication between the Jetson and the Bracelet.
  • Design and initiate latency tests for the system
    • Develop latency tests to measure and optimize response times across the integrated system, ensuring it meets performance requirements for the demo.
Posted on

Kapil’s Status Report for 11/9

1. What did you personally accomplish this week on the project?

  • This week, I confirmed that the NeoPixel part I ordered last week is on its way. Once it arrives, I plan to finalize and complete the circuit assembly.
  • In the meantime, I shifted my focus to integrating the components with the Jetson, which is an essential step for the full system setup. To facilitate this, I participated in a group meeting where we worked on setting up the Jetson and coordinating its communication with other system components.
  • During integration, I identified an issue with the Adafruit Feather’s connectivity. While it works smoothly on my local WiFi, it faces challenges when trying to connect to CMU’s network. This is a critical insight that will need addressing to ensure reliable operation in our intended environment.

2. Is your progress on schedule or behind?

  • My progress is slightly behind schedule.  I wanted to have finished my circuit and start working on  streamlining the design but am still waiting for the NeoPixel to arrive.

3. What deliverables do you hope to complete in the next week?

  • Next week, I hope to have the NeoPixel part in hand and complete the circuit assembly.
  • I aim to continue working on and ideally finalize the integration between the Adafruit Feather and the Jetson, addressing the WiFi connectivity issue to ensure seamless communication.
  • If the circuit is completed, I will run initial tests to confirm that the system operates as expected and adjust any configurations as necessary.
Posted on

Team’s Status Report 11/2

1. What are the most significant risks that could jeopardize the success of the project?

  • Power and Component Integration: For the embedded bracelet, we identified a power issue with the NeoPixel which caused unexpected behavior. Although we ordered a part to address this, any further power discrepancies could delay testing.
  • Integration with Jetson We have not received our NVIDIA Jetson yet and still need to complete integration between the three subsystems.

2. Were any changes made to the existing design of the system?

  • Power Adjustment for NeoPixel: To resolve the NeoPixel’s voltage requirements, we ordered a new component, modifying the power design for stable operation.
  • Real-time API and UI Enhancements: We designed a REST API for real-time data transmission and updates, which adds a dynamic element to the website interface. Upcoming UI enhancements, including a timer and confidence bar, will improve the usability and clarity of the feedback display.

3. Provide an updated schedule if changes have occurred:

  • The team remains on schedule with component integration. Kapil will continue with the 3D printed enclosure and communication setup while awaiting the NeoPixel component. Noah will transition to Jetson-based model testing and optimization. Mason’s API testing and UI improvements will align with the next integration phase, ensuring that system components function cohesively.

Photos and Documentation:

  • Initial bracelet circuit, highlighting successful haptic motor tests and NeoPixel power issues.

  • Images of the real-time labeling function, which demonstrates the model’s current performance and areas needing improvement.

Posted on

Noah’s Status Report for 11/2

I wanted to fully integrate all the computer vision components this week which was mostly a success! Here are some of the following tasks I completed:

  • Continued to create and test my new facial recognition model
    • This has been kind of a side project to increase the accuracy of the emotion detector in the long run.
    • It shows significant benefits, but I’ve decided to hold off for now until we can start with system integration
  • Trained a new model with over 200 epochs and 5 convolution layers
    • This model reaches an accuracy of 68.8% which is so close to the threshold we set out in the design report.
    • I believe we can make it to 70%, but I’ll need to dedicate an instance to training for over 6 hours likely.
      • Also need to reduce overfitting again
  • Integrated the model to produce labels in real-time
    • I’ve included some pictures below to show that it is working!
    • It has a noticeable bias to say that you are sad which is a bit of an issue
      • I’m still working to discover why this is.

Being mostly done with the model, I am on schedule and ready to work on system integration this upcoming week!

Goals for next week:

  • Continue to update the hyperparameters of the model to obtain the best accuracy as possible.
  • Download the model to the Jetson and use the webcam / Jetson to run the model instead of my computer
    • This will fulfill my portion of the system integration
    • I would like to transmit the outfit from my model to Mason’s website in order to ensure that we are getting reasonable metrics.

 

Posted on

Kapil’s Status Report for 11/2

1. What did you personally accomplish this week on the project?

  • This week, I assembled the preliminary circuit for the bracelet, bringing together the main components to start testing their functionality as a system. I successfully got the haptic feedback motor to operate with different vibration modes, confirming that the motor can provide the varied feedback required by the design.

  • During testing, I identified an issue with the NeoPixel’s power supply. The NeoPixel requires 5V for both power and the data line, but the Adafruit Feather can only supply 3.3V. Although we initially thought this voltage difference wouldn’t affect performance, I observed unexpected behavior in the NeoPixel’s operation. To address this, I ordered a part specifically designed to handle the voltage issue, ensuring consistent functionality for the NeoPixel.

https://www.adafruit.com/product/2945

  • While waiting for this part, I’m preparing to start working on the 3D printed enclosure, finalizing the design so it will be ready for printing. I also plan to begin establishing the communication protocols between the Adafruit Feather and the Jetson, which will be crucial for our system’s data flow.

2. Is your progress on schedule or behind?

  • My progress is on schedule, despite the NeoPixel power issue. The haptic motor testing was successful, and I’ve already ordered the part needed to resolve the NeoPixel issue. I’ll stay productive by focusing on the enclosure design and the Feather-Jetson communication setup while waiting for the part to arrive.

3. What deliverables do you hope to complete in the next week?

  • Next week, I plan to finalize the 3D printed enclosure design and begin printing, ensuring the bracelet housing is ready for the final assembly.
  • I’ll also aim to complete the initial setup for communication between the Feather and Jetson, which will allow data transmission and control commands to be integrated into the bracelet.
  • Once the new part arrives, I’ll revisit the NeoPixel setup to confirm that the circuit operates smoothly with the correct voltage levels.