Category: Noah’s Status Reports

Given that integration into our entire system went well, I spent the last week tidying things up and getting some user reviews. The following details some of the tasks I did this past week and over the break:

• • Added averaging of the detected emotions over the past 3 seconds
    • Decreases frequent buzzing in the braclet
    • Emotions are much more regulated and only change when the model is sufficiently confident.
  • Coordinated time with some friends to use our system and got user feedback
    • Tended to be pretty possible recognizing that this is only a first iteration
  • Started working on the poster and taking pictures of our final system.
    • Waiting on the bracelet’s completion, but it’s not a blocker.

List all unit tests and overall system test carried out for experimentation of the system. List any findings and design changes made from your analysis of test results and other data obtained from the experimentation.

*

I am done with my part of the project, and will mainly be working on the reports and such going forward.

Goals for next week:

• Finish the poster, final report, and presentation!

This week, I focused on getting the computer vision model running on the Jetson and integrating the webcam instead of my local computer’s native components. It went pretty well, and here is where I am at now.

• Used SSH to load my model and the facial recognition model onto the Jetson
• Configured the Jetson with a virtual environment that would allow my model to run. This was the complicated part of integration.
  • The Jetson’s native software is slightly old, so finding compatible packages is quite the challenge.
  • Sourced the appropriate Pytorch, Torchvision, and OpenCV packages
• Transitioned the model to run on the Jetson GPUs
  • This requires a lot of configuration on the Jetson including downloading the correct CUDA drivers and ensuring compatibility with our chosen version of Pytorch
• Worked on the output of the model so that it would send requests to both the web server and bracelet with proper formatting.

I am ahead of schedule, and my part of the project is done for the most part. I’ve hit MVP and will be making slight improvements where I see fit.

Goals for next week:

• Calibrate the model such that it assumes a neutral state when it is not confident in the detected emotion.
• Add averaging of the detected emotions over the past 3 seconds which should increase the confidence in our predictions.
• Add an additional output to indicate if a face is even present.
• Look into compiling the CV model onto onyx – a Jetson specific way of running models – so that there will be lower latency.

This week was mostly focused on getting prepared for integration of my CV component into the larger system as a whole. Here are some of the tasks we completed this week:

• Spent some more time doing a randomized grid search to determine the best hyperparameters for our model.
  • Made the model slightly better up to 70% accuracy which was our goal; however, it doesn’t translate super well to real-time facial recognition.
  • Might need to work on some calibration or use a different dataset
• Conducted research on the capabilities of the Jetson we chose and how to load our model onto that Jetson so that it would properly utilize the GPU’s
  • I have a pretty good idea of how this is done and will work on it this week once we are done configuring the SSH on the Jetson.

I am on schedule and ready to continue working on system integration this upcoming week!

Goals for next week:

• Start testing my model using a different dataset which closer mimics the resolution we can get from our webcam.
  • If this isn’t promising, we might advice adding some calibration to each individual person’s face.
• Download the model to the Jetson and use the webcam / Jetson to run the model instead of my computer
  • This will fulfill my portion of the system integration
  • I would like to transmit the outfit from my model to Mason’s website in order to ensure that we are getting reasonable metrics.

 

I wanted to fully integrate all the computer vision components this week which was mostly a success! Here are some of the following tasks I completed:

• Continued to create and test my new facial recognition model
  • This has been kind of a side project to increase the accuracy of the emotion detector in the long run.
  • It shows significant benefits, but I’ve decided to hold off for now until we can start with system integration
• Trained a new model with over 200 epochs and 5 convolution layers
  • This model reaches an accuracy of 68.8% which is so close to the threshold we set out in the design report.
  • I believe we can make it to 70%, but I’ll need to dedicate an instance to training for over 6 hours likely.
    • Also need to reduce overfitting again
• Integrated the model to produce labels in real-time
  • I’ve included some pictures below to show that it is working!
  • It has a noticeable bias to say that you are sad which is a bit of an issue
    • I’m still working to discover why this is.

Being mostly done with the model, I am on schedule and ready to work on system integration this upcoming week!

Goals for next week:

• Continue to update the hyperparameters of the model to obtain the best accuracy as possible.
• Download the model to the Jetson and use the webcam / Jetson to run the model instead of my computer
  • This will fulfill my portion of the system integration
  • I would like to transmit the outfit from my model to Mason’s website in order to ensure that we are getting reasonable metrics.

 

These past 2 weeks, I focused heavily on advancing the facial recognition and emotion detection components of the project. Given that we decided to create our own emotion detection model, I wanted to get a head start on this task to ensure that we could reach accuracy numbers that are high enough for our project to be successful. My primary accomplishments these couple weeks included:

• Leveraged the Haar-Cascade facial recognition model to act as a lightweight solution to detect faces in real-time
  • Integrated with open-CV to allow for real-time processing and emotion recognition in the near future
• Created the first iterations of the emotion detection model
  • Started testing with the FER-2013 dataset to classify emotions based on facial features.
  • Created a training loop using PyTorch. Learning about and implementing features like cyclic learning rates and gradient clipping to stabilize training and prevent overfitting.
    • The model is starting to show improvements reaching a test accuracy of over 65%. This is already at the acceptable range for our project; however, I think we have enough time to improve the model up to 70%.
    • The model is still pretty lightweight using 5 convolution layers; however, I am considering simplifying a little bit to keep it very lightweight.
• Significantly improved image preprocessing with various transformations which have kept our model from overfitting.

I am now ahead of schedule given that we have a usable emotion recognition model much before the deadline.

Goals for next week:

• Continue to update the hyperparameters of the model to obtain the best accuracy as possible.
• Download the model to my local machine which should allow me to integrate with OpenCV and test the recognition capabilities in real-time.
  • Then, Download our model onto the Nvidia Jetson to ensure that it will run in real-time as quick as we want it.
• After this, I want to experiment with some other facial recognition models and boundary boxes that might make our system more versatile.

 

My goals before the October break are to make major strides in the feature recognition component and image preprocessing component of the project. Having changed out project goals to make our model from scratch, I am putting additional efforts into making sure our facial recognition component is fully fleshed out. Here are some of the things that I did this week that constitute my effort toward this goal:

• Creating a facial recognition architecture to run quickly
  • Draws a bounding box around the face of the conversational partner
  • Currently working on potentially isolating the eyebrows, lips, and eyes of the partner as these are seen to be the most indicative of the emotion
    • Likely to try this programmed-in feature recognition as well as getting the model to recognize these patterns by itself.
• Connected the camera to my PC which with the help of OpenCV enabled me to process video in real-time
  • Allows for the initial attempt to draw the bounding box, although it was not very reliable as I believe my image pre-processing is incorrect with the size and resolution of the images
    • This is my next goal for early this week
• Look into both the RFB-320 facial recognition model and VGG13 emotional recognition model
  • I believe I will be creating variants of these models

Goals for next week:

• Potentially submit an order for a more reliable Logitech camera as this one has been giving me issues
• Keep working on the facial recognition and preprocessing to ensure the emotion recognition model can grow properly on top of that

For me, this week was mainly focused on fleshing out the computer vision component of our project. Originally, we had planned to utilize an existing model and made modifications to that model as well as integrating it into our system. However, after further discussion and research, we decided we would learn more by creating our own model. This would allow us to showcase our prowess, while also achieving viable results. Here is a bulleted list of the tasks completed:

• Conducted research on existing computer vision models
  • Noted the methods varying projects attempted and what level of success was achieved with each
  • Chose to utilize a convolutional neural network within Pytorch
    • This will allow us to run our model using CUDA software on the Jetson so that processing is in real-time
• Created the final design presentation
  • Conducted research on existing facial recognition models to understand the limits of our project in terms of latency and accuracy
  • Set an accuracy goal of 70-75%
• Lead discussion with my teammates regarding my choices for the model allowing for feedback
• Updated my part of the schedule to reflect the start of my work on our computer vision component
• Attended 2 team meetings to work together with teammates
• Found 2 existing datasets that we might use for our machine-learning model

I’d say that we are on schedule.

Goals for next week:

• Submit orders for our hardware components to ensure they come on time
• Get more feedback from our peers during the design presentation and make reasonable changes if any problems are noticed

This week was mainly focused on our proposal presentation, finalizing our respective roles within the project, and getting a start on the computer vision component of our project. Here is a bulleted list of the taste completed:

• Created the proposal presentation
  • Conducted research on existing facial recognition models to understand the limits of our project in terms of latency and accuracy
  • Finalized our hardware usage and found latency bounds on our wired and wireless connections
    • Set an accuracy goal of 70-75%
• Gave the presentation to the class and updated our goals based on responses from the professors, TAs, and classmates
• Finalized the schedule and division of work for the project
• Attended 3 team meetings to work together with teammates
• Attended proposal meetings for other teams and gave feedback
• Found 2 existing datasets that we might use for our machine-learning model
• Found an existing facial recognition model that we may base ours of

I’d say that we are on schedule.

Goals for next week:

• Submit orders for our hardware components to ensure they come on time
• Finalize which facial recognition models we will use and what additions, modifications, and integrations will be used to make it our own
• Get more feedback from our peers and make reasonable changes if any problems are noticed