Posted on

Yasser’s Status Report for 04/29/23

This week, after the team decided that it would be best to use dlib’s facial landmark model on all of our features to increase the fps and decrease the amount of false negatives (as a result of the low fps), I changed the model for eye tracking. After testing, the accuracy is comparable to the previous model (cnn-facial-landmark).

Progress is behind schedule. I was supposed to get done with migrating the features to Dlib by today but I only got eye tracking done. I plan on migrating the dlib model to the other features (mouth detection, head pose, and eyes closing detection) by tomorrow and start testing these features immediately after.

By next week, I hope to have testing done for all of the features, as well as the poster presentation and the demo video.

Posted on

Elinora’s Status Report for 04/29

This past week, I worked on finishing up the slides for our final presentation with my teammates and started working on our poster with Sirisha. For the web app, I cleaned up the formatting on the logs page, began looking into how to send data from the Python code on the Jetson to our Firestore, and set up authentication using Google OAuth (through FirebaseUI). I also spent some time with Sirisha trying to get necessary libraries for the CV code installed on the Jetson. 

My progress is behind in terms of testing because we have had difficulties with integrating the system, but we figured out the issue with installing the libraries and now only need the larger microSD card to be able to run the code. For the web app and project documentation tasks, my progress is on track. 

This coming week is our final week to work on the project. I hope to accomplish unit testing for the accelerometer in the car (as well as other units when the system is fully integrated) and full-system tests. Additionally, I will work with my teammates to complete the poster, final report, final video, and finalize/test the setup for our public demo. I also have a stretch goal of adding visual graphs for displaying the metrics on the web app. For now, we have decided that the logs page would be adequate for our purposes and that our time should be more focused on fully integrating and testing the system.

Posted on

Team Status Report for 04/29/23

The biggest change to our system at the moment is converting all of our features over to DLib.  We are trying to make this change in time because doing so would increase our accuracies for testing in nonideal conditions.  The costs incurred are that it will take time to make the switch as well as retest with the new model, and we don’t have a definitive answer as to how this change will affect the current accuracies we have.  However, we believe it is worth trying for the increase to the frame rate and more accurate head pose estimation.  We will also continue testing what we currently have in the case that the change can’t be made in time so we still have a good final product.

Another risk we are currently having is related to integrating the full CV code onto the Jetson Nano. We were having issues installing tensorflow onto the Jetson, but the current problem is that the microSD card in the Jetson Nano is too small for our code and needed libraries. We have ordered a 128gb microSD card that will arrive on 04/30, at which time we will set up the Jetson again with the new SD card and restart the process of installing the necessary libraries and be able to run our full code on the Jetson directly.

We have pushed entire system testing on the Jetson again (to this upcoming week) because of issues we are having with integrating the CV subsystem code onto the Jetson Nano. We have also added tasks for converting head pose and mouth detection from CNN facial landmarks to DLib. 

Unit Test Notes
Accelerometer (in Hamerschlag) Acceleration data fluctuates significantly when the accelerometer is stationary – inaccurate acceleration measures also leads to inaccurate speed estimations. Need to still test in a moving car to determine feasibility of this unit. 
Head pose estimation (CNN facial landmarks) When tested pre-integration, head pose worked very accurately. However, the lower fps caused by integration makes the head pose ineffective. This led to the design change to convert Head pose and mouth detection from CNN to DLib because the use of the same algorithm for all of our components would decrease runtime and increase fps. Will need to retest after converting to DLib. 
Mouth detection (CNN facial landmarks) Worked accurately for both pre and post integration tests. Will need to retest after converting to DLib. 
Eye-tracking Worked accurately pre and post integration
Blink detection Worked accurately pre-integration, but fps was too low for accurate blink detection after integration. We are expecting performance to improve after 
Connecting to WiFi hotspot in car and ssh-ing in Set up the Jetson to auto-connect to Elinora’s hotspot and checked that it was connected in the car. Did this 3x successfully in the car. There is a long connection time ~ 5 min and not waiting previously led us to think that the connection wasn’t working. 
Retrieving data from Firestore (in React) and displaying on the web app Created a test collection on Firestore and confirmed that the data from that test collection was retrieved and shown on the web app in table format. 
Hosting the web app Able to connect remotely from all 3 of our computers
Firebase authentication Able to create 3 accounts and log in. 
Audio feedback Tested the speaker separately and then connected it to a laptop to play audio feedback integrated into the classification code. Audio feedback was played virtually instantaneously (passing our latency requirement)

 

Posted on

Sirisha’s Status Report for 04/29/23

I spent the beginning of the week working on the final presentation slides and practicing the presentation since it was my turn to present.  I also worked on debugging the issue with setting up the Jetson in the car due to the personal hotspot not wanting to connect, and Elinora and I were able to fix this issue.  I also helped Yasser with getting the code to properly run on the Jetson, but it was taking much longer than predicted due to the libraries we needed to download and the lack of storage.  The most recent update with this is that we will need to get an SD card with more storage if we want to be able to run the code on the Jetson.  I also worked on the website and updated it for the demo.  Since all of the final assignment are due soon, I started to work on the final poster as well.

We are a little behind at the moment because we had another setback due to the discovery that we didn’t have enough storage for all of our code.  It was unfortunate that we had to find this out so late, especially because we started to remedy this by deleting unneccesary files and folders from the Jetson and it would tell us there was enough storage, but once we installed the libraries one at a time, we would see that there still wasn’t enough storage.

We have an SD card for the storage now and are going to work on this during the week.  We also want to make the switch from cnn-facial-landmarks to DLib to improve the accuracy for testing in non ideal conditions.  We will first make this change for the features that currently work so we can see if any changes in their accuracies is worth making the change for head pose estimation, or if we should eliminate head pose estimation.  We are also going to continue working on the poster and start the final report, video, and setup for the demo.

Posted on

Yasser’s Status Report for 04/22/2023

This week, I accomplished successfully implementing the calibration process for head pose estimation, as well as the classification that involves head pose determining when a driver is distracted or not. The calibration code works like this: the user is instructed to look straight at the road for a period of 10 seconds, during which the algorithm calculates the angle between the nose and the x and y axis, saving all of this data. Using these values, the classification determines whether the angle the driver is looking at exceeds the range of angles calculated during calibration. The updated code was integrated into the Jetson this week. To prepare for testing the code in a vehicle, there were some issues downloading Dlib onto the Jetson. I spent a couple of days trying to figure out any issues relating to installing Dlib. They have been resolved.

My progress is on schedule. Some bugs have arisen during integrating the headpose code with the rest of the algorithms which I’m currently working on to solve. After this, the team can move forward on testing the head pose feature.

In the next week, I hope to have the testing complete, as well as the final presentation slides done.

Posted on

Elinora’s Status Report for 04/22/23

Over the past two weeks since my status report, I first set up the WiFi on the Jetson with the new WiFi dongle that we ordered. I had issues connecting to the ethernet in HH in order to set up the dongle initially, and had to re-register the Jetson as my device for wired internet use through CMU Computing. After that, the setup for the dongle was pretty straightforward. We are now able to connect the Jetson to a phone WiFi hotspot (or CMU-SECURE) and ssh into the Jetson remotely. 

I also worked with Sirisha to implement audio feedback and add it into the CV classification code. We were slightly delayed on this task because we needed to order another connector to account for the port differences on the Nano (we had initially planned for the speaker to connect into the USB C port on the Xavier). We then conducted a stationary test on her laptop looking away from the screen for >2s and were able to trigger the audio feedback playing through the speaker. 

For the web app, I set up cloud storage with firebase firestore. I tested adding and retrieving documents from the firestore in the web app itself. I set up hosting for the web app using firebase as well. 

My progress is mostly on schedule – I am behind on integrating data from the device into the web app, but this is because we decided to deprioritize the web app in favor of making sure the device itself works well. This upcoming week before our public demo, I will add code to store feedback data from the device directly into the firestore. After we set up account authentication, I will also add filtering of the feedback logs so that the user can only see logs for their own device. 

 

Posted on

Sirisha’s Status Report for 04/22/23

This past week, I spent time with integrating the software and hardware, finishing up the final parts of the software features, and started testing. Some of the last things that needed to be done with the software included tuning the accelerometer code and figuring out how to get audio feedback to play from the speaker.  This part took a little longer than expected because since we switched from the Jetson Xavier to the Jetson Nano, the ports we originally assumed we’d have were not on the Nano, so we had to wait for some new adapters to arrive so we could work with the ports we have.  There were also some issues with the most common ways to play audio files that we could find, so it took a little longer to find a method that worked for us, but we were able to find one and it was able to be played from the speaker connected to the Jetson during classification.  We started doing testing with the laptop to get some baseline accuracies.  We weren’t able to start testing with the webcam on the Jetson until later because when we ran the code on the Jetson, we had to download a lot of the libraries, but they were causing some issues with the Jetson and it required us to do a lot of more setup.  This issue is practically fixed now.  I also spent a lot of time working on the slides for the final presentation and practicing the presentation since it is my turn for this week.

As of right now, we are on track to finish.  All that is left is testing in various settings in the most accurate way we can find, aside from being in a car, for purposes of the final demo.  The code for the facial detection and eye tracking is all done, code for feedback is done, and accelerometer is almost done, the web application is basically done, and everything is integrating together well.

For next week, we hope to have testing finished and have a test environment that we can use for the demo since we can’t use a car.

Posted on

Team Status Report for 04/22

Currently, one of our most significant risks is that the accelerometer is producing inaccurate acceleration values, and inaccurate speed values as a result. We tried a variety of calibration techniques, but the recorded speed is off by 1mph after approximately 2 minutes. We have not yet tested the accelerometer in the car because we wanted to focus on testing the rest of the system, but we will are aiming to test the accelerometer separately this upcoming week. Fortunately, this risk is less relevant for our final demo as we decided to simulate accelerometer data that the user can control. We chose this method because it is not feasible to move the device at 5 mph in Wiegand gym while all of the other groups are testing around us. Instead, audio feedback will be determined by the user-controlled simulated accelerometer data. 

Another risk that we encountered was that after integrating the different features of the CV code together, the frame rate was ~3fps, significantly lower than our goal of 5fps. This lower fps greatly reduced the accuracy of our blink detection algorithm. To reduce the fps, we would need to implement gpu parallelization or switch the other facial detection algorithms over to dlib (yawn detection and eye-direction), which would be time consuming and less accurate. Instead, we decided to remove blink-detection (inaccurate with the current low fps) as we are still able to detect when eyes are fully closed and yawning to identify drowsiness. 

Since our last status report, our design changes were: removing blink-detection from our device (as mentioned above) and lowering our fps goal to 3fps based on the removal of blink detection.

Posted on

Elinora’s Status Report for 04/08/23

This week, I continued to work with Sirisha on the accelerometer and it now prints the acceleration values and estimated y velocity values. Initially, while the command “12cdetect -r -y 1” showed that the accelerometer was connected at port 68, we were unable to read the actual data from the accelerometer. We ended up having to solder our wires into the holes of one of our spare accelerometers to make the connection more direct. This soldering process took surprisingly long, but we were able to get values printed to the terminal on the Nano. I also wrote some code to estimate the y velocity based on y acceleration values (standard integration). However, this procedure is not very accurate and estimating the velocity causes drifting errors over time, which could jeopardize the feature of our project in which feedback is given depending on the velocity of the car. We are considering purchasing a new sensor that measures velocity directly to address this issue, but may considering integrating it as a stretch goal because of how much time it took to debug using this accelerometer. 

I also set up hosting of our web app using Firebase to https://drivewise-ef9a0.web.app/

Unfortunately, the WiFi dongle did not arrive until later in the week and I was unable to pick it up and start setting it up to work with the Nano. Otherwise, I am on track with the goals that I set for myself last week and our new Gantt chart. 

Next week, I hope to set up the Wifi dongle to work with the Nano, assemble the entire device and test that it still works with the power cord that we have, and work with Sirisha to send data from the Nano to the web app and display it. We want to do this using storage on Firebase. 

Here is a breakdown of the tests that I have run and are planning to run:

  • Accelerometer
    • Try subtracting the average of the first 10 y acceleration readings (per run) to calibrate the sensor before estimating velocity. Then test this in 5 runs when the device is running on a table in HH to determine that the 
    • In the car, drive at 4, 5, and 6 mph to test whether the feedback is only triggered at > 5mph. Also test while the car is stationary and on for 5 minutes (making sure that the estimated velocity does not just continue accumulating). 
  • Web App
    • Test that the recorded data displays correctly on the web app
    • Check that a user only views data from their device by doing 1 ride with a set device id of 0 and 1 test ride with a set device id of 1 and make different login accounts for each
    • Test that multiple users can be logged into different accounts on the hosted web app at the same time
  • Jetson Nano
    • Test that the Nano is able to be powered by the car (with USB A to USB C cord plugged into USB A port on car)
  • Overall System
    • Test each of our distraction cases in ideal conditions in both a stationary and moving car in an empty parking lot
    • Test each of our distraction cases (other than blinking) in nonideal conditions in both a stationary and moving (very slow) car in an empty parking lot
Posted on

Team Status Report for 04/08/23

Currently, our most significant risk is head pose estimation – head pose is not being successfully classified for distracted driving. Finding a relation between a driver’s distance from the camera and the angles of the head that will constitute as looking away from the road is very difficult. To solve this, we are looking into ways to better calibrate the head pose estimation. One idea for calibration is to have the driver look up, down, left and right, measuring the angles and determining the threshold angles of looking away from the road. We will test this calibration step with a variety of chair heights and distances from the steering wheel. 

Another risk is that since the accelerometer does not directly measure velocity, our calculated mph values increasingly drift from the actual velocity value over time. This affects our ability to control issuing of feedback based on the velocity of the car. This issue occurs because  we are calculating velocity by integrating samples of the acceleration (by adding (accel_curr – accel_prev) * 0.5 * (curr_time – prev_time)  to prev calculated velocity). We also tried subtracting the initial measured y acceleration from each later measured y acceleration, but the calculated y acceleration still becomes very inaccurate over time. When updating our velocity calculation every 0.5s and keeping the accelerometer stationary, the calculated y acceleration becomes -2 m/s (~1mph in reverse) after only a minute. Possible mitigation strategies include subtracting the average y acceleration value that (over 10 or so samples) as a calibration step. Another option would be to purchase a different sensor that measures velocity directly. 

No changes have been made to the existing design of the system since last week. 

We updated our Gantt chart to reflect what we have completed and reprioritize the tasks that we have left. This coming week we finish integration and get started with testing the entire system in a car (stationary initially). In the current schedule, Yasser has 2 weeks of slack and Sirisha and Elinora have 1 week of slack. 

Images of head pose working (when angle and distance from camera are kept constant):