Category: Team Status Reports

The biggest change to our system at the moment is converting all of our features over to DLib.  We are trying to make this change in time because doing so would increase our accuracies for testing in nonideal conditions.  The costs incurred are that it will take time to make the switch as well as retest with the new model, and we don’t have a definitive answer as to how this change will affect the current accuracies we have.  However, we believe it is worth trying for the increase to the frame rate and more accurate head pose estimation.  We will also continue testing what we currently have in the case that the change can’t be made in time so we still have a good final product.

Another risk we are currently having is related to integrating the full CV code onto the Jetson Nano. We were having issues installing tensorflow onto the Jetson, but the current problem is that the microSD card in the Jetson Nano is too small for our code and needed libraries. We have ordered a 128gb microSD card that will arrive on 04/30, at which time we will set up the Jetson again with the new SD card and restart the process of installing the necessary libraries and be able to run our full code on the Jetson directly.

We have pushed entire system testing on the Jetson again (to this upcoming week) because of issues we are having with integrating the CV subsystem code onto the Jetson Nano. We have also added tasks for converting head pose and mouth detection from CNN facial landmarks to DLib. 

Unit Test	Notes
Accelerometer (in Hamerschlag)	Acceleration data fluctuates significantly when the accelerometer is stationary – inaccurate acceleration measures also leads to inaccurate speed estimations. Need to still test in a moving car to determine feasibility of this unit.
Head pose estimation (CNN facial landmarks)	When tested pre-integration, head pose worked very accurately. However, the lower fps caused by integration makes the head pose ineffective. This led to the design change to convert Head pose and mouth detection from CNN to DLib because the use of the same algorithm for all of our components would decrease runtime and increase fps. Will need to retest after converting to DLib.
Mouth detection (CNN facial landmarks)	Worked accurately for both pre and post integration tests. Will need to retest after converting to DLib.
Eye-tracking	Worked accurately pre and post integration
Blink detection	Worked accurately pre-integration, but fps was too low for accurate blink detection after integration. We are expecting performance to improve after
Connecting to WiFi hotspot in car and ssh-ing in	Set up the Jetson to auto-connect to Elinora’s hotspot and checked that it was connected in the car. Did this 3x successfully in the car. There is a long connection time ~ 5 min and not waiting previously led us to think that the connection wasn’t working.
Retrieving data from Firestore (in React) and displaying on the web app	Created a test collection on Firestore and confirmed that the data from that test collection was retrieved and shown on the web app in table format.
Hosting the web app	Able to connect remotely from all 3 of our computers
Firebase authentication	Able to create 3 accounts and log in.
Audio feedback	Tested the speaker separately and then connected it to a laptop to play audio feedback integrated into the classification code. Audio feedback was played virtually instantaneously (passing our latency requirement)

 

Currently, one of our most significant risks is that the accelerometer is producing inaccurate acceleration values, and inaccurate speed values as a result. We tried a variety of calibration techniques, but the recorded speed is off by 1mph after approximately 2 minutes. We have not yet tested the accelerometer in the car because we wanted to focus on testing the rest of the system, but we will are aiming to test the accelerometer separately this upcoming week. Fortunately, this risk is less relevant for our final demo as we decided to simulate accelerometer data that the user can control. We chose this method because it is not feasible to move the device at 5 mph in Wiegand gym while all of the other groups are testing around us. Instead, audio feedback will be determined by the user-controlled simulated accelerometer data. 

Another risk that we encountered was that after integrating the different features of the CV code together, the frame rate was ~3fps, significantly lower than our goal of 5fps. This lower fps greatly reduced the accuracy of our blink detection algorithm. To reduce the fps, we would need to implement gpu parallelization or switch the other facial detection algorithms over to dlib (yawn detection and eye-direction), which would be time consuming and less accurate. Instead, we decided to remove blink-detection (inaccurate with the current low fps) as we are still able to detect when eyes are fully closed and yawning to identify drowsiness. 

Since our last status report, our design changes were: removing blink-detection from our device (as mentioned above) and lowering our fps goal to 3fps based on the removal of blink detection.

Currently, our most significant risk is head pose estimation – head pose is not being successfully classified for distracted driving. Finding a relation between a driver’s distance from the camera and the angles of the head that will constitute as looking away from the road is very difficult. To solve this, we are looking into ways to better calibrate the head pose estimation. One idea for calibration is to have the driver look up, down, left and right, measuring the angles and determining the threshold angles of looking away from the road. We will test this calibration step with a variety of chair heights and distances from the steering wheel. 

Another risk is that since the accelerometer does not directly measure velocity, our calculated mph values increasingly drift from the actual velocity value over time. This affects our ability to control issuing of feedback based on the velocity of the car. This issue occurs because  we are calculating velocity by integrating samples of the acceleration (by adding (accel_curr – accel_prev) * 0.5 * (curr_time – prev_time)  to prev calculated velocity). We also tried subtracting the initial measured y acceleration from each later measured y acceleration, but the calculated y acceleration still becomes very inaccurate over time. When updating our velocity calculation every 0.5s and keeping the accelerometer stationary, the calculated y acceleration becomes -2 m/s (~1mph in reverse) after only a minute. Possible mitigation strategies include subtracting the average y acceleration value that (over 10 or so samples) as a calibration step. Another option would be to purchase a different sensor that measures velocity directly. 

No changes have been made to the existing design of the system since last week. 

We updated our Gantt chart to reflect what we have completed and reprioritize the tasks that we have left. This coming week we finish integration and get started with testing the entire system in a car (stationary initially). In the current schedule, Yasser has 2 weeks of slack and Sirisha and Elinora have 1 week of slack. 

Images of head pose working (when angle and distance from camera are kept constant):

Setting up the Jetson and accelerometer took significantly longer than expected, so we are behind on setting up hardware and integrating the system. After the Jetson is capable of connecting to WiFi (which should be completed by the end of this week), integration will be much easier and be able to occur outside of the classroom. While different aspects of the CV are already completed (calibration, eye-tracking, head-pose estimation, and yawning/blinking detection), integrating these steps to work together in one process is still a significant risk that needs to be resolved with debugging and user testing. Another risk coming up is setting up communication between the device and the web app. This is an area that we don’t have significant experience in and foresee a decent chunk of time being spent on debugging. We will mitigate this risk by using some our slack time to focus on this communication. 

We swapped our Jetson AGX Xavier for a Jetson Nano this week because we were worried about the power consumption of the Xavier (and whether a car power outlet would be able to fully support the device when connected to the camera, speaker, and accelerometer) and the Xavier seemed more difficult to setup. Both Jetsons were borrowed from the 18500 parts inventory, so no monetary costs were incurred and the overall time to set up the Nano was less than the setup time for the Xavier would have been. Unfortunately, the Nano does have lower computing power capabilities than the Xavier, so there is a risk that we may be unable to meet our accuracy requirements because of this switch. 

 

Right now, the most significant risks are making sure that the computer vision code is complete and working as soon as possible because if it isn’t, we won’t be able to progress with the other tasks. The eye tracking code has been completed but we would need approval from our professor this week to see if the way we’re classifying a driver as looking away from the road is valid.

Since finishing up with the computer vision algorithms is the main priority right now, Sirisha has been tasked with helping with the mouth detection algorithm to speed up the process.

We also need to make sure all of the hardware components have arrived for us to start integrating soon. Camera has been ordered after considering whether the initial camera we had planned on ordering was of an adequate size for placing inside a vehicle.

No changes were made to the system design.

The only change to the schedule is pushing the computer vision code and hardware integration back.

As of right now, the most significant risks that could jeopardize the success of the project are making sure that the algorithms used for eye tracking and facial detection are able to be completed on time and can be done with the appropriate frame rate.  This is imperative to get done as soon as possible because many of the other features cannot be completed until that part is done.  We have been working to constantly test the algorithms to ensure that they are working as expected, but in the event that they aren’t, we have a backup plan of switching from OpenCV DNN to Dlib.

We were looking into changing the model of the NVIDIA Jetson because the one we currently have could possibly utilize more power than what a car can provide. If this change needs to happen, it won’t incur any extra costs because there are other models in inventory and the other hardware components that we have are compatible with the other models. Also, in between the design presentation and report, we have added back the feature of the device working in non ideal conditions (low lighting and potential obstruction of the eyes by sunglasses or a hat). This was done based on feedback by faculty, but at the moment we are still unsure if we will for sure add back non ideal conditions because other faculty/TA feedback mentioned that we shouldn’t add it back. If it is added back, the cost incurred will be having to spend time working on the algorithm with non ideal conditions. This will lead to less time perfecting the accuracy of the ideal conditions algorithm. We are also changing the design slightly to account for edge cases in which the driver would be looking away from the windshield or mirrors while the car is moving but still not considered distracted. This would occur when the driver is looking left or right while making a turn or looking over their shoulder while reversing. For now, we will limit audio feedback for signs of distraction to be given only when the car is above a threshold speed (tentatively 5mph), replacing our previous condition of the car being stationary. This mph will be adjusted based on recorded speeds of turns and reversing during user testing. If we have extra time, we are considering detecting whether the turn signal is on or car is in reverse to more accurately detect these edge conditions.

After our meeting with the professor and TA this week, we are changing the deadline for the completion of the eye tracking and the head pose estimation tracking algorithms to be next week. We made this change because the CV/ML aspects of our project are the most critical to achieve our goals and having the algorithms completed earlier will allow us to have more time for testing/tuning to create a more robust system. Also, we have shifted the task of assembling the device and testing individual components, specifically the accelerometer and camera, to next week after those components have been delivered. Otherwise, we are on track with our schedule.

 

The significant risks that could jeopardize this project is our lack of knowledge about wifi cards and which one to buy. The wifi card component is very important for our project as it will allow us to transfer driver data to the cloud, allowing drivers to keep track of how they’re driving. This will be mitigated with consulting the TAs and anyone else who has experience with wifi cards.

Some changes that were made to the design was adding a head pose estimation system. Eye tracking on driver’s with sunglasses is not feasible and after critiques from the professor in class during the design presentation, we’ve decided to tackle this issue with head pose estimation where the person’s head position would be measured to determine whether they’re distracted/drowsy. This will add an extra week to the schedule and push back our progress by a week. This is being mitigated by the 3 week slack time buffer we had at the very beginning.

Other than adding the head pose estimation code, everything else from the schedule last week looks relatively the same.