Author: ycorzo

This week, after the team decided that it would be best to use dlib’s facial landmark model on all of our features to increase the fps and decrease the amount of false negatives (as a result of the low fps), I changed the model for eye tracking. After testing, the accuracy is comparable to the previous model (cnn-facial-landmark).

Progress is behind schedule. I was supposed to get done with migrating the features to Dlib by today but I only got eye tracking done. I plan on migrating the dlib model to the other features (mouth detection, head pose, and eyes closing detection) by tomorrow and start testing these features immediately after.

By next week, I hope to have testing done for all of the features, as well as the poster presentation and the demo video.

This week, I accomplished successfully implementing the calibration process for head pose estimation, as well as the classification that involves head pose determining when a driver is distracted or not. The calibration code works like this: the user is instructed to look straight at the road for a period of 10 seconds, during which the algorithm calculates the angle between the nose and the x and y axis, saving all of this data. Using these values, the classification determines whether the angle the driver is looking at exceeds the range of angles calculated during calibration. The updated code was integrated into the Jetson this week. To prepare for testing the code in a vehicle, there were some issues downloading Dlib onto the Jetson. I spent a couple of days trying to figure out any issues relating to installing Dlib. They have been resolved.

My progress is on schedule. Some bugs have arisen during integrating the headpose code with the rest of the algorithms which I’m currently working on to solve. After this, the team can move forward on testing the head pose feature.

In the next week, I hope to have the testing complete, as well as the final presentation slides done.

Personally, I accomplished starting integrating the computer vision code into the jetson and started testing the computer vision code with the camera on the jetson. Bugs have risen in the integration which have to be fixed, specifically with mouth detection detecting certain frames where a user’s mouth is closed and eye tracking not always successfully tracking a user’s distracting when looking up.

According to the Gantt chart, I am not behind on progress.

In the next week, we hope to have the pose estimation classification done and working as well having the integration bugs of the cv code fixed and working by our next team’s meeting on Wednesday. After Wednesday, we hope to have started testing our device in a vehicle.

Tests that I plan on running are with eye tracking, having the user look left, right, up, and down with respect of the pupil location. With mouth detection, have the user yawn for a period of at least 5 seconds and see if the system correctly classifies this as a sign of drowsiness. With eye detection, have the user close their eye lids for more than 2 seconds and see if the system correctly classifies this as being distracted/drowsy. With respect to head pose, have the user turn their head far left, right, up and down for more than 2 seconds and correctly classify this as the driver being distracted. All of these tests will be done in a vehicle.

This week I personally accomplished fixing the looking up and down feature of the eye tracking algorithm by minimizing the range of y values that the pupil can move to that would constitute as looking at the road. The mouth and blinking detection code has been done with the help of Sirisha, as well as the classification code for these detections. I also did start the head pose estimation code that will measure the driver’s head position.

From my goals last week, my progress is on schedule. The head pose estimation code is to be done by tomorrow which I do aim at accomplishing.

For next week, I aim to have the head pose estimation code done and integrating all of the computer vision features together in time for the interim demo. I also aim to have some testing done in a car by the end of this week, after the interim demo.

This week, I personally accomplished finishing writing the eye tracking algorithm and the classification of determining whether a driver is looking away from the road for more than 2 seconds, as well as the calibration needed for this process. After preliminary testing, I did notice some bugs in the algorithm due to noise. After getting some advice from my research advisor, I tuned my algorithm to instead calculate the distance of the pupil moving to the two far end points of the eye. If the distance between the center of the pupil is very near these two endpoints, then this determines whether a driver is looking away. I’ll speak to the professor in our weekly meeting next week to see if this is enough for the eye tracking algorithm. I also started working on the blinking feature.

Despite my advances, I am still behind schedule due to other assignments from other classes and testing of the eye tracking algorithm. I plan on focusing on the blinking and mouth detection feature this week. After these two features are complete, all of the computer vision code will have been finished. Testing and integration with the hardware can soon be commenced.

Next week I plan on having the algorithms for the blinking and mouth detection features done as well as the classification needed to determine whether a driver is sleepy by the frequency of blinks and lengths of yawning.

Personally, I accomplished getting the eye tracking algorithm working. There is an issue with the frame rate being slow on my webcam which can be attributed to the high resolution so further testing on our purchased camera will be needed to see what the issue is. Below is a screen shot of the eye tracking (red dots around my eyes, might not be fully visible in this photo).

 

My progress is still behind. I had planned on having the classification and calibration step done. However, after meeting with the professor, I need to focus on getting the gaze tracking algorithm first with determining when a driver’s pupils are looking away from the road, which I plan on getting done by Wednesday.

After I’m done with that, I plan on working on the mouth detection, calibration step, and classification. By next week, I hope on getting the code that determines whether a driver is looking away from the road done and get started on the calibration step.

 

Earlier this week, I accomplished wrapping up the design presentation slides, specifically with the block diagrams regarding the facial and landmark detection algorithms, as well as the mouth and eye detection. After the presentation, I accomplished researching the frameworks and models needed for head pose estimation. After speaking with the professor after the design presentation, he suggested we circumvent the sunglasses issue on the driver with measuring head movement to see if a driver is distracted/drowsy. A Tensorflow Model and Caffe based OpenCV DNN is needed, both of which are being used for the facial detection and landmark algorithms we intend to use in this project. Besides researching about head pose detection, I started coding the facial detection algorithm using OpenCV and the Caffe facial detector model.

My progress ahead. By the end of this week I was tasked with having all of the algorithms needed researched. Since I had that done earlier in the week, I decided to move on with next week’s task of writing the code for the facial and landmark detector. As shown in the image, the facial detection code is done.

By the end of next week, I expect to have the landmark detector code done, as well as the mouth and eye detection code and the design document as well.

Some team shortfalls this week was not having my team members in pittsburgh when the design presentation slides were due. I especially needed help with writing the speaker notes on the slide with the hardware block diagram as I had very little knowledge on the hardware used for this project, as well as the UI and the servers where our webapp will be hosted on. However, we got through this obstacle with constant communication.

The most significant risks that could jeopardize the success of this project is adjusting our algorithms and approach for low light settings and drivers with sunglasses. To mitigate this risk, we’ll have to reduce the scope of this project and specify having good lighting conditions where the driver’s face is clear and visible and add a requirement for the driver that sunglasses be taken off. Another issue could be the accuracy of OpenCV’s DNN for facial detection. Only preliminary testing has been done but if we find that this model does not give us the accuracy that we want or if using GPU parallelization with CUDA is not compatible with the Volta GPU on the Xavier or if it’s out of scope then we’ll resort to using Dlib which has a higher accuracy.

There were some changes to made to the design. We added a calibration step to the software after the driver input is detected. The reason why this step is necessary is because a point of reference is needed when checking if a driver is distracted or not. To be more specific, in this calibration step we are measuring and recording the distance of the lips when the mouth is closed along with eye measurements of the driver looking at the road. Costs that the change incurs are more code/time spent on this feature. This cost is mitigated by the 3 week buffer we originally had.

Some changes were made to the schedule, such as a week of adding the calibration process in addition to moving the start time of designing the UI of the webapp by a week in advance.