For eye classification, I created a baseline set of vectors extracted from a subset of the Closed Eyes in the Wild dataset. This way, if the user does not properly calibrate at the beginning, then we will still have a good baseline model to use to classify their eyes. Additionally, I helped debug an issue we were having with GPU acceleration and reflashed the Jetson image. Moreover, I worked with my team throughout this week to work through other problems that we encountered, such as with our systemd, audio, and head pose. 

Last week, I said I would be working on taking videos in Danielle’s car for testing. Instead of taking videos, we decided to go ahead and run many tests of our complete system in the car. Danielle and I did 100 trials of various poses and simulated drowsiness (while parked). Danielle and I also did 30 trials for measuring our system latency, specifically how long our system takes to detect and output an audio alert. Additionally, I tested the power supply with our running system on the Jetson Xavier.  I also gathered more face detection metrics.

Finally, I worked on the presentation. I will be presenting on Monday/Wednesday, so I worked on the slides and my script. 

I am on schedule, and the next steps are to finish preparing for the presentation and then help create our demo video, poster, and final paper.

My progress for the project was focused on gathering our final metrics. To do this, we want to test both with databases and with ourselves. I created a total of three automated scripts to analyze our datasets that we found online or created ourselves. One script can process a given set of photos of an individual with their eyes open or closed to test how accurate our eye classification model is. I set the script up so that the final output of the script tells you how accurate the eye classification model was, thus making this script easy to run on large sets of images (specifically, I designed it for LFW and MRL Eye Dataset, details below). This is for gathering data for our “Eye classification matches truth >=90% of the time” metric. The next script I made can process a given video stream and retrieve the same information as above. This was made to process videos that we take of ourselves and others while we are driving a car, so that we can test in the proper driving environment. Finally, I configured our working overall system to work on videos from the DROZY database and UPNA Head Pose Database to gather data for our “Distinguishes distracted vs normal >=90% of the time” metric. The DROZY database is used for gathering metrics for drowsiness being detected, and the UPNA Head Pose Databse is used for metrics for distractedness being detected. 

I have begun to gather metrics from a number of databases and compile my results in a spreadsheet. The databases that I am using for gathering metrics are the following:

• LFW for testing if eyes are accurately classified as open (http://vis-www.cs.umass.edu/lfw/index.html)
• MRL Eye Dataset for testing if eyes are accurately classified as open or closed (http://mrl.cs.vsb.cz/eyedataset) 
• DROZY for testing accuracy of drowsiness being detected (http://www.drozy.ulg.ac.be/)
• Head Pose Image Database for testings if head poses are accurately identified (http://crowley-coutaz.fr/Head%20Pose%20Image%20Database.html)
• UPNA Head Pose Database for testing accuracy of distractedness being detected (http://www.unavarra.es/gi4e/databases/hpdb)

I am on schedule. The next step is to record videos in Danielle’s car so that I can run my scripts on those videos.

This week, I was successfully able to complete and test my code for creating and training our eye classification model using SVM. To create and train this SVM model, I first have a calibration system where the user follows prompts to keep their eyes open for a period of time, in which I calculate their various eye aspect ratios and create a dataset of vectors based on this. Then, I repeat this with the user’s eyes being closed. Afterwards, I use this data to train a SVM classifier for that specific person. My SVM classifier has two labels: 0, which classifies the eyes as open, and 1, which classifies the eyes are closed. It is important to note that I am only using our custom calibration data for my training my model, and not data from a database, since each person has unique thresholds. Then, during my real-time eye classification, I use this trained model to determine if the user’s eyes are closed or not. If the user’s eyes are closed for at least 1 second, my code detects that the user is drowsy. If the user is detected to be drowsy, I currently have a visual alert, which will later be changed to an audio alert. I have a screenshot of this below. So far, with testing with Heidi and myself, my model has been accurate. I am in the process of testing with DROZY, a drowsiness database, to see how accurate my overall drowsiness detector is. The next action for this code is to hand it over to Danielle, who will add the focus timer, audio prompts for calibration, and audio alerts for drowsiness being detected. 

I am currently on schedule. Next week, I will work with Heidi to integrate her head pose algorithm with my eye classification algorithm, so it is in one seamless file to run on the Jetson. If we cannot complete this in time before the demo, we will just have our calibration mechanisms integrated. 

Resources:

DROZY: http://www.drozy.ulg.ac.be/

Since the previous status report, I worked on completing the Design Report with my teammates as well as adding to and testing our FocusEd codebase on my laptop. The first addition I made was that I integrated my eye classification code with Heidi’s HOG facial detection code. An image of this is shown below. While I completed this integration step earlier than I had designated on our schedule, it was a logical step to integrate the early version of two as soon as possible to then proceed to training and testing.

Next I worked on training of our eye classification model. Since every individual will have a different EAR threshold (which is the EAR value in our algorithm that determines if an individual’s eyes are closed), we wanted to train a model in order to accurately determine whether or not an individual’s eyes are closed or not. I am working on building a Support Vector Machine (SVM) classifier to detects long eye blinks as a pattern of EAR values in a short temporal window, as stated in T Soukupova and Jan Cech’s research paper “Real-time eye blink detection using facial landmarks” (this was the first paper that introduced EAR). I am using Jupyter Notebook in order to do so, which is something that is new to me and thus required time this week to set up and learn how to use. My EAR value, which belongs to a larger temporal window of a frame, will be given as input to the fully-trained SVM machine learning model.

Finally, another change I added was that I made my eye classification code simpler to run in terms of where the video input is found on the device the code is running on in anticipation of beginning Xavier testing of my code this Tuesday. I am currently slightly behind schedule since I hoped to have already tested my code on the Xavier, but I was able to complete the integration step mentioned before earlier than intended to make up for this. In the next week, I hope to finish training and then test on the Xavier starting Tuesday.

Sources

T Soukupova and Jan Cech. Real-time eye blink detection using facial landmarks. In 21st Computer Vision Winter Workshop (CVWW 2016), pages 1–8, 2016. https://vision.fe.uni-lj.si/cvww2016/proceedings/papers/05.pdf

For me, this week began with taking our TA Abha’s feedback on our draft of our design presentation slides and making changes accordingly. This included completely redesigning our user flowchart to show how the system turns on, specify what algorithms are running in parallel, and clarify how the driver response is checked. I also helped my team specify in our software block diagram what is off the shelf and what we are creating. 

Additionally, this week I started to test my eye classification code and made changes accordingly. I also began to properly document our code to make it as clear as possible for our debugging process. 

Finally, I worked on our Design Review Report with my teammates, as this is due next week. Thus, next week my main focus will be to complete the design report. I will then continue working on my eye classification algorithm and begin training with an already-existing dataset. My progress is on schedule.

This week our group met after Wednesday’s proposal presentation to discuss our next steps. Specifically for me, my next steps were to finish researching eye detection algorithms and to select the best option to then begin working on programming our eye detection over this weekend and during the next week. I hope to then finish on schedule by 3/4/2021 to begin testing. Therefore, I researched and solidified that we would be using the eye detection algorithm of Dlib’s Histogram of Oriented Gradients (HOG) and Linear Support Vector Machines (SVM) method for object detection. The reason for this is that, according to the Towards Data Science article and other tutorials listed below, HOG is a better option compared to Dlib’s CNN algorithm since it has a faster computation time for real-time video processing. However, there is a tradeoff that HOG is slightly less accurate than CNN for non-frontal faces, but we are aiming to place our camera with a direct frontal perspective of the driver to help compensate. 

During our team meeting, Heidi and I decided to use the same algorithms for facial and eye detection, thus not relying on two different simultaneous algorithms to compute facial and eye detection, when the facial detection will already be determining the locations of the eyes. 

Then, I set up our codebase through a private Github repository and began working on setting up Dlib. I also started reading the user manual for the NVIDIA Jetson Xavier NX Developer Kit. Finally, I worked on perfecting our budget so that we can order all our necessary parts by Monday. Since we are using a much more expensive microprocessor ($99 NVIDIA Jetson Nano vs. $399 NVIDIA Jetson Xavier), we need to make sure that we stay under our $600 total budget.  

By next week, I hope to have my eye detection code working with preliminary testing, all pushed to the repository. This includes finding datasets for training. Then, I want to get started with eye classification to determine whether the eye is opened or closed. I will use the Eye Aspect Ratio (EAR) equation for this. 

Sources

Why HOG

 

https://maelfabien.github.io/tutorials/face-detection/#3-real-time-face-detection-1

https://towardsdatascience.com/cnn-based-face-detector-from-dlib-c3696195e01c

https://www.pyimagesearch.com/2014/11/10/histogram-oriented-gradients-object-detection/

 

Eye Detection

https://www.educative.io/edpresso/eye-blink-detection-using-opencv-python-and-dlib

https://www.pyimagesearch.com/2017/04/24/eye-blink-detection-opencv-python-dlib/

https://towardsdatascience.com/a-guide-to-face-detection-in-python-3eab0f6b9fc1

 

NVIDIA Xavier User Guide

https://developer.download.nvidia.com/assets/embedded/secure/jetson/Xavier%20NX/Jetson_Xavier_NX_Developer_Kit_User_Guide.pdf?a89SNG_TxN9K0wiYbBWmdVF5WdJvtGJEvS–YZxbCnhAurtz2dqKfuxi93w_rhKTroNbZb0Eu41atCdOGAqSVw02a2m7EphpIa4rIVu325GWAvCe1wMA5fBO2H6BgZb0rD0xy5jzxr5XfH9BHgKmihXzbslTwi7mEJ5k3DOA7xcuWxk8N9FofDfeNSo_kE0bbfDDOjwR