Posted on

Vaheeshta’s Status Report for 5/8

On Sunday, I met with Danielle and took videos of us demonstrating our system as she drove around Pittsburgh. Then, I made more changes to our Final presentation, polished my presentation script, and practiced for the Final presentation I gave on Monday. I am thankful that I memorized my script because, near the end of the live presentation, my computer froze over. I was not able to see the slides, but my audio was still working, so I presented completely from memory. I greatly appreciate my team for all their help in handling this technical difficulty!

After I presented, Heidi and I tackled our systemd bugs that we were facing the previous week. We wanted FocusEd to start as soon as the Jetson turned on, but the systemd service we tried to set up was giving us permission errors. Once we fixed one permission error, we were facing more issues related to the audio not playing. After much trial and error, we figured out our bug was related to the certain folder we were running in. Now, our system successfully runs on boot, making it a fully usable system. Heidi and I also worked to improve our FPS and delay, so now we run at 17+ FPS. We are still using 10W of power from our power bank and thus using 4 CPU cores (out of the 6 available). The latency improvements came from reducing the camera resolution.

Afterwards, I took videos of myself using the faster system for our final demo video. This included screen recordings on the Jetson of our display with our eye landmarks and facial areas so that it is more clear what we are using to make our estimations. 

I am on schedule and hope, in the next week, to help my team finish up our demo video and complete our final report.

Posted on

Vaheeshta’s Status Report for 5/1

For eye classification, I created a baseline set of vectors extracted from a subset of the Closed Eyes in the Wild dataset. This way, if the user does not properly calibrate at the beginning, then we will still have a good baseline model to use to classify their eyes. Additionally, I helped debug an issue we were having with GPU acceleration and reflashed the Jetson image. Moreover, I worked with my team throughout this week to work through other problems that we encountered, such as with our systemd, audio, and head pose. 

Last week, I said I would be working on taking videos in Danielle’s car for testing. Instead of taking videos, we decided to go ahead and run many tests of our complete system in the car. Danielle and I did 100 trials of various poses and simulated drowsiness (while parked). Danielle and I also did 30 trials for measuring our system latency, specifically how long our system takes to detect and output an audio alert. Additionally, I tested the power supply with our running system on the Jetson Xavier.  I also gathered more face detection metrics.

Finally, I worked on the presentation. I will be presenting on Monday/Wednesday, so I worked on the slides and my script. 

I am on schedule, and the next steps are to finish preparing for the presentation and then help create our demo video, poster, and final paper.

Posted on

Vaheeshta’s Status Report for 4/24

My progress for the project was focused on gathering our final metrics. To do this, we want to test both with databases and with ourselves. I created a total of three automated scripts to analyze our datasets that we found online or created ourselves. One script can process a given set of photos of an individual with their eyes open or closed to test how accurate our eye classification model is. I set the script up so that the final output of the script tells you how accurate the eye classification model was, thus making this script easy to run on large sets of images (specifically, I designed it for LFW and MRL Eye Dataset, details below). This is for gathering data for our “Eye classification matches truth >=90% of the time” metric. The next script I made can process a given video stream and retrieve the same information as above. This was made to process videos that we take of ourselves and others while we are driving a car, so that we can test in the proper driving environment. Finally, I configured our working overall system to work on videos from the DROZY database and UPNA Head Pose Database to gather data for our “Distinguishes distracted vs normal >=90% of the time” metric. The DROZY database is used for gathering metrics for drowsiness being detected, and the UPNA Head Pose Databse is used for metrics for distractedness being detected. 

I have begun to gather metrics from a number of databases and compile my results in a spreadsheet. The databases that I am using for gathering metrics are the following:

I am on schedule. The next step is to record videos in Danielle’s car so that I can run my scripts on those videos.

Posted on

Team Status Report for 4/10

One risk that could jeopardize our success is when we implement GPU acceleration to remove any delay of the system. It is important to be as close to real-time as possible. Right now we noticed a small delay when we integrated the head pose and eye classification code. Our plan for this is to get it as close to realtime first on the CPU and then implement GPU acceleration using CUDA Python and following the NVIDIA walk through. Additionally, for our testing we have been using a monitor to visually see how our algorithms are working so when we remove the cv2.show calls this should also improve the delay we see. For the interim demo we will most likely have this delay still but are working on minimizing it by checking the frames per second.

There have not been any significant changes to our existing design since last week. We have just been working on integration. To improve our integrated design, the calibration for head pose will resemble the calibration for eye classification. The ratio used will be the area of the cheek areas of each side of the face to determine whether the driver is looking right, left, forward, etc.

Some changes were made to the Gantt chart. We updated it with the actual dates we integrated the two algorithms and added additional tasks to ensure our MVP will be ready to test in a car setting.

Our updated Gantt chart is below.

 

Gantt Chart pdf

Posted on

Vaheeshta’s Status Report for 4/10

This week, I successfully integrated all my code with Heidi’s and Danielle’s code to create our working demo. We all met together, in-person, on Friday and Saturday to make sure that this happened smoothly. I also set the integrated code up on the Jetson Xavier NX, with Danielle and Heidi’s help, and tested it. Therefore, for our demo we can now show the integrated system working on the Jetson Xavier NX. When integrating the eye classification and head pose estimation systems together, I made sure that our calibration sequences happen sequentially when the user boots up the system: The system first asks the user to calibrate head pose for 10 seconds, then it asks the user to keep their eyes open for 10 seconds for calibrating eye open classification, and finally it asks the user to keep their eyes closed for 10 seconds for calibrating eye closed classification. Thus, at this moment 30 seconds are spent for calibration when the system is turned on. Additionally, when testing on the Jetson Xavier, I faced issues with updating several libraries, but with Heidi and Danielle’s help we were able to fix this.  

My progress is, for the most part, on schedule. Along with integration, I also was hoping to test the power bank with the Xavier this week, but upon opening the power bank we received from the 18-500 inventory I noticed that the kit was missing the male-to-male DC connector that connects the power bank to the Xavier. Thus, Danielle ordered us a new one that should arrive next week for me to test. Therefore, next week I will be testing the power bank and working with my teammates to implement any feedback from the demo. 

 

Posted on

Team Status Report for 4/3

At this moment, one risk that could jeopardize our success is that we may not have given ourselves enough time for adding concurrency to our integration of our drowsiness detection system and head pose estimation system, as well as testing it. We want to use Python threading to have our eye classification and head pose estimation run at the same time after the facial landmarking. Since we all do not have much experience with concurrency, we may have underestimated the challenges we may face with this. Our contingency plan is that we will not use concurrency, and instead do the EAR calculations and head pose classifications linearly, and then determine if the driver is drowsy. 

One change we made is that we are only going to use our own custom datasets for training of eye classification. Thus, we will only be using public datasets for testing. For instance, we will use DROZY, a drowsiness database, for testing our drowsiness detection. Additionally, another change we made is that, after comparing our calibration systems for head pose and eye classification, we decided to model the head pose calibration after the eye classification calibration. Therefore, each will use the calibration time to generate vectors to then train separate SVM models. This way, we can make our head pose system more accurate for the user. Additionally, we clarified what type of audio prompts we would like to use during calibration, so that our user experience will be as smooth as possible for this crucial step. We want the user to perform our calibration requests while still not inconveniencing the driver with repetitive, and thus seemingly excessive, calibration requests.

 

Posted on

Vaheeshta’s Status Report for 4/3

This week, I was successfully able to complete and test my code for creating and training our eye classification model using SVM. To create and train this SVM model, I first have a calibration system where the user follows prompts to keep their eyes open for a period of time, in which I calculate their various eye aspect ratios and create a dataset of vectors based on this. Then, I repeat this with the user’s eyes being closed. Afterwards, I use this data to train a SVM classifier for that specific person. My SVM classifier has two labels: 0, which classifies the eyes as open, and 1, which classifies the eyes are closed. It is important to note that I am only using our custom calibration data for my training my model, and not data from a database, since each person has unique thresholds. Then, during my real-time eye classification, I use this trained model to determine if the user’s eyes are closed or not. If the user’s eyes are closed for at least 1 second, my code detects that the user is drowsy. If the user is detected to be drowsy, I currently have a visual alert, which will later be changed to an audio alert. I have a screenshot of this below. So far, with testing with Heidi and myself, my model has been accurate. I am in the process of testing with DROZY, a drowsiness database, to see how accurate my overall drowsiness detector is. The next action for this code is to hand it over to Danielle, who will add the focus timer, audio prompts for calibration, and audio alerts for drowsiness being detected. 

I am currently on schedule. Next week, I will work with Heidi to integrate her head pose algorithm with my eye classification algorithm, so it is in one seamless file to run on the Jetson. If we cannot complete this in time before the demo, we will just have our calibration mechanisms integrated. 

Resources:

DROZY: http://www.drozy.ulg.ac.be/

Posted on

Vaheeshta’s Status Report for 3/27

Since the previous status report, I worked on completing the Design Report with my teammates as well as adding to and testing our FocusEd codebase on my laptop. The first addition I made was that I integrated my eye classification code with Heidi’s HOG facial detection code. An image of this is shown below. While I completed this integration step earlier than I had designated on our schedule, it was a logical step to integrate the early version of two as soon as possible to then proceed to training and testing.

Next I worked on training of our eye classification model. Since every individual will have a different EAR threshold (which is the EAR value in our algorithm that determines if an individual’s eyes are closed), we wanted to train a model in order to accurately determine whether or not an individual’s eyes are closed or not. I am working on building a Support Vector Machine (SVM) classifier to detects long eye blinks as a pattern of EAR values in a short temporal window, as stated in T Soukupova and Jan Cech’s research paper “Real-time eye blink detection using facial landmarks” (this was the first paper that introduced EAR). I am using Jupyter Notebook in order to do so, which is something that is new to me and thus required time this week to set up and learn how to use. My EAR value, which belongs to a larger temporal window of a frame, will be given as input to the fully-trained SVM machine learning model.

Finally, another change I added was that I made my eye classification code simpler to run in terms of where the video input is found on the device the code is running on in anticipation of beginning Xavier testing of my code this Tuesday. I am currently slightly behind schedule since I hoped to have already tested my code on the Xavier, but I was able to complete the integration step mentioned before earlier than intended to make up for this. In the next week, I hope to finish training and then test on the Xavier starting Tuesday.

Sources

T Soukupova and Jan Cech. Real-time eye blink detection using facial landmarks. In 21st Computer Vision Winter Workshop (CVWW 2016), pages 1–8, 2016. https://vision.fe.uni-lj.si/cvww2016/proceedings/papers/05.pdf

Posted on

Vaheeshta’s Status Report for 3/13

For me, this week began with taking our TA Abha’s feedback on our draft of our design presentation slides and making changes accordingly. This included completely redesigning our user flowchart to show how the system turns on, specify what algorithms are running in parallel, and clarify how the driver response is checked. I also helped my team specify in our software block diagram what is off the shelf and what we are creating. 

Additionally, this week I started to test my eye classification code and made changes accordingly. I also began to properly document our code to make it as clear as possible for our debugging process. 

Finally, I worked on our Design Review Report with my teammates, as this is due next week. Thus, next week my main focus will be to complete the design report. I will then continue working on my eye classification algorithm and begin training with an already-existing dataset. My progress is on schedule.

Posted on

Vaheeshta’s Status Report for 3/6

This week, my accomplishments for our design presentation are that I created the hardware system specification block diagram, user story diagram, and the solution approach table. Additionally, I ordered all of our team’s parts on Monday, and some parts have already arrived. Most notably, I was able to obtain the TalentCell Rechargeable 12V battery pack from the ECE inventory, saving us money in our budget. Thus, we currently have about $156 to spare in our budget. 

Moreover, I started to work on the code for our eye classification using the Eye Aspect Ratio (EAR). For EAR, the proportion between height and width of the eye is calculated, with a higher EAR implying open eyes and smaller EAR implying closed eyes. Thus, EAR decreases when the eyes close, and return to a normal level when the eyes are open. EAR is the best option because it easily extracts a single feature from each image and reduces the amount of computational processing and storage memory required compared to other eye classification options. My code uses OpenCV, Dlib, Numpy, and SciPy libraries. It will be easy to integrate it with Heidi’s code with facial landmarking, since it will use the same facial landmark coordinates. This has been pushed to our Github repo at https://github.com/vaheeshta/focused.  

Additionally, I wanted to do more research into if there was a better way to detect drowsiness based on eye movement, other than just tracking if eyes are continuously closed for 2 seconds. For detecting drowsiness, I found a paper detailing a drowsiness pattern and ML model that does exactly this, using EAR, HOG, and SVM just like we are. Their drowsiness classification is based on two rules, the second of which being similar to what we are using. Also, I discovered a public database of human drowsiness called DROZY that I want to use for testing my eye classification. 

I am ahead of schedule, since Heidi and I decided to combine my eye detection task with her facial detection and facial landmarking task. Next week, I hope to test my eye classification algorithm and make necessary changes. 

https://www.sciencedirect.com/science/article/pii/S0957417420303298