Posted on

Danielle’s Status Report for 04/24

During this week, I worked on the audio prompts. I downloaded Audacity because it was the best platform I found for recording audio to convert into either a .wav or .mp3 file. I recorded the audio for start up, all of the calibration, as well as the refocus prompt, and reduced the noise in each of the files to make sure they are audible. I also made sure to include in our audio prompts some of the things that we discussed from the ethics discussion about being transparent to users, such as mentioning when recording will take place or warnings about driving and calibration. I researched about the best module for playing audio, and it seemed that the playsound module was the best. However, using it on its own without threading made the sound distorted and made the program unresponsive. Thus, using the source listed below, I was able to add threading to our program to make the audio work as intended without stopping the program. I also added threading to other aspects of our project. I began on GPU acceleration but haven’t made much progress because I was sick. I also met with my group today to discuss what we need to do going forward in the next week before our final presentation.

I am still on schedule, but I would have liked to accomplish more of the GPU acceleration than I did. I got my second Covid shot a couple of days ago, and I’ve been really incapacitated for the past couple of days feeling the effects of the vaccine. During this week, we’ll be working exclusively on the project to get things ready for the final presentation.

This next week, I will have GPU acceleration done and we will be testing within my car to gather metrics for our presentation.

Sources:

Posted on

Heidi’s Status Report for 4/24

I have continued working on the head pose algorithm. I was able to get 4 directions to be identified, left, right, forward and down. As our professor suggested the hardest part was differentiating between forward and down. I did several iterations for the ratio calculated for the SVM classifier. The method that finally worked was to add another triangle to the equation. For the interim demo I had two triangles, one for each side of the face over the cheek area and I would subtract the area of each and divide by 2. This gave a good distinction for forward and right poses. Then when I added left this method still seemed to work. But to include the down direction, I added a third triangle that looked at the area between the nose, left jaw and right jaw. This new method takes the left cheek area subtracts the right cheek and divides by 2 just like before but then also subtracts the new triangle area. This ratio gives an even better distinction between the poses than before. I first got this version working on my laptop and then tested on the jetson and experienced similar results. I tested this on myself and a friend.

My progress is still on schedule, although now we are towards the end and have to pick up the pace. I will be pushing to get more data for the head pose to be as robust as possible for the final demo.

This next week, I will be collecting more head pose data to add to our pretrained model. Will also be working with the rest of my groupmates to collect our metrics for the presentation coming up.

 

Posted on

Danielle’s Status Report for 04/10

During this week, I focused primarily on creating the focus timers for both head pose estimation and the eye classification. I experienced several problems working with dlib and OpenCV on my desktop computer, so I had to go through the whole installation process again on my laptop which seemed to clear up any issues I was having, and I was able to test the code on my laptop. For eye classification, I created the code so that after 1 second of closed eyes, it will indicate on the screen that the driver is distracted. For the head pose estimation focus timer, after 2 seconds of a distracted head pose, a message will pop up on the screen, indicating that the driver is distracted. Following testing of the focus timers separately, I integrated the focus timers within our main code file and tested. I also began to write the code for audio prompts for calibration that I will finish this week. I met with my group several times this weekend in person to test the integrated code on the Jetson and prepare for the interim demo. I also started working on the ethics assignment. I also ordered other parts that we needed such as the Male to Male connector that was missing from the power bank package we got from the 18-500 inventory.

I would say that my progress is on schedule! I implemented both focus timers as planned, helped with the integration process, and began working on the code for the audio prompts for calibration which is what I had planned to complete for the week.

In the next week, I hope to have the audio prompts for calibration finished with the recordings of the audio, and I will be working on integrating the custom datasets with the LFW dataset. I will also be aiding my team with implementing any of the feedback we get from the interim demo and completing the ethics assignment.

Posted on

Heidi’s Status Report for 4/10

After some trial and error, and advice from Professor Savvides I have implemented the new version for head pose calibration and classification. Following Vaheeshta’s example for eye calibration and classification I was able to do the same for head pose. I was struggling at first because I was attempting to use the entire landmark vector of 68 points. From our meeting on Wednesday, Professor Savvides explained that similarly to the EAR ratio the eye calibration and classification was using, I needed a ratio for my implementation. For this ratio I did the area of the left and right cheeks. If the driver looks to the right, the area of the left cheek is larger on the screen and vice versa for the left side. Right now I just have two classes, forward and right. Now that this version is working, I will also add left, up, and down directions. Additionally this week I worked on integrating my previous head pose algorithm with her eye classifier. We now have a working version of our code combined on the Jetson which is exciting. I also began the ethics assignment. Updated code has been pushed to our Github https://github.com/vaheeshta/focused.

My progress is on schedule. I will be finishing up the new version of the head pose between today and tomorrow. 

This next week, we will practice for our demo on Monday and implement the feedback we receive on Monday. Also I will be adding the left, up and down directions and finishing the ethics assignment.

Posted on

Team Status Report for 4/10

One risk that could jeopardize our success is when we implement GPU acceleration to remove any delay of the system. It is important to be as close to real-time as possible. Right now we noticed a small delay when we integrated the head pose and eye classification code. Our plan for this is to get it as close to realtime first on the CPU and then implement GPU acceleration using CUDA Python and following the NVIDIA walk through. Additionally, for our testing we have been using a monitor to visually see how our algorithms are working so when we remove the cv2.show calls this should also improve the delay we see. For the interim demo we will most likely have this delay still but are working on minimizing it by checking the frames per second.

There have not been any significant changes to our existing design since last week. We have just been working on integration. To improve our integrated design, the calibration for head pose will resemble the calibration for eye classification. The ratio used will be the area of the cheek areas of each side of the face to determine whether the driver is looking right, left, forward, etc.

Some changes were made to the Gantt chart. We updated it with the actual dates we integrated the two algorithms and added additional tasks to ensure our MVP will be ready to test in a car setting.

Our updated Gantt chart is below.

 

Gantt Chart pdf

Posted on

Vaheeshta’s Status Report for 4/10

This week, I successfully integrated all my code with Heidi’s and Danielle’s code to create our working demo. We all met together, in-person, on Friday and Saturday to make sure that this happened smoothly. I also set the integrated code up on the Jetson Xavier NX, with Danielle and Heidi’s help, and tested it. Therefore, for our demo we can now show the integrated system working on the Jetson Xavier NX. When integrating the eye classification and head pose estimation systems together, I made sure that our calibration sequences happen sequentially when the user boots up the system: The system first asks the user to calibrate head pose for 10 seconds, then it asks the user to keep their eyes open for 10 seconds for calibrating eye open classification, and finally it asks the user to keep their eyes closed for 10 seconds for calibrating eye closed classification. Thus, at this moment 30 seconds are spent for calibration when the system is turned on. Additionally, when testing on the Jetson Xavier, I faced issues with updating several libraries, but with Heidi and Danielle’s help we were able to fix this.  

My progress is, for the most part, on schedule. Along with integration, I also was hoping to test the power bank with the Xavier this week, but upon opening the power bank we received from the 18-500 inventory I noticed that the kit was missing the male-to-male DC connector that connects the power bank to the Xavier. Thus, Danielle ordered us a new one that should arrive next week for me to test. Therefore, next week I will be testing the power bank and working with my teammates to implement any feedback from the demo. 

 

Posted on

Danielle’s Status Report for 04/03

During this week, I set up the static IP on my local network so that Heidi and Vaheeshta are able to work at my apartment concurrently. This week, I also worked on the focus timer for Heidi’s Head Pose Estimation that still needs to be tested this coming week. Following our meeting with Professor Savvides and Abha, Professor Savvides suggested that we actually should take some of our own custom datasets so we are not blindsided if we need them in the future. Thus, I took images of myself and a friend in my driveway simulating normal driving and distracted driving behaviors that we may need. Following talking with Heidi and Vaheeshta, we realized that integration may be difficult and we underestimated the amount of time needed, so we decided that we want to python threading. Since we all have little experience with this, I did some research on python threading and found a couple tutorials that we can potentially use to accomplish what we need to. Also, we have decided on the audio prompts that we wish to use for calibration, so I have done some research and plan on writing the code this week.

I am mostly on schedule. Things have been slightly difficult because I experienced some side effects (i.e. extreme fatigue) following my vaccine on Wednesday for a couple of days, and because I am not currently in Pittsburgh to test on the Jetson. Once I get back to Pittsburgh on Monday, I plan to immediately start testing the focus timer on Heidi’s head pose estimation.

In the next week, I hope to have the focus timer tested on the Jetson because I am having some difficulty testing it currently on my laptop when I usually have my desktop computer at my apartment. I also will be writing the code for the audio prompts for the calibration steps and testing it with Vaheeshta.

Sources:

 

Posted on

Team Status Report for 4/3

At this moment, one risk that could jeopardize our success is that we may not have given ourselves enough time for adding concurrency to our integration of our drowsiness detection system and head pose estimation system, as well as testing it. We want to use Python threading to have our eye classification and head pose estimation run at the same time after the facial landmarking. Since we all do not have much experience with concurrency, we may have underestimated the challenges we may face with this. Our contingency plan is that we will not use concurrency, and instead do the EAR calculations and head pose classifications linearly, and then determine if the driver is drowsy. 

One change we made is that we are only going to use our own custom datasets for training of eye classification. Thus, we will only be using public datasets for testing. For instance, we will use DROZY, a drowsiness database, for testing our drowsiness detection. Additionally, another change we made is that, after comparing our calibration systems for head pose and eye classification, we decided to model the head pose calibration after the eye classification calibration. Therefore, each will use the calibration time to generate vectors to then train separate SVM models. This way, we can make our head pose system more accurate for the user. Additionally, we clarified what type of audio prompts we would like to use during calibration, so that our user experience will be as smooth as possible for this crucial step. We want the user to perform our calibration requests while still not inconveniencing the driver with repetitive, and thus seemingly excessive, calibration requests.

 

Posted on

Heidi’s Status Report for 4/3

After our meeting on Monday with Professor Savvides and Abha, when I was showing the progress of the head pose code, a background object in my laptop webcam was detected as a face. I added a check that when the face detector from dLib is looking for faces, it only chooses one face and that one face is the largest one in the frame. I tested this with Vaheeshta and I both in the frame, where her face in the video was smaller than mine. Her face was not detected, and only my face was. When she came closer and our faces were about the same size then the detector would switch between the two of us. Additionally, I compared my current method of calibration with that of the eye classifier coded by Vaheeshta. Her method collects vectors from the user’s face to train a classifier based on the person’s vectors to determine if the user is drowsy or not. My method was just having the user hold still and create a mean facial landmark vector to create the axis. I am working on implementing her method of calibration and training to simplify the integration of the head pose and eye classifier. My classifier will detect down, right, forward positions to start and, once this proves successful, I will then add up and left positions.

My progress is still on schedule. The head pose algorithm is in a good place and the updated calibration implementation will be completed this weekend. I was not able to test on the Jetson this week so I will test the new calibration implementation and head pose this coming week. 

This next week, I will work to integrate the head pose algorithm with Vaheeshta’s eye classification algorithm and finish the new calibration and training method. At the least we want our calibration steps to be integrated by interim demo.

Posted on

Vaheeshta’s Status Report for 4/3

This week, I was successfully able to complete and test my code for creating and training our eye classification model using SVM. To create and train this SVM model, I first have a calibration system where the user follows prompts to keep their eyes open for a period of time, in which I calculate their various eye aspect ratios and create a dataset of vectors based on this. Then, I repeat this with the user’s eyes being closed. Afterwards, I use this data to train a SVM classifier for that specific person. My SVM classifier has two labels: 0, which classifies the eyes as open, and 1, which classifies the eyes are closed. It is important to note that I am only using our custom calibration data for my training my model, and not data from a database, since each person has unique thresholds. Then, during my real-time eye classification, I use this trained model to determine if the user’s eyes are closed or not. If the user’s eyes are closed for at least 1 second, my code detects that the user is drowsy. If the user is detected to be drowsy, I currently have a visual alert, which will later be changed to an audio alert. I have a screenshot of this below. So far, with testing with Heidi and myself, my model has been accurate. I am in the process of testing with DROZY, a drowsiness database, to see how accurate my overall drowsiness detector is. The next action for this code is to hand it over to Danielle, who will add the focus timer, audio prompts for calibration, and audio alerts for drowsiness being detected. 

I am currently on schedule. Next week, I will work with Heidi to integrate her head pose algorithm with my eye classification algorithm, so it is in one seamless file to run on the Jetson. If we cannot complete this in time before the demo, we will just have our calibration mechanisms integrated. 

Resources:

DROZY: http://www.drozy.ulg.ac.be/