Author: hbatres

This past week we had our final presentation. Afterwards I spent time debugging our script and a permission error on the Jetson with Vaheeshta. We wanted to have our python script run on boot, however, since we were accessing camera, audio, and writing files we had some sudo permission errors. After switching to a usb webcam Vaheeshta and I were able to get the on boot script to work. The issue was in the folder we had our script. We checked that it worked with the Raspberry Pi camera module as well. Additionally, we improved our frames per second from about 8 fps to 15 fps. This was accomplished by decreasing the size of the cv2 window the Jetson was seeing. However, if the frame was under a width of 300, the face was too small for detection so our adjustments for videos and testing was to make sure the frame width was over 300. While Vaheeshta and Danielle focused on the final video editing, I finished the poster for the public demo next week.

Progress is on schedule. We have completed our MVP so we just have to focus on the final deliverables.

This next week, I will be working with my team to finish up the video which we will present at our public demo and will work on the final report.

This past week I added more images to the head pose estimation. When testing on the jetson this week, we realized that it is more sensitive when dealing with all four directions. With Danielle and Vaheeshta’s help I added more front facing photos to help provide a wider range of “forward” facing positions for the driver to not be alerted at the smallest change. Additionally I changed the calculation for the head pose ratio. When testing in the car, the distance of the driver’s face from the camera impacted the reliability of estimation. Instead, now it is doing the left cheek area divided by the right cheek area to have a ratio that is independent of size of the face. It is still more sensitive than we would like, but with the metrics we collected I am happy with it’s estimation. Using sklearn’s accuracy score calculations, I ran the model with 80/20 and 50/50 train/test split. Additionally, using the random state variable which shuffles the photos this helped test for robustness of faces used for training. I was able to get an average of 93% accuracy for head pose estimation. The lowest was 86% and this was with fewer photos and a random state of 42. This makes sense as the range of photos for each direction was less so increasing the shuffling variable affects the accuracy more. In combination with Vaheeshta’s eye classification, we now have a complete system. I also worked on creating a systemd service to run a bash script on boot for the jetson however as mentioned in the team status report, I ran into permission issues with gstreamer and we spent time as a team debugging but prioritized gathering our metrics. 

While some of my work was redone progress is still on schedule. We completed our final presentation and the major blocks of our project.

This next week, I will be working with my team on the video demo and poster in preparation for the public demo’s coming up. With feedback from the presentation on Monday/Wednesday we can make a good outline for our final report as well.

After some trial and error, and advice from Professor Savvides I have implemented the new version for head pose calibration and classification. Following Vaheeshta’s example for eye calibration and classification I was able to do the same for head pose. I was struggling at first because I was attempting to use the entire landmark vector of 68 points. From our meeting on Wednesday, Professor Savvides explained that similarly to the EAR ratio the eye calibration and classification was using, I needed a ratio for my implementation. For this ratio I did the area of the left and right cheeks. If the driver looks to the right, the area of the left cheek is larger on the screen and vice versa for the left side. Right now I just have two classes, forward and right. Now that this version is working, I will also add left, up, and down directions. Additionally this week I worked on integrating my previous head pose algorithm with her eye classifier. We now have a working version of our code combined on the Jetson which is exciting. I also began the ethics assignment. Updated code has been pushed to our Github https://github.com/vaheeshta/focused.

My progress is on schedule. I will be finishing up the new version of the head pose between today and tomorrow. 

This next week, we will practice for our demo on Monday and implement the feedback we receive on Monday. Also I will be adding the left, up and down directions and finishing the ethics assignment.

After our meeting on Monday with Professor Savvides and Abha, when I was showing the progress of the head pose code, a background object in my laptop webcam was detected as a face. I added a check that when the face detector from dLib is looking for faces, it only chooses one face and that one face is the largest one in the frame. I tested this with Vaheeshta and I both in the frame, where her face in the video was smaller than mine. Her face was not detected, and only my face was. When she came closer and our faces were about the same size then the detector would switch between the two of us. Additionally, I compared my current method of calibration with that of the eye classifier coded by Vaheeshta. Her method collects vectors from the user’s face to train a classifier based on the person’s vectors to determine if the user is drowsy or not. My method was just having the user hold still and create a mean facial landmark vector to create the axis. I am working on implementing her method of calibration and training to simplify the integration of the head pose and eye classifier. My classifier will detect down, right, forward positions to start and, once this proves successful, I will then add up and left positions.

My progress is still on schedule. The head pose algorithm is in a good place and the updated calibration implementation will be completed this weekend. I was not able to test on the Jetson this week so I will test the new calibration implementation and head pose this coming week. 

This next week, I will work to integrate the head pose algorithm with Vaheeshta’s eye classification algorithm and finish the new calibration and training method. At the least we want our calibration steps to be integrated by interim demo.

Since the last status report I worked with my group mates to finish and submit our design report. I have a version of the head pose algorithm. Based on a discussion at our weekly meeting with Professor Savvides and Abha, the initial approach of following examples that take 2D points and convert them to 3D points for head pose estimation was changed due to the conversion of 2D to 3D could cause a decrease in performance time and add unnecessary complexity to our project. Instead I am implementing the axis method we discussed with our professor that draws a x and y axis over the driver’s face with the origin being the landmark point of the tip of the nose. There is a calibration period, currently for the head pose I have it at 20 seconds, that the driver is prompted to stay still and an average of the facial landmarks are taken from each frame to determine the origin points. After that period is over, if the tip of the nose landmark is a negative distance from the origin, then they are labeled as distracted. I was working under an assumption that the user would be most still toward the last 5 seconds and was only calculating the average based on those last seconds. At the next weekly meeting with Professor Mukherjee and Abha, I described the progress of the algorithm and from additional feedback I will no longer be using that assumption and instead use all landmark detection in the 20 seconds window and doing a mean square average instead of a basic average calculation. I also added a threshold to the head pose code. This threshold will be updated once we have video of us driving. We also got the Jetson up and running and was happy to see that we were getting 10 fps without any gpu acceleration with the example algorithms I had from earlier and the current head pose code I have been working on. The updated code can be seen on our GitHub repo https://github.com/vaheeshta/focused.

I think my progress is on schedule. I made up the time from the last status report and did a quick change to a different method for head pose. Right now I have threshold and distracted versus normal classification just based on the y axis (up and down). I need to finish the left and right direction and I plan to complete that this weekend. 

Next week I hope to test the current version of the head pose estimation on the Jetson in a controlled setting and then integrate with the eye classification algorithm.

This week I worked on the head pose example. The example I found works with input images and requires passing in the landmark vector to be able to calculate the angle the face is looking at. It uses the 2D points to create a 3D model of the face to determine where the person is looking. Unfortunately I have run into integrating the landmark example with this one for video input. I also tested to see how many fps my laptop webcam was getting for the face detection and landmark examples. These two I’m getting between 0.75 and 0.8 fps, which makes me hopeful for having enough fps with the power of the Xavier. I started working on the design report as well, contributing to the requirements section of the report. 

I am a little behind schedule. I was hoping to have a head pose example working by today but ran into some issues getting it the head pose example to use the landmarks array from my previous example. Will have it done by the end of the weekend so I’ll be back on schedule quickly. Most of our equipment has arrived, so I can test the fps we will be getting for the examples developed last week. 

This next week I hope to finish the head pose example and start working on building our custom dataset. From our feedback on the design presentation, professors and TA would like to see the beginning of the dataset. Also this next week, I will be working with my teammates on finishing the design report.

Sources:

https://learnopencv.com/head-pose-estimation-using-opencv-and-dli