status report – Page 3 – Team B1: FocusED

March 27, 2021March 27, 2021

Vaheeshta’s Status Report for 3/27

Since the previous status report, I worked on completing the Design Report with my teammates as well as adding to and testing our FocusEd codebase on my laptop. The first addition I made was that I integrated my eye classification code with Heidi’s HOG facial detection code. An image of this is shown below. While I completed this integration step earlier than I had designated on our schedule, it was a logical step to integrate the early version of two as soon as possible to then proceed to training and testing.

Next I worked on training of our eye classification model. Since every individual will have a different EAR threshold (which is the EAR value in our algorithm that determines if an individual’s eyes are closed), we wanted to train a model in order to accurately determine whether or not an individual’s eyes are closed or not. I am working on building a Support Vector Machine (SVM) classifier to detects long eye blinks as a pattern of EAR values in a short temporal window, as stated in T Soukupova and Jan Cech’s research paper “Real-time eye blink detection using facial landmarks” (this was the first paper that introduced EAR). I am using Jupyter Notebook in order to do so, which is something that is new to me and thus required time this week to set up and learn how to use. My EAR value, which belongs to a larger temporal window of a frame, will be given as input to the fully-trained SVM machine learning model.

Finally, another change I added was that I made my eye classification code simpler to run in terms of where the video input is found on the device the code is running on in anticipation of beginning Xavier testing of my code this Tuesday. I am currently slightly behind schedule since I hoped to have already tested my code on the Xavier, but I was able to complete the integration step mentioned before earlier than intended to make up for this. In the next week, I hope to finish training and then test on the Xavier starting Tuesday.

Sources

T Soukupova and Jan Cech. Real-time eye blink detection using facial landmarks. In 21st Computer Vision Winter Workshop (CVWW 2016), pages 1–8, 2016. https://vision.fe.uni-lj.si/cvww2016/proceedings/papers/05.pdf

March 27, 2021

Team Status Report for 03/27

The most significant risks that we have, especially with the interim demo coming up, is that integration of head pose and eye classification will not work/be difficult. Currently, both Heidi and Vaheeshta are working on the algorithms separately and are testing on their own laptops. If integration takes too long or we run into problems, it may push back the rest of our schedule. In order to manage these risks, our team is communicating with each other extremely well. Also, Heidi and Vaheeshta are both using the same landmarking algorithm so there is a point of reference that may make this integration easier. Danielle set up the Jetson this week, so we will continue testing on the Jetson and will test together to ensure that we can solve any issues that may arise.

There are currently no changes being made to the design of the system. Perhaps within the next couple of weeks, as we begin integration of the algorithms, this may change.

There have been some changes to the schedule following the advice of Professor Mukherjee. We are currently putting a pin in creating our custom dataset, and focusing more on making sure that our algorithms are working.

Danielle has set up the Jetson and configured the Raspberry Pi Camera, and Heidi has begun testing of her algorithms on the Jetson! Pictures below !!! 🙂

March 27, 2021

Heidi’s Status Report for 3/27

Since the last status report I worked with my group mates to finish and submit our design report. I have a version of the head pose algorithm. Based on a discussion at our weekly meeting with Professor Savvides and Abha, the initial approach of following examples that take 2D points and convert them to 3D points for head pose estimation was changed due to the conversion of 2D to 3D could cause a decrease in performance time and add unnecessary complexity to our project. Instead I am implementing the axis method we discussed with our professor that draws a x and y axis over the driver’s face with the origin being the landmark point of the tip of the nose. There is a calibration period, currently for the head pose I have it at 20 seconds, that the driver is prompted to stay still and an average of the facial landmarks are taken from each frame to determine the origin points. After that period is over, if the tip of the nose landmark is a negative distance from the origin, then they are labeled as distracted. I was working under an assumption that the user would be most still toward the last 5 seconds and was only calculating the average based on those last seconds. At the next weekly meeting with Professor Mukherjee and Abha, I described the progress of the algorithm and from additional feedback I will no longer be using that assumption and instead use all landmark detection in the 20 seconds window and doing a mean square average instead of a basic average calculation. I also added a threshold to the head pose code. This threshold will be updated once we have video of us driving. We also got the Jetson up and running and was happy to see that we were getting 10 fps without any gpu acceleration with the example algorithms I had from earlier and the current head pose code I have been working on. The updated code can be seen on our GitHub repo https://github.com/vaheeshta/focused.

I think my progress is on schedule. I made up the time from the last status report and did a quick change to a different method for head pose. Right now I have threshold and distracted versus normal classification just based on the y axis (up and down). I need to finish the left and right direction and I plan to complete that this weekend.

Next week I hope to test the current version of the head pose estimation on the Jetson in a controlled setting and then integrate with the eye classification algorithm.

March 27, 2021

Danielle’s Status Report for 03/27

During the week of 03/14-03/20, I worked rather extensively on the design report with my teammates. My sections included the Abstract, Introduction, Architecture Overview, and sections of the Design Trade Studies. I also spent time proofreading the report and editing the figures that we would include within it. During this week, I have spent time working on the Jetson to get it ready for my teammates to start integrating their algorithms and testing on the Jetson. Using NVIDIA’s startup guide, I did the first boot of the Jetson by writing the image onto the microSD that we purchased and configuring the Jetson. I also installed dlib on the Jetson, which we will be using for our project. Following this, I used the youtube video listed below to write a python script that would configure the Raspberry Pi V2 Camera Module with GStreamer to work on the Jetson. I ran into some issues with initially running the script because of an assertion failure, but after diagnosing the problem — I plugged in the camera while the Jetson was on — I was able to get the camera to start working. I am currently working on creating a static IP for the Jetson using the source below that will allow us to SSH into the Jetson and work on it that way.

I would say that my progress is on schedule. Following our conversations with Professor Mukherjee during our meeting on Monday, he suggested that we hold off on doing the custom datasets until we have our algorithms working at about 80%, because we may not need the extra data and do not want to waste time. We have readjusted our Gantt chart as a result of this, and I am focusing more on the integration currently as shown from my work above.

In the next day, I will have the static IP set up. During the week, I will be working on helping my teammates with integration. I will also work on the focus timer algorithm so we can integrate that with the other algorithms.

Sources:

March 13, 2021

Heidi’s Status Report for 3/13

This week I worked on the head pose example. The example I found works with input images and requires passing in the landmark vector to be able to calculate the angle the face is looking at. It uses the 2D points to create a 3D model of the face to determine where the person is looking. Unfortunately I have run into integrating the landmark example with this one for video input. I also tested to see how many fps my laptop webcam was getting for the face detection and landmark examples. These two I’m getting between 0.75 and 0.8 fps, which makes me hopeful for having enough fps with the power of the Xavier. I started working on the design report as well, contributing to the requirements section of the report.

I am a little behind schedule. I was hoping to have a head pose example working by today but ran into some issues getting it the head pose example to use the landmarks array from my previous example. Will have it done by the end of the weekend so I’ll be back on schedule quickly. Most of our equipment has arrived, so I can test the fps we will be getting for the examples developed last week.

This next week I hope to finish the head pose example and start working on building our custom dataset. From our feedback on the design presentation, professors and TA would like to see the beginning of the dataset. Also this next week, I will be working with my teammates on finishing the design report.

Sources:

https://learnopencv.com/head-pose-estimation-using-opencv-and-dli

March 13, 2021March 13, 2021

Team Status Report for 3/13

One significant risk that could jeopardize our project is calibration of our landmark detection. From the feedback we received on our design presentation, there was a question on would we need to calibrate our EAR algorithm and if we needed to calibrate every time the user turned on the system. To mitigate this, we will dedicate time this week when beginning to set up how we will take our picture for the data set to make sure that our design works with images we are getting.

From the design presentation and the positive feedback we received the design of the system has remained the same. We have been asked to color code like we did on the software block diagram what will be off the shelf and what we will be making ourselves. The switch to an audio output was well received so we will order a USB speaker to attach to the Xavier. Because of our daylight scope of our project, we had planned to take pictures in different lighting conditions but from the recommendation of the professor with the time constraint of the class our scope will change to daytime with good lighting conditions. This does not change our system design but will simplify creating our dataset.

No changes were made to our schedule this week.

We are on schedule. We will be focusing on the design report this week. The primary hardware, the Jetson Xavier arrived so we will be working towards creating our training data set and will build upon the working examples we have from last week.

March 13, 2021

Vaheeshta’s Status Report for 3/13

For me, this week began with taking our TA Abha’s feedback on our draft of our design presentation slides and making changes accordingly. This included completely redesigning our user flowchart to show how the system turns on, specify what algorithms are running in parallel, and clarify how the driver response is checked. I also helped my team specify in our software block diagram what is off the shelf and what we are creating.

Additionally, this week I started to test my eye classification code and made changes accordingly. I also began to properly document our code to make it as clear as possible for our debugging process.

Finally, I worked on our Design Review Report with my teammates, as this is due next week. Thus, next week my main focus will be to complete the design report. I will then continue working on my eye classification algorithm and begin training with an already-existing dataset. My progress is on schedule.

March 13, 2021

Danielle’s Status Report for 03/13

This week, I worked primarily on prepping the design presentation. I created speaker notes and met with my team in order to practice presenting and making sure we practiced answers to potential questions. Following the presentation, I began working on the design report. We are using Overleaf, a LaTeX platform, in order to write the design report simultaneously. I created a base outline of things that we need to make sure we touch on in every portion of the report so we can make it a cohesive paper and not leave anything out. I have written several of the baseline sections and will continue to work on it in the next few days.

The group is on schedule. We have been working diligently all week on the design presentation. As for my own progress, I am on schedule since I spent all of last week prepping for the design presentation, and continuing on the report. We will be meeting tomorrow to fully discuss the feedback that we received from Abha and Professor Savvides so we can discuss at our weekly meeting, and I can incorporate such suggestions into my sections of the design report.

In the next week, we will have the design report completed and submitted. I will also be capturing our custom dataset and beginning training.

March 6, 2021March 7, 2021

Vaheeshta’s Status Report for 3/6

This week, my accomplishments for our design presentation are that I created the hardware system specification block diagram, user story diagram, and the solution approach table. Additionally, I ordered all of our team’s parts on Monday, and some parts have already arrived. Most notably, I was able to obtain the TalentCell Rechargeable 12V battery pack from the ECE inventory, saving us money in our budget. Thus, we currently have about $156 to spare in our budget.

Moreover, I started to work on the code for our eye classification using the Eye Aspect Ratio (EAR). For EAR, the proportion between height and width of the eye is calculated, with a higher EAR implying open eyes and smaller EAR implying closed eyes. Thus, EAR decreases when the eyes close, and return to a normal level when the eyes are open. EAR is the best option because it easily extracts a single feature from each image and reduces the amount of computational processing and storage memory required compared to other eye classification options. My code uses OpenCV, Dlib, Numpy, and SciPy libraries. It will be easy to integrate it with Heidi’s code with facial landmarking, since it will use the same facial landmark coordinates. This has been pushed to our Github repo at https://github.com/vaheeshta/focused.

Additionally, I wanted to do more research into if there was a better way to detect drowsiness based on eye movement, other than just tracking if eyes are continuously closed for 2 seconds. For detecting drowsiness, I found a paper detailing a drowsiness pattern and ML model that does exactly this, using EAR, HOG, and SVM just like we are. Their drowsiness classification is based on two rules, the second of which being similar to what we are using. Also, I discovered a public database of human drowsiness called DROZY that I want to use for testing my eye classification.

I am ahead of schedule, since Heidi and I decided to combine my eye detection task with her facial detection and facial landmarking task. Next week, I hope to test my eye classification algorithm and make necessary changes.

https://www.sciencedirect.com/science/article/pii/S0957417420303298

March 6, 2021March 7, 2021

Team Status Report for 3/6

One significant risk that could jeopardize our success is training. It is possible that we underestimated the time required to complete our training. We foresee possible issues related to varying conditions such as different daylight lighting, various positions of our camera on car dashboards due to different dashboard heights, and distinct eye aspect ratios of various individuals. During our team meetings this week, we tried to manage this risk by adding more time to training when we updated our schedule, which is discussed more below.

After feedback from our Monday meeting with Professor Savvides, we decided to change our driver alert system. Originally, we planned to have a Bluetooth-enabled vibration device that would alert the driver if they were distracted or drowsy. Our main components for this was a Raspberry Pi and a vibration motor. However, after talking with Professor Savvides and our TA, we found that this would not be reasonable in our time frame. Therefore, we eliminated the vibration device and replaced it with a speaker attached to our Jetson Xavier NX. This significantly shifted our hardware block diagram, requirements, system specifications, and schedule.This shift towards an audio alert system did reduce our costs as well.

We have attached the updated schedule below that accounts for our recent changes in our design. We were able to consolidate tasks after taking out the tasks related to the vibration device, so that each team member still completes an equal amount of work. In our updated plan, we decided to add more time related to training, integration, and testing our several metrics.

We are slightly ahead of schedule, since we have a simple working example of face detection and a simple working example of eye classification. They can be found at our Github repo, https://github.com/vaheeshta/focused.