Posted on

Danielle’s Status Report for 03/06

This week, I worked on getting our design presentation ready. I will be presenting, so I have been working on speaker notes as well. I’ve met with my team multiple times to reassess our project, order hardware, re-evaluate our Gantt chart, etc. I’ve spent most of my time with the presentation itself and editing the slides, such as the implementation plan and problem area. I have also spent time evaluating the sources that my teammates have found to begin creating algorithms. Since we are no longer creating a vibration device for an alert and will instead be doing a sound alert, I have spent some time looking into the logistics and assessing how to accomplish this.

The group is definitely on schedule. We have been working diligently all week on the design presentation. As for my own progress, I am generally on schedule. I would have liked to have completed my speaker notes to the point where I would only need to adjust them slightly before the presentation, but, due to some family circumstances, I have not finished them completely yet. In order to catch up, I will be spending Sunday getting ready for the presentation and creating speaker notes.

In the next week, we will have presented the design presentation and will have the design report mostly completed so we can continue to edit. I will also be aiding my teammates as we begin the software portion of our project.

Posted on

Heidi’s Status Report for 3/6

This week I worked with my team members on the design presentation. From last week, I was able to follow along with an existing example to get a simple face detection python script to work. The script uses OpenCV to read in the frames from my laptop webcam and using dlib’s HoG Face Detector. Additionally, with the face detector working I was able to also continue and follow a facial landmark example using dlib shape predictor. From observation, detection works pretty well, head pose does effect detection, if I look down or up the bounding box no longer appears over my face. The dataset that dlib’s face detector uses is the LFW (Labeled Faces in the Wild). This dataset has 2585 images. Last week I was looking at the CMU Multi-pie face dataset and was hoping to train the dlib face detector with this one instead. This dataset is extremely large with about 750,000 images of 337 people with 19 illumination conditions. As I’m not sure if we will be able to use this dataset (since we would require a license), this data set and the Yale face database did help consider how we would label our own dataset, to build upon the LFW dataset. We can classify images as either center-light, left-light, or right-light which is what the Yale face database does, but we could also classify as bright-light, normal-light, and low-light. I also looked at an example of face landmarking training that uses dlib for a custom shape predictor.  Both code examples were uploaded to our team’s GitHub repo.

Progress is still on schedule. We ordered our equipment on Monday and in the meantime I am trying to play around with pre-trained models to see how the face detection would work with landmarking and continue to research how we will be training our models. Design presentation is on Monday/Wednesday and based on the feedback we receive we can start writing the design report. 

This next week I hope to use this face detection script but with a different dataset to practice training a model with a new dataset. The dataset I plan to train with is the DriveFacce or the Yale face database. Additionally we can hopefully build on top of these and create our custom dataset following this format, especially when considering different light conditions. Also this next week, I will be working with my teammates on the design report.

 

Sources:

Face Detection Example

https://learnopencv.com/face-detection-opencv-dlib-and-deep-learning-c-python/

Face Landmark Model

https://github.com/davisking/dlib-models/blob/master/shape_predictor_68_face_landmarks.dat.bz2

Face Datasets

http://vis-www.cs.umass.edu/lfw/

https://search.wellspringsoftware.net/opportunity/FBX2:4283?qd=CMU%20Multi-pie

http://adas.cvc.uab.es/elektra/enigma-portfolio/cvc11-drivface-dataset/

Dlib official Face Landmark Example:

http://dlib.net/train_shape_predictor.py.html

http://dlib.net/imaging.html#shape_predictor

Training Example:

http://dlib.net/train_object_detector.py.html

 

Posted on

Danielle’s Status Report for 02/27

This week, I worked on getting our project proposal presentation ready and researching some of the justifications for our metrics. Since I am assigned to work on the vibration device for our project, I found a couple tutorials (links listed below) for using a Grove Vibration motor with a Raspberry Pi. Going through these tutorials, I tried to assess the code and see the basis of what I will have to do in order to create a successful vibration device. I do not have much experience with RPi, so I will be researching similar tutorials to begin planning the design. I was able to also find a circuitio page that shows how to properly connect the vibration motor to the RPi, so I believe this will help when we begin constructing the project.

I would say that the group is on schedule. We have discussed our plans following the proposal presentation and have reassessed some of our design plans which we wish to discuss during our meeting on Monday. We also wish to begin ordering supplies at the beginning of this week to truly begin working on the project. As for my own progress, I would say that my progress is slightly behind schedule. I spent some time researching, but I hoped to put in more time this week. To catch up, I will spend the next couple of days focusing my efforts on researching and preparing for the design report.

In the next week, I hope to have the design report and presentation ready fairly early so we can focus on editing as needed. I will be giving the Design Presentation, so I will be spending time practicing to ensure that everything goes smoothly and that our research is solid to justify our design and metrics.

Sources:

 

Posted on

Vaheeshta’s Status Report for 2/27

This week our group met after Wednesday’s proposal presentation to discuss our next steps. Specifically for me, my next steps were to finish researching eye detection algorithms and to select the best option to then begin working on programming our eye detection over this weekend and during the next week. I hope to then finish on schedule by 3/4/2021 to begin testing. Therefore, I researched and solidified that we would be using the eye detection algorithm of Dlib’s Histogram of Oriented Gradients (HOG) and Linear Support Vector Machines (SVM) method for object detection. The reason for this is that, according to the Towards Data Science article and other tutorials listed below, HOG is a better option compared to Dlib’s CNN algorithm since it has a faster computation time for real-time video processing. However, there is a tradeoff that HOG is slightly less accurate than CNN for non-frontal faces, but we are aiming to place our camera with a direct frontal perspective of the driver to help compensate. 

During our team meeting, Heidi and I decided to use the same algorithms for facial and eye detection, thus not relying on two different simultaneous algorithms to compute facial and eye detection, when the facial detection will already be determining the locations of the eyes. 

Then, I set up our codebase through a private Github repository and began working on setting up Dlib. I also started reading the user manual for the NVIDIA Jetson Xavier NX Developer Kit. Finally, I worked on perfecting our budget so that we can order all our necessary parts by Monday. Since we are using a much more expensive microprocessor ($99 NVIDIA Jetson Nano vs. $399 NVIDIA Jetson Xavier), we need to make sure that we stay under our $600 total budget.  

By next week, I hope to have my eye detection code working with preliminary testing, all pushed to the repository. This includes finding datasets for training. Then, I want to get started with eye classification to determine whether the eye is opened or closed. I will use the Eye Aspect Ratio (EAR) equation for this. 

Sources

Why HOG

 

https://maelfabien.github.io/tutorials/face-detection/#3-real-time-face-detection-1

https://towardsdatascience.com/cnn-based-face-detector-from-dlib-c3696195e01c

https://www.pyimagesearch.com/2014/11/10/histogram-oriented-gradients-object-detection/

 

Eye Detection

https://www.educative.io/edpresso/eye-blink-detection-using-opencv-python-and-dlib

https://www.pyimagesearch.com/2017/04/24/eye-blink-detection-opencv-python-dlib/

https://towardsdatascience.com/a-guide-to-face-detection-in-python-3eab0f6b9fc1

 

NVIDIA Xavier User Guide

https://developer.download.nvidia.com/assets/embedded/secure/jetson/Xavier%20NX/Jetson_Xavier_NX_Developer_Kit_User_Guide.pdf?a89SNG_TxN9K0wiYbBWmdVF5WdJvtGJEvS–YZxbCnhAurtz2dqKfuxi93w_rhKTroNbZb0Eu41atCdOGAqSVw02a2m7EphpIa4rIVu325GWAvCe1wMA5fBO2H6BgZb0rD0xy5jzxr5XfH9BHgKmihXzbslTwi7mEJ5k3DOA7xcuWxk8N9FofDfeNSo_kE0bbfDDOjwR

Posted on

Team Status Report for 02/27

The most significant risks that could jeopardize the success of the project is that, based on professor feedback from our proposal presentation, we have decided to use the NVIDIA Jetson Xavier NX Developer Kit instead of the Nano as previously decided. The biggest risk associated with this is the cost, namely that it costs $399, which is significantly higher than the Jetson Nano which was $99. This eats into our overall budget pretty heavily and does not allow room for any mistakes to occur. If, for some reason, the Xavier were to somehow become usable, we would have to re-evaluate how to proceed with our contingency plans. To lower the risk of damaging the Jetson Xavier in some way, all of us will educate ourselves on how to properly handle it. We will also be storing it at Danielle’s apartment since the carpet in Heidi and Vaheeshta’s apartments pose a significant ESD risk, as well. Our current contingency plan is to try to find a cheaper, used Jetson Xavier option on a resale website such as eBay. We will discuss with our TA, Abha, about any other suggestions that she may have for contingency plans.

We have made several changes to the existing design of the system, following our Project Proposal presentation from Wednesday. We met to take into account the feedback that we received and think about how it would affect our project. As stated above, we have decided to use the NVIDIA Jetson Xavier NX Developer Kit instead of the Nano. We have also decided to adjust the scope by removing eye detection as one of our requirements, since it combines with facial detection, and instead using DLib 68 points landmarking. We also intend to look into adjusting our landmarking accuracy by researching more based on Professor Savvides’ comments following our presentation.

None of our changes have affected the schedule of the project, so there are no updates.

Posted on

Heidi’s Status Report for 2/27

This week I practiced for the proposal presentation on Wednesday. Based on the feedback we received about upgrading from the Jetson Nano to the Jetson Xavier NX, I went back and did some more research between the two microprocessor options. The Xavier NX has a plug-in WLAN and Bluetooth module with antennas which would save us the cost of buying an Intel Wifi Card to add onto the Nano. From the introduction video, one of the demo’s they showed included Gaze estimation which used NVIDIA facial landmarks which uses the CMU Multi-PIE Face Database. This gave a bit more clarification for me on the region keypoint pixel error that was mentioned during our proposal presentation for facial landmarking metrics. From our feedback on our status reports, I moved away from research papers to project examples. I looked into an example with OpenCV, Dlib and Python to begin experimenting with facial detection and landmarking. I have downloaded the necessary packages, including Dlib and its requirements to my laptop to begin working. There were various updates I needed to add to my laptop for Dlib to properly install. I also looked into data sets for head poses, as an option instead of our data collection. The DriveAHead data set and UAH-DriveSet seem like they will be the best for our needs, particularly the UAH-DriveSet.

 

Progress is on schedule. The proposal presentation went well and we have discussed as a group how to address the suggestions we received and will bring this up in our meeting on Monday with Abha. Our plans to order immediately after the presentation were pushed back because we received the suggestion to upgrade to Xavier NX but after discussing with Abha we should be able to submit the order on Monday.

 

This next week, we will order the Xavier NX and other hardware components. I will work with my teammates to prepare our design presentation and ensure that we have updated our requirement metrics. I hope to also have a simple facial detection python script working with Dlib and OpenCV. 

 

Sources:

NVIDIA Jetson Xavier NX Developer Kit

https://www.jetsonhacks.com/2020/05/16/nvidia-jetson-xavier-nx-developer-kit/

https://developer.nvidia.com/embedded/learn/get-started-jetson-xavier-nx-devkit#intro

Facial Landmarks Estimation

https://ngc.nvidia.com/catalog/models/nvidia:tlt_fpenet

Facial landmarks with Dlib, OpenCV, and Python

https://www.pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/

https://towardsdatascience.com/facial-mapping-landmarks-with-dlib-python-160abcf7d672

Dlib Setup

https://stackoverflow.com/questions/54719496/installing-dlib-in-python-on-mac

Data set

http://www.robesafe.uah.es/personal/eduardo.romera/uah-driveset/

https://cvhci.anthropomatik.kit.edu/download/publications/2017_CVPR_Haurilet.pdf

The CMU Multi-PIE Face Database

http://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html

 

Posted on

Vaheeshta’s Status Report for 2/20

This week for me involved working on our project proposal and researching eye detection and classification methods. My main focus on our project proposal was researching and defining our testing, verification, and metrics for our nine requirements. I also dove into learning the basics of facial landmarking and eye detection as well as researching various eye detection methods. Some eye detection algorithms I identified in various research papers and online literature are as follows: Peng Wang, Matthew B. Green, and Qiang Ji’s Haar wavelet and AdaBoost algorithm that has a 94.5% eye detection accuracy; the Viola-Jones algorithm from the OpenCV library; D. Sidibe, P. Montesinos, S. Janaqi’s algorithm with a 98.4% detection rate; and Dlib’s Convolutional Neural Networks. Since computer vision is a very new topic for me, I taught myself an introduction to Haar cascade classifiers using OpenCV, some basics of Histogram of Oriented Gradients (HOG) using Dlib, and Convolutional Neural Networks (CNN) using Dlib. For instance, Dlib’s facial landmarking detector produces 68 (x, y) coordinates that map to specific facial structures, such as the left eye with [42, 48] and the right eye with [36, 42].

The next step involves finding which algorithm would be the best option for our project, taking both accuracy and performance into consideration. At this rate, my progress is right on schedule.

Sources

https://www.ecse.rpi.edu/~qji/Papers/frgc_eye.pdf

https://lear.inrialpes.fr/people/triggs/pubs/Dalal-cvpr05.pdf

https://hal-ujm.archives ouvertes.fr/file/index/docid/374366/filename/article_ivcnz2006.pdf

https://towardsdatascience.com/a-guide-to-face-detection-in-python-3eab0f6b9fc1

https://www.pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/

Posted on

Team Status Report for 2/20

The most significant risks at the moment that could jeopardize the success of the project is integrating all the software algorithms we have together and integrating the hardware components. We plan on testing each algorithm extensively before integrating in steps with the others. Once the algorithms are fully integrated for each hardware part. The plan to test communication between the two devices is to start with simple messaging examples and then proceed to our actual software connecting between the devices. We have done research about each of the components to ensure compatibility. Additional research will be done throughout the project as needed. Another concern will be making sure we are able to power both the Nano and Pi with power banks and enough power is received for multiple hours to run software on each. Before integration we will make sure that they work with charging cables, to observe performance and then move to using power banks. Another risk with the power source will be having scripts automatically run the algorithms when power is plugged in or to switch to using a switch to turn the Pi and Nano on and off. Our plan for this to test our script that runs the algorithms in the correct sequence to ensure it works properly without us intervening. Once this is working fully, we will add the command to start upon plugged into outlet, and then plugged into power bank.

So far no changes have been made, but there might be after the proposal presentation on Monday. Thus, there is no updated schedule.

Posted on

Danielle’s Status Report for 02/20

Throughout this week, I have been working on the Project Proposal presentation, specifically on the use case, problem statement, and technical challenges slides. I have met various times with my team to discuss the project and make determinations about how we plan to proceed with the project. I spent a lot of time making sure that our presentation was visually appealing and added the themes and images. One aspect of our project that will be a challenge for us is the timing and differentiating between normal driving movements and distracted driving. While I was driving home this weekend, I was monitoring my own driving techniques and assessing how long I was checking my blind spot, etc. to help make determinations about how long people look away from the road in order to change lanes or turn. I also spent some time doing research about the timing on the internet.

I believe my progress is on schedule for the project. I have been putting in a lot of time to make sure that our presentation was professional and portrayed our project well. We have already finished our presentation and will be meeting today and tomorrow to practice presenting and making small adjustments, so I believe I am on track currently.

In the next week, I hope to research more about timing than I did this week. I also hope to make a more concrete determination about the vibration device that we intend to use as an alert system for the driver, so we can begin to order parts, as well as for the bluetooth wireless capabilities.

Sources:

Posted on

Heidi’s Status Report for 2/20

This week I worked with my team members on the proposal presentation. I created the gantt chart template in excel for our group to use and primarily worked on the requirements and design slides of the presentation. Since I’ve been assigned to work on the facial detection, landmarking and head pose aspects of our project, I have collected a few sources in these areas to help familiarize myself with the technology and algorithms behind them, including some tutorials. From these sources, I’ve decided to code in Python for the face detection and head pose algorithms because I am most familiar with Python and it is compatible with OpenCV. I have also collected tutorials for the initial set up of the Jetson Nano and setup with the wifi bluetooth card and camera module.

Progress is on schedule since the proposal presentation is this coming Monday. In the gantt chart the time before receiving the materials, we will be researching. Once we receive feedback from the professors I will go ahead and order the hardware we agreed to use. 

This next week I hope to use the sources that I found to work out some simple face detection examples on my laptop to better understand how to use OpenCV and determine if any other libraries will be necessary before switching code over to the Jetson Nano. Unless we receive feedback that tells us otherwise, ordering the Nano and the other hardware components will also happen this week, and while waiting for the equipment to arrive, I’ll practice on my laptop.

Sources

Jetson Nano + Wifi + Bluetooth

https://desertbot.io/blog/how-to-add-a-dual-wifi-bluetooth-card-to-a-jetson-nano-intel-8265

https://www.jetsonhacks.com/2019/04/08/jetson-nano-intel-wifi-and-bluetooth/

https://learn.sparkfun.com/tutorials/adding-wifi-to-the-nvidia-jetson/all

Jetson Nano + Raspberry Pi Camera

https://www.jetsonhacks.com/2019/04/02/jetson-nano-raspberry-pi-camera/

Face Detection

https://towardsdatascience.com/face-detection-in-2-minutes-using-opencv-python-90f89d7c0f81

https://realpython.com/face-detection-in-python-using-a-webcam/

https://ieeexplore.ieee.org/abstract/document/8203730?casa_token=5ykY6xwCRgIAAAAA:BFeXXcQDtJOs7EvsYlFwoBGgsddHWTLQpzI4VrZz3EFs5QAztw2gbq5hBaHs7IgUuX2Bwi4a9A