Posted on

Team Status Report for 02/25/23

The significant risks that could jeopardize this project is our lack of knowledge about wifi cards and which one to buy. The wifi card component is very important for our project as it will allow us to transfer driver data to the cloud, allowing drivers to keep track of how they’re driving. This will be mitigated with consulting the TAs and anyone else who has experience with wifi cards.

Some changes that were made to the design was adding a head pose estimation system. Eye tracking on driver’s with sunglasses is not feasible and after critiques from the professor in class during the design presentation, we’ve decided to tackle this issue with head pose estimation where the person’s head position would be measured to determine whether they’re distracted/drowsy. This will add an extra week to the schedule and push back our progress by a week. This is being mitigated by the 3 week slack time buffer we had at the very beginning.

Other than adding the head pose estimation code, everything else from the schedule last week looks relatively the same.

Posted on

Yasser’s Status Report for 02/25/23

Earlier this week, I accomplished wrapping up the design presentation slides, specifically with the block diagrams regarding the facial and landmark detection algorithms, as well as the mouth and eye detection. After the presentation, I accomplished researching the frameworks and models needed for head pose estimation. After speaking with the professor after the design presentation, he suggested we circumvent the sunglasses issue on the driver with measuring head movement to see if a driver is distracted/drowsy. A Tensorflow Model and Caffe based OpenCV DNN is needed, both of which are being used for the facial detection and landmark algorithms we intend to use in this project. Besides researching about head pose detection, I started coding the facial detection algorithm using OpenCV and the Caffe facial detector model.

My progress ahead. By the end of this week I was tasked with having all of the algorithms needed researched. Since I had that done earlier in the week, I decided to move on with next week’s task of writing the code for the facial and landmark detector. As shown in the image, the facial detection code is done.

By the end of next week, I expect to have the landmark detector code done, as well as the mouth and eye detection code and the design document as well.

Some team shortfalls this week was not having my team members in pittsburgh when the design presentation slides were due. I especially needed help with writing the speaker notes on the slide with the hardware block diagram as I had very little knowledge on the hardware used for this project, as well as the UI and the servers where our webapp will be hosted on. However, we got through this obstacle with constant communication.

Posted on

Elinora’s Status Report for 02/25/23

This week was pretty busy for all of us and we had design presentations during class, so our Gantt chart schedule was relatively light. On Sunday I spent a lot of time finishing up our design presentation slides and speaker notes with Yasser and Sirisha. I submitted an order request for an accelerometer, which should arrive next week to begin testing on. The last part that I am in charge of ordering is the Wifi card for the Jetson, but I want to have a conversation with a TA that has more experience with using Jetsons before I select a card and submit the order form. I also met with Sirisha outside of class to create a more final design for the web app and which technologies we wanted to use. We decided that we wanted to use React for the frontend, Firebase for login capabilities, and that we want to send data from the Jetson to our cloud data storage (on an AWS S3 bucket) at routine intervals every few minutes. We also decided that we wanted to include a log page in the UI design that would record the time that each notification to the user took place and the reason for that notification (inattention or drowsiness and a more specific description).

I misread the schedule last week (mixed up start of week vs end of week dates) on our Gantt chart and thought that we were behind schedule in my last status report. The tasks that I thought needed to be completed by the end of this week are actually scheduled to be completed by the end of next week. I am on track according to our actual schedule that we confirmed as a group. 

The deliverables I listed in my status report for last week (since I misread our schedule) are the deliverables that I hope to complete by the end of next week. After talking with a TA about the Wifi card, I will submit an order form by the end of next week. With the initial logistics and page designs for the web app all set, Sirisha and I are ready to set up the initial structure of the web app together next week. We will also divide up which pages each of us will work on and I will start implementing the basic format of my assigned pages. Next week I am scheduled to conduct initial testing on the individual hardware components. I want to test out playing sounds from code on the Jetson to the external speaker that we ordered. Additionally, I want to measure the power consumption of the Jetson while running basic code (likely just the code sending audio data to the speaker) to see how feasible it will be for us to fully power the whole device with the 12V power outlet in a car. I will also work with Yasser and Sirisha to apply the feedback from our design presentation to write our design report. 

Posted on

Sirisha’s Status Report for 02/25/23

I spent the majority of the time at the beginning of this week working on the design presentation slides.  Specifically, I contributed to almost all of the topics for the slides required in the design presentation guidance page aside from the information that my teammates are more familiar with.  I also helped with the speaker notes so that we’d all be equally familiar with the content of the presentation.  In addition, I did more research into the remaining hardware components we need for the device and had them ordered.  I also continued brainstorming and designing the UI for the web app with Elinora.  We discussed exactly what components we wanted in the web app and what kind of information each page would display.  We already had a few pages planned out from last week, but now we have all of the pages planned out to display the  charts tracking a user’s driving history and feedback and a log of all of the feedback.  I also started doing research into using React.js and created the initial React app.  I also did some more research into how wifi will be integrated into the device in the car, but it’s been a bit challenging so far to exactly figure out how to implement this component.

According to the Gantt chart, by the end of the week, I was supposed to have completed working on the design presentation and initial design documentation, working on the device schematic and part selection, and refining the web app design.  The design presentation was completed, the parts selection have been done, and the web app design is completed as well.  The design documentation has been started, but I think at this stage, there should be a little more fleshed out.  To catch up on this part, I will dedicate more time to it now that my schedule has cleared up more this week.

By next week, the design documentation should be worked on and completed as a team.  We also hope to have all of our hardware components acquired so that we can test them all individually to make sure that they work.  I will also be working with Elinora to start creating the pages in the React app and implementing our UI designs.  I also want to continue figuring out how to incorporate the wifi with the device in the car.

Posted on

Team Status Report for 02/18/23

The most significant risks that could jeopardize the success of this project is adjusting our algorithms and approach for low light settings and drivers with sunglasses. To mitigate this risk, we’ll have to reduce the scope of this project and specify having good lighting conditions where the driver’s face is clear and visible and add a requirement for the driver that sunglasses be taken off. Another issue could be the accuracy of OpenCV’s DNN for facial detection. Only preliminary testing has been done but if we find that this model does not give us the accuracy that we want or if using GPU parallelization with CUDA is not compatible with the Volta GPU on the Xavier or if it’s out of scope then we’ll resort to using Dlib which has a higher accuracy.

There were some changes to made to the design. We added a calibration step to the software after the driver input is detected. The reason why this step is necessary is because a point of reference is needed when checking if a driver is distracted or not. To be more specific, in this calibration step we are measuring and recording the distance of the lips when the mouth is closed along with eye measurements of the driver looking at the road. Costs that the change incurs are more code/time spent on this feature. This cost is mitigated by the 3 week buffer we originally had.

Some changes were made to the schedule, such as a week of adding the calibration process in addition to moving the start time of designing the UI of the webapp by a week in advance.

Posted on

Yasser’s Status Report for 02/18/23

This week I personally accomplished drawing the diagrams included in the Design review dealing with the computer component, specifically with how the OpenCV DNN module works and how to incorporate the Caffe learning module used for facial detection and the landmark detector used to obtain eye and mouth detection.

In addition, I started reading OpenCV’s CUDA API in order to optimize our image processing code and obtain our desired frame rate of 5 fps however, CUDA applications were not originally intended to be used on the Volta architecture. There are documentation for making CUDA applications compatible with the Volta architecture ( https://docs.nvidia.com/cuda/volta-compatibility-guide/#about-this-document). Furthermore I read tutorials on how to detect eyes with OpenCV with the use of thresholding which creates binary masks used to segment out the eyeballs from the rest of the eye (useful for eye tracking) and for segmenting out the mouth from the rest of the face.

My progress is up to schedule at the moment as I have researched the computer vision algorithms necessary for this project: OpenCV DNN for facial detection, CNN-facial-landmarks for obtaining facial landmarks (used to obtain eye and mouth detection), and creating a region of interest on a binary mask with the size of a driver’s eye and mouth for eye and mouth detection.

Deliverables that I hope to complete by next week are writing the initial facial detection algorithm and facial landmark algorithm using OpenCV’s DNN and CNN-facial-landmark model.

Posted on

Elinora’s Status Report for 02/18/23

I spent the majority of my time this week working on aspects of our upcoming design presentation. After discussing and deciding the design/implementation of our project with my teammates, I put together the system block diagram for our presentation. Sirisha and I also worked together to make more fleshed-out UI designs for the web application aspect of our project with Google Slides. We designed the login, registration, and main statistics (home) pages.  

Capstone UI Design as of 2_18_23

Outside of the presentation, I spent some time looking into how to send the driver inattention data from the Jetson (where CV and classification is taking place) to our web application while in a moving car. It looks like adding a wifi card to the Jetson with bluetooth capabilities will work for our purposes, but I want to talk more with one of the TAs/professors with more experience working with Jetsons on Monday to confirm this before submitting an order for the part. I also submitted orders for our speaker and a cable to connect the audio jack of the speaker to the USB-C port on the Jetson. 

My progress is behind schedule according to our Gantt chart. Specifically the “Device Schematic/Parts Selection” and “Testing Each Component Individually” tasks will need to be pushed to next week when we finish submitting our parts orders and receive the parts that we want to test individually. When setting our original deadlines, we did not take into consideration the time it would take to decide, approve/order, and receive each of our individual components. We do have our borrowed Jetson, but all of our other components are either ordered and not yet received or will be ordered on Monday. I updated our Gantt chart to reflect this. 

In the next week, I hope to finish submitting all of our part orders with Yasser and Sirisha and test out playing sounds from code on the Jetson to the external speaker that we ordered. I will also work with Sirisha to implement the initial setup of our web application (creating the initial pages and some basic components in React.js) according to our UI design. Additionally, I want to measure the power consumption of the Jetson while running basic code (likely just the code sending audio data to the speaker) to see how feasible it will be for us to fully power the whole device with the 12V power outlet in a car. 

The particular ECE courses I’ve taken that cover the engineering principles our team used to develop our design are: 10-301, 18-220, 18-341/18-447, 17-437 (Web App Development), and 05-391 (Designing Human-Centered Software). 

 

Posted on

Sirisha’s Status Report for 02/18/23

This week, I worked on researching and ordering the appropriate components for the embedded device and helped with designing the device schematic.  I also started working on the design for the UI of the web app, but after the weekly meeting with our advisor, the plans for the app might need to change depending on how the device will be able to access wifi in the car.  Since our design presentation is next week, I also worked on the slides.  Our progress is mostly on track according to the Gantt chart; however, due to the potential change in plans with how the data will be stored and tracked, we might need longer for designing the UI.  Next week, we hope to have all of our hardware components and aim to test them all individually to make sure they are working, and if not, we will have enough time to purchase more.  I also hope to refine the design for the UI and start work on the design report.

Posted on

Yasser’s Status Report for 02/11/23

This week I did more research on what computer vision components to implement/design for the project. I focused more on how to utilize the OpenCV library for facial detection and eye tracking by following some of the OpenCV deep learning tutorials from PyImageSearch (https://pyimagesearch.com/2017/08/21/deep-learning-with-opencv/). In addition to this, I researched how to incorporate OpenCV with a GPU to make the computation faster, thus allowing us to reach our goal of a quick feedback response to the driver. That is how I stumbled upon this video
(https://www.youtube.com/watchv=LQymI4uOL7w&list=PLkmvobsnE0GEkJXGxOjV9uuyozS8QGx3M&index=2&ab_channel=NicolaiNielsen-ComputerVision%26AI). All in all, my personal accomplishment was having a rough idea on how to use OpenCV DNN in this project, as well as using other OpenCV packages with a GPU to make computations faster. My progress is on schedule now. I still must specifically research how to incorporate a DNN
module into the Xavier Jetson’s GPU, such as an API. Deliverables that I hope to complete in the next week is the design review, specifically my
section which deals with the computer vision software. This will involve researching any APIs that connect with the Jetson’s GPU for faster computation as well as researching libraries that aid in image classification.

Posted on

Team Status Report for 02/11/23

Overall, our goals for this week were to research which specific parts we wanted to use, research how to apply CV/ML algorithms, begin designing the web application, and start working on the slides for our design presentation. Next week (according to our gantt chart) we will finalize all aspects of our initial project design and complete the design presentation slides. We will also continue researching facial detection algorithms and order components/begin initial testing on any that arrive. 

To answer the additional question for this week, our project includes considerations for public safety, environmental safety, and cultural factors. The overall project goal to monitor distracted driving practices and relay feedback to drivers to reduce unsafe behaviors promotes public safety (both for the driver and others on the road). Additionally, we are considering privacy concerns by making each user’s distracted driving data only available to them, with no feature to view others’ data on the web application. Although less directly related, we are also considering cultural factors in the sense that this product (if widely adopted) could create increased awareness for what constitutes distracted driving and could bring about a cultural shift in driving behavior. 

At this point, the most significant risk that could jeopardize the success of our project is not having a concrete plan on approaching the project with well-thought out hardware and software component selections. To mitigate this, we are each doing individual research and then discussing as a group all aspects of our design before finalizing it. Our design proposal presentation is the most important part of our project to get right to reduce later headaches and set us up for success for the rest of the semester. Further along the line, the risk that we are most concerned about is our ability to tune our chosen algorithm to accommodate low-light conditions and obstructions to viewing the driver’s eyes (e.g. sunglasses or a hat). We are attempting to best mitigate this risk by planning for 4 weeks of testing and tuning the algorithm and an additional 3 weeks of slack time at the end of our schedule. 

So far, no significant changes have been made from the user and technical requirements and general hardware and software components that we presented in our proposal presentation. After we receive feedback next week on our proposal presentation, we are expecting that there may be some changes that we will have to make.  We are currently on track with our initial schedule.