Posted on

Team Status Report for 03/18/23

As of right now, the most significant risks that could jeopardize the success of the project are making sure that the algorithms used for eye tracking and facial detection are able to be completed on time and can be done with the appropriate frame rate.  This is imperative to get done as soon as possible because many of the other features cannot be completed until that part is done.  We have been working to constantly test the algorithms to ensure that they are working as expected, but in the event that they aren’t, we have a backup plan of switching from OpenCV DNN to Dlib.

We were looking into changing the model of the NVIDIA Jetson because the one we currently have could possibly utilize more power than what a car can provide. If this change needs to happen, it won’t incur any extra costs because there are other models in inventory and the other hardware components that we have are compatible with the other models. Also, in between the design presentation and report, we have added back the feature of the device working in non ideal conditions (low lighting and potential obstruction of the eyes by sunglasses or a hat). This was done based on feedback by faculty, but at the moment we are still unsure if we will for sure add back non ideal conditions because other faculty/TA feedback mentioned that we shouldn’t add it back. If it is added back, the cost incurred will be having to spend time working on the algorithm with non ideal conditions. This will lead to less time perfecting the accuracy of the ideal conditions algorithm. We are also changing the design slightly to account for edge cases in which the driver would be looking away from the windshield or mirrors while the car is moving but still not considered distracted. This would occur when the driver is looking left or right while making a turn or looking over their shoulder while reversing. For now, we will limit audio feedback for signs of distraction to be given only when the car is above a threshold speed (tentatively 5mph), replacing our previous condition of the car being stationary. This mph will be adjusted based on recorded speeds of turns and reversing during user testing. If we have extra time, we are considering detecting whether the turn signal is on or car is in reverse to more accurately detect these edge conditions.

After our meeting with the professor and TA this week, we are changing the deadline for the completion of the eye tracking and the head pose estimation tracking algorithms to be next week. We made this change because the CV/ML aspects of our project are the most critical to achieve our goals and having the algorithms completed earlier will allow us to have more time for testing/tuning to create a more robust system. Also, we have shifted the task of assembling the device and testing individual components, specifically the accelerometer and camera, to next week after those components have been delivered. Otherwise, we are on track with our schedule.

 

Posted on

Elinora’s Status Report for 03/18/23

This week, I mainly focused on translating our web app design into code with Sirisha, primarily working on the login and registration pages. I also looked into how we could use Firebase to store our login information for the web app and read through the intro tutorials on its website. I spent a lot of my time this week outside of class on the ethics assignment as well. 

I am mostly on schedule according to our Gantt chart. We had previously scheduled for me to assemble the device this past week, but we are still waiting on some ordered components to arrive. Last week I had set the goal that I would research and decide on a cellular dongle, but I decided to hold off on that task for a while since we won’t need the cellular dongle for 2 more weeks. 

Next week, I will finish up the structure of the web app with Sirisha and create and feed dummy data into the web app to calculate and display the corresponding graphics on the metrics page. Once we receive the accelerometer and camera, I will test the accelerometer and assemble the device. I will also be working on the second half of the ethics assignment and attending the ethics lecture, so I will make sure to plan ahead and reserve more time to work on tasks for the project outside of class.

Posted on

Yasser’s Status Report for 03/18/23

Personally, I accomplished getting the eye tracking algorithm working. There is an issue with the frame rate being slow on my webcam which can be attributed to the high resolution so further testing on our purchased camera will be needed to see what the issue is. Below is a screen shot of the eye tracking (red dots around my eyes, might not be fully visible in this photo).

 

My progress is still behind. I had planned on having the classification and calibration step done. However, after meeting with the professor, I need to focus on getting the gaze tracking algorithm first with determining when a driver’s pupils are looking away from the road, which I plan on getting done by Wednesday.

After I’m done with that, I plan on working on the mouth detection, calibration step, and classification. By next week, I hope on getting the code that determines whether a driver is looking away from the road done and get started on the calibration step.

 

Posted on

Sirisha’s Status Report for 03/18/23

This past week, I worked mainly on the web application and testing and starting to set up the hardware.  I continued to implement the designs of the pages that were worked on earlier.  At the moment, almost all of the pages have been set up, and we are working on them making look like our designs.  We are looking into Firebase as well as an authentication system to incorporate into the web application.  I also started doing some testing of the hardware to make sure each component is working individually and, in the event we find something that requires us to use a different component and purchase something new, we want to have enough time for the order to be delivered and tested again.  We had a concern regarding the Jetson because it could potentially use more power than what a standard car is able to provide, so we have been looking at possible alternatives if in practice the Jetson actually cannot work properly in a car.  We also had to refine a few design choices, such as some more edge cases to consider with our project and if some features should actually be included, since we are getting conflicting feedback from faculty and TAs about this area.  A good chunk of this week was also used to work on the ethics assignment.

Our progress was mostly on track for this past week.  Due to the delay in our hardware components being delivered and some refinements we had to make in the computer vision algorithms, we weren’t able to integrate any hardware with software yet, and we are still testing individual components to make sure everything works.  This part can’t be worked on much until the hardware is finalized and delivered, which should be able to be finished very soon due to the research and testing we did.  The hosting of the website is also yet to be done, but that is something that should take a very short amount of time.

In the next week, we hope to have the eye tracking algorithms complete with the ability to detect the direction that the eye is looking in, the website should be hosted, and the hardware components should all be delivered and individually tested by then.

Posted on

Sirisha’s Status Report for 03/11/23

The majority of the time this past week was spent on the design report and additional research as we determined certain changes needed to be made since we refined some aspects of the project.  For the design report, I worked on the abstract, the design trade studies, parts of the system implementation, test, verification, and validation, project management, and related work, in addition to assisting my teammates with their sections as needed and reviewing and revising the entire report at the end.  Throughout working on the design report, we all needed to conduct some more research in order to be thorough with our descriptions.  I also needed to review all of the work we had done up until this report to make sure everyone was on the same page about what we decided for our project and any changes that were made.  While working on the design report, we also started to individually test some of our hardware components to make sure they all work, but we are still waiting on some components to be delivered since some changes with our software algorithm decisions necessitated a change in the hardware we needed.  I also continued working on the website and implementing our UI designs from last week.

According to the Gantt chart, I was supposed to have finished any work with the device schematics and part selection, component testing, refining the design documentation, and setting up the initial webapp structure.  The work with testing components needs to be pushed because we had to change some of the hardware we needed to order.  Also, because of how much time the design report was taking up last week, not as much time was allocated to continuing the initial webapp structure.  These can be alleviated by next week because we have a lighter week according to the Gantt chart.

By next week, I aim to finish the initial webapp setup and assist the others in their work so we can start integration.

Posted on

Yasser’s Status Report for 03/11/23

For this status report, I accomplished finishing writing the design report for our project. Specifically, I was tasked with writing the design requirements, design trade studies for the facial detection models and camera, as well as the system implementation dealing with the calibration process and computer vision. For the system implementation, I also added diagrams and flow charts detailing the overall process of how the computer vision algorithms will detect whether a driver is attentive or distracted. Furthermore, I was also tasked with quoting my sources for the information I was responsible for writing.

My progress is behind schedule. By this week, I should’ve already had the eye and mouth detection algorithms coded. However, this has not been accomplished since most of my attention was geared towards writing the design document. As a result, the actions that must be taken in order to catch up must be to focus on the eye and mouth detection code and get it done mid week. Afterwards, I’ll aim to get started on the calibration step code and the classification. There is also a 3 week buffer embedded into our Gantt schedule so this buffer could also be used to offset my delay.

In the next week, I aim to have the eye and mouth detection code written, as well as getting started on the calibration step and classification. If possible, finishing the latter two will be great and get me up to speed.

Posted on

Team Status Report for 03/11/23

One of our current risks is that the Jetson AGX Xavier will require more power than the 12V car outlet can provide. This week we will take initial measurements to determine how much power the Jetson is consuming when on and play some music through the speaker to get a better idea of whether 12V will be sufficient. If our Jetson ends up consuming too much power, we can mitigate this by borrowing a different Jetson from the ECE department that has lower power consumption but slower computation abilities (such as a Jetson Nano). Another risk is that the audio-guided calibration process will be unclear to the user. To address this, we will conduct testing of the calibration process with at least 3 users that aren’t one of us and get their feedback on how easy the process is to follow.

We had 2 main changes to the design of the system since last week. First, we decided to use a cellular dongle instead of a WiFi dongle to send data from the Jetson to our web application. This change will allow us to connect directly from the Jetson to the internet without requiring any additional efforts from the user (such as setting up their own hotspot for the Jetson to connect to).  The second change was that we added a logs page to the web application that will allow for a user to view a record with more detailed reasoning for each of the instances that triggered feedback from their DriveWise device. The most current page layout for the web application is below:

We are on track with our Gantt chart and our overall schedule has not been changed. 

After establishing our subsystems, we have determined that the main tools we will need to be able to learn how to use will be OpenCV DNN and Firebase. OpenCV DNN will be used for the CV/ML subsystem and Firebase will be used to enable login/registration capabilities on the web application. 

Posted on

Elinora’s Status Report for 03/11/23

Earlier in the week, Sirsha and I set up the initial structure of our web application in React over VSCode LiveShare. We programmed the frontend basics for the login and stats pages and tested that we could access these pages by clicking different buttons on the web app (like the “Login” button leading from the login page to the stats/home page). We also decided to add an additional page to our UI design in which users can access the individual log for each time that feedback is triggered from their device. Viewing this page provides users with more detail regarding the kind of inattention that they exhibited and the layout of the page is included in this week’s team status report. I spent the rest of my time this week working on the design report with Yasser and Sirisha. More specifically, I was assigned the introduction, use-case requirements, architecture, and parts of the implementation and design trade studies sections. I also briefly tested the speaker that we ordered and confirmed that it played sound clearly from a .wav file when connected to my computer. 

I am pretty much on schedule with our Gantt chart. After we decided to use a cellular data dongle instead of a WiFi dongle to send data from the jetson to the web application, we wanted to conduct more research before ordering a particular cellular dongle. Since transmitting the data to the web application is not scheduled in the Gantt chart until 2 weeks from now, I will hold off on submitting the order until we are sure. Otherwise, I am on schedule with my tasks.  

For my deliverables next week, I am scheduled to assemble the hardware components of the device and test power consumption. I also want to research different cellular data dongles and decide on one by the end of the week. For the web application, I will more fully flesh out the login and registration pages this week and potentially set up Firebase for the actual login/registration capabilities of these pages.

Posted on

Team Status Report for 02/25/23

The significant risks that could jeopardize this project is our lack of knowledge about wifi cards and which one to buy. The wifi card component is very important for our project as it will allow us to transfer driver data to the cloud, allowing drivers to keep track of how they’re driving. This will be mitigated with consulting the TAs and anyone else who has experience with wifi cards.

Some changes that were made to the design was adding a head pose estimation system. Eye tracking on driver’s with sunglasses is not feasible and after critiques from the professor in class during the design presentation, we’ve decided to tackle this issue with head pose estimation where the person’s head position would be measured to determine whether they’re distracted/drowsy. This will add an extra week to the schedule and push back our progress by a week. This is being mitigated by the 3 week slack time buffer we had at the very beginning.

Other than adding the head pose estimation code, everything else from the schedule last week looks relatively the same.

Posted on

Yasser’s Status Report for 02/25/23

Earlier this week, I accomplished wrapping up the design presentation slides, specifically with the block diagrams regarding the facial and landmark detection algorithms, as well as the mouth and eye detection. After the presentation, I accomplished researching the frameworks and models needed for head pose estimation. After speaking with the professor after the design presentation, he suggested we circumvent the sunglasses issue on the driver with measuring head movement to see if a driver is distracted/drowsy. A Tensorflow Model and Caffe based OpenCV DNN is needed, both of which are being used for the facial detection and landmark algorithms we intend to use in this project. Besides researching about head pose detection, I started coding the facial detection algorithm using OpenCV and the Caffe facial detector model.

My progress ahead. By the end of this week I was tasked with having all of the algorithms needed researched. Since I had that done earlier in the week, I decided to move on with next week’s task of writing the code for the facial and landmark detector. As shown in the image, the facial detection code is done.

By the end of next week, I expect to have the landmark detector code done, as well as the mouth and eye detection code and the design document as well.

Some team shortfalls this week was not having my team members in pittsburgh when the design presentation slides were due. I especially needed help with writing the speaker notes on the slide with the hardware block diagram as I had very little knowledge on the hardware used for this project, as well as the UI and the servers where our webapp will be hosted on. However, we got through this obstacle with constant communication.