Posted on

Team Status Report for 4/27

This week we mainly worked on the final presentation and slides and made sure that we had all the information presented well. We also worked on making final adjustments to the overall integration of all three components and we will continue to work on this for the rest of this week as well before the final demo on Friday. The most significant risk at this time is making sure that we are able to successfully demo our final product and we can ensure that everything goes well by doing several practice runs this week. We are all very confident that our demo will go well and are looking forward to completing our final product. We are all on schedule as well and were able to stay on track with the Gantt chart that we made earlier this semester. For next week, we will make sure to submit the final poster, video, and report.

Below are the unit tests and overall system tests carried out for experimentation of our system for the Camera-Based Detection. For the EEG, the test results/ procedure were outlined in Rohan’s individual report last week. 

  • Yawning, microsleep, gaze, phone, other people, user away
    • Methodology
      • Test among 5 different users
      • Engage in each behavior 10 times over 5 minutes, leaving opportunities for the models to make true positive, false positive, and false negative predictions
      • Record number of true positives, false positives, and false negatives
      • Behaviors
        • Yawning: user yawns
        • Sleep: user closes eyes for at least 5 seconds
        • Gaze: user looks to left of screen for at least 3 seconds
        • Phone: user picks up and uses phone for 10 seconds
        • Other people: another person enters frame for 10 seconds
        • User is away: user leaves for 30 seconds, replaced by another user
    • Results

  • Overall system usability and usefulness
    • Methodology
      • Test among 10 different users
      • Have them complete 10 minute work session using the app
      • Ask after if they think the app is easy to use and is useful for improving their focus
    • Results
      • 9 of 10 users agreed
  • Data capture and analysis latency
    • Methodology
      • Video processing
        • Measure time between processing of each frame
        • Get average over a 5 minute recording
    • Results
      • Video processing
        • 100 ms
      • Focus/flow detection
        • 0.004ms
Posted on

Karen’s Status Report for 4/27

This week I am continuing to work on integration and improvements to the UI. I am working on improving the user flow focusing on the calibration page and display. Previously, clicking the get started button would trigger a Python script to start running which would take a long time as modules are imported and it would also cause a new terminal window to open. This was a hindrance to the user experience, so I made some changes so that a Python script is running at all times in the background. I changed some logic with the session ID creation in the backend to allow for this new user flow.

I am also working with Rohan to integrate focus and flow together in the web app. I am working on adding toggle buttons that allow you to toggle between displaying the focus vs flow predictions. I will also be working with him on setting up a calibration phase for the EEG headset to calibrate the readings in the web app.

Overall I’ve been making great progress, am on schedule, and am just adding some final touches to the UI.

Focus and flow toggle buttons:

Posted on

Rohan’s Status Report for 4/27

This week I worked on some further experimentation with the flow state model. I tested out different configurations of training, validation, and test sets. I also experimented with adjusting the size of the model to see if we could learn the training data without overfitting such that the model can generalize to new data. Regardless of the training, validation, test splits and the size of the model, I was unable to improve the performance on unseen data which indicates we likely just need more recordings with new individuals as well as with the same people over multiple days. I also realized that the way I was normalizing the data in training, validation, and test set evaluation was different than the process I implemented for normalization during inference. So, I have been working with Arnav and Karen to resolve this issue which also introduces a need for an EEG calibration phase. We have discussed a few possible approaches for implementing this and also have a backup plan to mitigate the risk of the calibration not working out which would make it possible for us to make inferences without any calibration if necessary. My progress is on schedule and mainly will involve last minute testing to ensure we are ready for our demo on Friday next week.

Posted on

Arnav’s Status Report for 4/27

This week I spent time preparing for the final presentation, improving the Session History Page, and additional testing for the whole system. For the final presentation, I made sure that we had the Session History Page integrated with the camera so that we could include this in the current solution slide for the presentation. I also made sure to do some additional research and read extra documentation so that I knew all the details regarding the project and could present all of the work we have done this semester. For the Session History Page, I made it so the date of the session is now on the x-axis (rather than the session id) and the user can easily see how they are improving over time. Below is a screenshot of how the page looks:

I am on schedule and am looking forward to the final demo this Friday. I am now working with Rohan and Karen to make sure all the components of the project are fully integrated for the demo and that all parts of the UI are working as expected. I am also working on our respective parts of the final poster, final video, and final report. This project has been a great learning experience and I am excited to show everyone our final product!

Posted on

Team Status Report for 4/20

This week we have all been mainly working on the final stages of our individual parts of the project and are also working on adding information to the final presentation slides this Monday. Karen and Arnav are working on finishing up the last couple of features of the project and ensuring that all the data looks visually presentable for the presentation and demo. Rohan has spent a lot of time this week collecting EEG flow state data with Professor Dueck and also working on training models for the focus state detection so that we can show both the flow state and focus state on the application. 

We have all the software and hardware components integrated and have all the data we need to display on the application. The most significant risk right now is our EEG based brain state detection models for flow and focus classification. Our plan for mitigating this risk has been to collect more data in the past week and run more comprehensive validation tests to understand where the models are performing well and where they are performing poorly. Now that we have done more validation of the models, we can determine whether we need to collect more data or if we should spend time tweaking model parameters to improve performance. We can also use the SHAP explainability values to sanity check that our models are picking up on reasonable features.

Posted on

Rohan’s Status Report for 4/20

This week I spent most of my time outside of classes working with Professor Dueck and her collaborative pianists and singers to collect more flow state data and I also collected focus/distracted data with Karen. I retrained both the flow and focus state models with the new data and re-ran the Shapley value plots to see if the models are still picking up on features that match existing research. Our flow state model is now trained on 5,574 flow data points and 7,778 not in flow data points and our focus state model is trained on 11,000 focused data points and 9,000 distracted data points. 

The flow state model incorporates data from 6 recordings of I-Hsiang (pianist), 7 recordings of Justin (pianist), and 2 recordings of Rick (singer). I evaluated the model performance against 3 test sets: 

The first one is pulled from data from the recordings used in training but the data points themselves were not included in the training set. 

Precision: 0.9098

Recall: 0.9205

F1 Score: 0.9151

The second test set was from a recording of Elettra playing the piano which was not included in the training set at all.

Precision: 0.7928

Recall: 0.4106

F1 Score: 0.5410

The third test set was from two recordings of Ricky singing which were not included in the training set at all.

Precision: 0.7455

Recall: 0.6415

F1 Score: 0.6896

The focus state model incorporates data from 2 recordings of Rohan, 4 recordings of Karen, and 2 recordings of Arnav. I evaluated the model performance against 3 test sets: 

The first one is pulled from data from the recordings used in training but the data points themselves were not included in the training set.

Precision: 0.9257

Recall: 0.9141

F1 Score: 0.9199

The second test set was from two recordings of Karen which were not included in the training set at all. 

Precision: 0.6511

Recall: 0.5570

F1 Score: 0.6004

The third test set was from two recordings of Ricky singing which were not included in the training set at all.

Precision: 0.5737

Recall: 0.5374

F1 Score: 0.5550

Finally, I reran the Shapley values for both the flow and focus state models and found that the features they are picking up still match up with existing research on these brain states. Furthermore, features that are particularly prominent in flow states such as theta waves are contributing heavily to a classification of flow in the flow state models but is contributing strongly towards a classification of non-focus in the focus state model which is very interesting because it demonstrates that the models are picking up on the distinction between flow and focus brain states.

Flow Shapley values:


Focus Shapley values:

As I have worked on this project I have learned a lot about the field of neuroscience and the scientific process used in neuroscience experiments/research. I have also learned about the classical opera music world and how to do applied machine learning to try to crack unsolved problems. It has been particularly interesting to bring my expertise in computing to an interdisciplinary project where we are thinking about machine learning, neuroscience, and music and how this intersection can help us understand the brain in new ways. The primary learning strategy I have used to acquire this new knowledge is to discuss my project with as many people as possible and in particular those who are experts in the fields that I know less about (i.e. neuroscience and music). I give frequent updates to Professor Grover and Professor Dueck who are experts in neuroscience and music respectively and have offered significant guidance and pointers to books, papers, and other online resources which have taught me a lot. I have also learned what it is like working on an open-ended research project on a problem that is currently unsolved as opposed to implementing something which has already been done before.

Posted on

Karen’s Status Report for 4/20

Now that I have finished implementation of all the distraction and behavior detections, I spent time testing and verifying the integrated system. I tweaked thresholds so that the detections would work across different users. 

My testing plan is outlined below, and the results of testing can be found here.

  • Test among 5 different users
  • Engage in each behavior 10 times over 5 minutes, leaving opportunities for the models to make true positive, false positive, and false negative predictions
  • Record number of true positives, false positives, and false negatives
  • Behaviors
    • Yawning: user yawns
    • Sleep: user closes eyes for at least 5 seconds
    • Gaze: user looks to left of screen for at least 3 seconds
    • Phone: user picks up and uses phone for 10 seconds
    • Other people: another person enters frame for 10 seconds
    • User is away: user leaves for 30 seconds, replaced by another user

After completing testing, I focused my effort on integration, getting flow state and distractions displayed on the current session page. I also made significant improvements to the UI to reach our usability and usefulness requirements.This required a lot of experimentation with different UI libraries and visualization options. I’ve attached some screenshots of the iterative process of designing the current session page.

My next focus is improving the session summary page and fully integrating the EEG headset with our web app. Currently, the EEG data visualizations are using mock data that I am generating. I would also like to improve the facial recognition component as the “user is away” event results in many false positives at the moment. With these next steps in mind and the progress I have made so far, I am on track and schedule.

One of the new skills I gained by working on this project is how to organize a large codebase. In the past, I’ve worked on smaller projects individually, giving me leeway to be more sloppy with version control, documentation, and code organization without major consequences. But with working in a team of three on a significantly larger project with many moving parts, I was able to improve my code management, collaboration, and version control skills with git. This is also my first experience fully developing a web app, implementing everything from the backend to the frontend. From this experience, I’ve gained a lot of knowledge about full-stack development, React, and Django. In terms of learning strategies and resources, Google, Reddit, Stack Overflow, YouTube, and Medium are where I was able to find a lot of information on how others are using and debugging Python libraries and React for frontend development. It was helpful for me to read Medium article tutorials and watch YouTube tutorials of a “beginner” project using a new library or tool I wanted to use. From there, I was able to get familiar with the basics and use that knowledge to work on our capstone project with a bigger scope.


Posted on

Arnav’s Status Report for 4/20

This week and last week, I spent a lot of time improving the overall UI of the application, adding a Session History page that allows the user to see statistics from previous work sessions, and testing the backend integration to ensure that all components are working well. 

I continued to build upon the UI enhancements I made last week and made it even more visually appealing. I did more research on the Chakra UI and also worked with Photoshop to add images of the headset and camera on the homepage. Below are pictures of how the current homepage looks:

The Session History page is where users can see data from all the previous sessions in one viewing allowing them to see how they are improving in limiting the number of distractions. Currently, we keep track of the session_id and frequency for each distraction when sending the data to the Django backend. I created a new endpoint (http://127.0.0.1:8000/api/session_history) and added the total number of distractions to the data payload along with the session_id and timestamp so that the frontend can fetch this information from the endpoint and display it on the react page. I made sure to update the backend so that it checks to see if the session_id already exists in the database before updating the number of distractions. This ensures that the data from the specific session_id is only shown once on the endpoint and not every time a new distraction occurs.  Users can now see a graph for Total Distractions over all sessions and when the user clicks on the bar chart, it takes the user to the specific Session Summary page corresponding to that session. Below is an example of how the Session Summary page looks:

Throughout the development of this project, I found it essential to learn several new skills and tools, particularly in frontend development using React component libraries, backend development with Django, and integrating both with hardware (camera and Emotiv EEG headset). To efficiently acquire these skills, I relied on tutorial videos on YouTube to understand React and used Stack Overflow and various Medium articles to troubleshoot specific issues I encountered with Django and database integration. Additionally, for the hardware integration, I referred to specific documentation and community forums from the EEG headset and Camera API for real-world insights and coding examples. These methods were efficient, offering targeted, real-time solutions that I could immediately apply to address any concerns during the project.

I am on schedule and am looking forward to the last couple of weeks. I am now working on integrating the Session History page with the EEG data so that our app can display a graph for the % of time in a flow/ focus state for all the given sessions. I will also continue to work on further improving the UI and preparing for the final presentation on Monday.

Posted on

Team Status Report for 4/6

The most significant risk right now is the accuracy of our flow state detector. After determining a good level of accuracy on our testing data from Professor Dueck in a musician setting, we went on to perform validation in the other settings this week. Rohan performed and recorded EEG data for an activity to stimulate flow state (playing a video game) with noise-canceling headphones and music to encourage flow state for 15 minutes. He then recorded a baseline of performing a regular work task with distractions (intermittent conversation with a friend) for another 15 minutes. On the intended flow recording, our model predicted .254% of the time was in flow and in the intended not-in-flow recording, our model predicted .544% of the time was in flow. We have a few ideas as to why the model may be performing poorly in this validation context. First of all we think that in both settings, Rohan did not enter a flow state because neither task was second nature or particularly enjoyable and also 15 minutes is likely too short of a period to enter a flow state. To further validate the flow detection model, we plan to have Rohan’s roommate who is an avid video gamer wear the headset while playing games for an extended period of time to see how the model performs in detecting flow in a gaming environment. Depending on how this goes, we plan to validate the model again in the music setting to see if our model has overtrained to detect flow specifically in a musical setting.

Rohan also implemented a custom focus state detector. We collected recordings of Arnav, Karen, and Rohan in 15 minute focused and then 15 minute distracted settings while collecting EEG data from the headset. The model achieved high test accuracy on data it had not seen before and had strong precision, recall, and F1 scores. We collected data with Karen wearing the headset again, this time for 20 minutes in focus and 20 minutes distracted to use for model validation. When we ran this data through the custom focus detector, we saw a disproportionately high amount of distracted classifications and overall poor performance. We realized that the original training set only had 28 high quality focus data points for Karen compared to 932 high quality distracted data points for Karen. So, we attribute the poor performance to the skewed training data, plan to incorporate this validation data as training data for the model, and collect new validation data to ensure that the model is performing well. As a backup, we inspected the Emotiv Performance Metrics for focus detection and saw a clear distinction in the average focus Performance Metric in the focused recording as compared to the distracted recording. 

Finally, as an attempt to further validate our custom models, Rohan applied Shapley values which are a measure used in explainable AI to understand which input features are contributing most significantly to the flow vs not in flow classification. 

Validation for video processing distraction detection:

  • Yawning, microsleep, gaze
    • Test among 3 different users
    • Have user engage in behavior 10 times
    • Record number of true positives, false positives, and false negatives
  • Phone pick-up
    • Test among 5 different phones (different colors, Android/iPhone)
    • Have user pick-up and use phone 10 times (5-10 second pick-up and use)
    • Record number of true positives, false positives, and false negatives
  • Other people
    • Test among 1 user and 3 other people
    • Have other person enter frame 10 times (5-10 second interaction)
    • Record number of true positives, false positives, and false negatives
  • Face recognition
    • Test among 3 different users
    • Register user’s face in calibration
    • Imposter takes place of user 3 times in a given work session
      • Imposter takes user’s place in 30 second intervals
    • Record number of true positives, false positives, and false negatives for imposter being recognized
  • Performance
    • Calculate average FPS over every 10 frames captured, logic below
    • Get average FPS over a 1 minute recording

        if COUNTER % FPS_AVG_FRAME_COUNT == 0:

            FPS = FPS_AVG_FRAME_COUNT / (time.time() – START_TIME)

            START_TIME = time.time()

        COUNTER += 1

Overall, integration is going smoothly as we have all of the distraction types integrated into the frontend and backend of the web app except for face recognition. Besides the hiccup in the accuracy of our flow state detector, our team is overall on schedule.

Besides improving the focus and flow state detector performance, Arnav and I will be focusing this coming week on improving the UI to improve the user experience.

Posted on

Arnav’s Status Report for 4/6

This week, I put a lot of effort into combining the backend and frontend parts of our project, focusing especially on adding EEG data to the frontend for the interim demo. I also looked for new ways to make our app look better and be more user-friendly by trying out different React UI component libraries. 

I worked with Rohan on integrating real-time EEG data visualization into our frontend. We created an interactive pie chart that differentiates between moments of focus and distraction, offering users a visually engaging way to understand their focus levels.

For the new UI, I looked at various React UI component libraries including Chakra UI, Material UI, and React Bootstrap. Among them, Chakra UI stood out because it is easy to use and made our app look much better. I revamped the homepage with images and descriptions of the emotive headset and camera. These enhancements help describe the technology behind the focus-tracking app, providing users with clear insights into how the system works to monitor their focus levels. Below is the new-looking homepage:

Regarding the testing/verification for my subsystem, I focused on ensuring seamless communication between our Django backend and React frontend. First, I worked on API Response Testing by sending numerous requests to our backend every minute, aiming to simulate how a user would interact with the app. This was crucial for verifying that data from the EEG headset and camera was being processed quickly and accurately. My target was for our backend to handle these requests in under a second, ensuring users experienced no noticeable delays. Next, I tested the UI Responsiveness. I wanted to make sure that as soon as our backend processed new data, our frontend would immediately reflect these updates (in under 500 milliseconds) without any need for users to manually refresh their browsers. Both tests were successfully completed and I will work on continuing to test other parts of the integration over the next week. 

Overall, I am on schedule and next week I will continue to work on enhancing the frontend for the rest of the pages and make sure that it engages the audience for the final demo. The backend integration is already complete and I will continue to test it further to ensure it is fully accurate and meets the requirements stated in our design report.