{"id":217,"date":"2024-04-06T22:32:43","date_gmt":"2024-04-07T03:32:43","guid":{"rendered":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame0\/?p=217"},"modified":"2024-04-06T22:32:43","modified_gmt":"2024-04-07T03:32:43","slug":"team-status-report-for-4-6","status":"publish","type":"post","link":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame0\/2024\/04\/06\/team-status-report-for-4-6\/","title":{"rendered":"Team Status Report for 4\/6"},"content":{"rendered":"

The most significant risk right now is the accuracy of our flow state detector. After determining a good level of accuracy on our testing data from Professor Dueck in a musician setting, we went on to perform validation in the other settings this week. Rohan performed and recorded EEG data for an activity to stimulate flow state (playing a video game) with noise-canceling headphones and music to encourage flow state for 15 minutes. He then recorded a baseline of performing a regular work task with distractions (intermittent conversation with a friend) for another 15 minutes. On the intended flow recording, our model predicted .254% of the time was in flow and in the intended not-in-flow recording, our model predicted .544% of the time was in flow. We have a few ideas as to why the model may be performing poorly in this validation context. First of all we think that in both settings, Rohan did not enter a flow state because neither task was second nature or particularly enjoyable and also 15 minutes is likely too short of a period to enter a flow state. To further validate the flow detection model, we plan to have Rohan\u2019s roommate who is an avid video gamer wear the headset while playing games for an extended period of time to see how the model performs in detecting flow in a gaming environment. Depending on how this goes, we plan to validate the model again in the music setting to see if our model has overtrained to detect flow specifically in a musical setting.<\/span><\/p>\n

Rohan also implemented a custom focus state detector. We collected recordings of Arnav, Karen, and Rohan in 15 minute focused and then 15 minute distracted settings while collecting EEG data from the headset. The model achieved high test accuracy on data it had not seen before and had strong precision, recall, and F1 scores. We collected data with Karen wearing the headset again, this time for 20 minutes in focus and 20 minutes distracted to use for model validation. When we ran this data through the custom focus detector, we saw a disproportionately high amount of distracted classifications and overall poor performance. We realized that the original training set only had 28 high quality focus data points for Karen compared to 932 high quality distracted data points for Karen. So, we attribute the poor performance to the skewed training data, plan to incorporate this validation data as training data for the model, and collect new validation data to ensure that the model is performing well. As a backup, we inspected the Emotiv Performance Metrics for focus detection and saw a clear distinction in the average focus Performance Metric in the focused recording as compared to the distracted recording.\u00a0<\/span><\/p>\n

Finally, as an attempt to further validate our custom models, Rohan applied Shapley values which are a measure used in explainable AI to understand which input features are contributing most significantly to the flow vs not in flow classification.\u00a0<\/span><\/p>\n

Validation for video processing distraction detection:<\/span><\/p>\n