Category: Uncategorized

This week, we researched parts for our design implementation and questioned our product in order to solidify the main goals we are pursuing. We have compiled a parts list and found APIs to integrate with different components of our project.

The most significant risks we are facing are in regard to the robust audio detection and processing algorithm. We want to be able to turn the page based solely on aligning audio inputs to the sheet music being displayed, however having extremely flexible ML models to process this is extremely challenging to execute given our backgrounds. To fix this, we intend to use standardized measured sheet music and pre-processed MIDI files of the music.

We are working on several changes to the design at the moment. Design proposal coming soon!

This week, I researched eye tracking algorithms and real-time machine learning audio processing algorithms in order to better quantify requirements for latency, precision, and accuracy for both eye tracking and audio processing.

I also worked on the specifications noted in the design proposal and further refined what open source APIs are available for sheet music processing into MIDI files.

I worked on the design proposal presentation. I worked on the Use case Requirements slide containing the audio latency and audio accuracy. I also worked on the Technical challenges slide, solution approaches slides, and the testing verification & metrics slide. Lastly, my team and I worked on the User Experience Flow chart and data path for the last slide. I also researched the Jetson Nano Dev Board as a possible substitute for our Google Coral Board. Additionally, I read through the audio alignment sheet music research paper published in 2008 for possible ideas for audio alignment for our project.

I worked on collecting the specifications needed for the Tobii Eye-Tracker 5. This ensures the camera is operatable by our Google Coral board. Also, downloading the Pro Developer SDK for the camera allows all devices to communicate through Python. I also worked on collecting information on how to filter the eye-tracker data to improve both precision and accuracy. This consisted of reading through a Microsoft paper detailing different filters and their overall effect on eye-tracking accuracy and precision.

Source: https://www.microsoft.com/en-us/research/wp-content/uploads/2017/01/everyday_eyetracking-1.pdf

We are excited to present SoundSync, a novel system that automatically turns pages during rehearsals and performances. While playing, musicians have to turn pages, resulting in a loss of focus and inability to make music while they’re using their hands to flip a page. Musicians may have the choice of having a professional page turner or foot pedal, however both result in disruptions. SoundSync aims to autonomously turn pages in real time using eye tracking and audio processing to ensure accuracy and focused playing for the duration of practice.