This week my team focused again on integration of the three systems: Front-End, Eye-Tracking, and Audio Alignment. Mainly, we worked on making our audio alignment update the cursor in a more robust manner. This week my team did not really work separately. We all worked on improving audio alignment and eye tracking together during our work sessions this week.
For audio alignment, we worked on reducing our alignment computational latency and reducing overall audio alignment latency. There was a problem where the computing chroma matrix for the first audio sample had too high of a latency, around 1.5 seconds. We wanted our overall audio latency to be under 500 milliseconds. However, the computations that occur for the remaining audio samples averaged around 20 milliseconds. To solve this problem, we called the function before we do the first costly computation before the audio alignment starts to ensure we never experience this latency during the user’s performance.
After that, Caleb, Sanjana, and I worked trying to make the cursor update more smoothly and quicker based off solely the audio alignment. To do this we made sure our audio alignment picks the best matching subset of notes detected when comparing them to all notes in the MIDI file. This required us to rewrite some of our existing algorithm for audio alignment. I also worked on improving the eye-tracking heuristic model with Sanjana and Caleb. We essentially made the interpolation more robust by sending the duration of the user looking at the line of music they are currently playing to help the audio alignment better decide where the user is located.
So far, my progress is on schedule. Until the next 2 demos we have quite some work to finish up to make the system more aesthetically pleasing and robust.