I began coding a user interface in Python with basic functionality – to upload a sheet music file/MIDI file and pick an instrument. This will later be integrated with the backend. My team and I spent considerable time on the design review report as well. We also ordered parts and read several of Dr. Dannenberg’s past papers. One major technical takeaway from this was on the subject on spectral analysis and chroma vectors. One paper discussed how chroma vectors reduce detailed spectra to 12-element vectors, each representing energy associated with one of the 12 pitch classes (there are 7 notes and 5 unique half steps between notes). Comparing these chroma vectors provides a robust distance metric for score alignment, which is one of our biggest challenges. I also looked into how segmentation, along with state graphs and penalty functions, could be used to handle complex realtime audio data.
With the exception of eye tracking, I am on schedule. The eye tracker was ordered this week, so I expect to begin collecting data from it in the next 2 weeks. As individual components are completed, they will be continuously integrated with my frontend.
My deliverables for next week are a frontend that’s able to display some data from either the microphone or camera.