Category: Mathias’s Status Reports

This week I focused on making the backend more robust. Previously the backend only worked when tested with correct inputs so if I sent in an invalid input there would be undefined behavior/system crashes. To do this I first changed the scanning portion of the code so that if not musicXML was produced we would return a error back to the caller instead of failing. I also changed the highlighting code to check if the beats lie within a specified range before attempting to highlight.

 

This week I’ll be mostly working on integration, I’ll be working with integrating Aakash’s solution into my component and having that all run on the Raspberry Pi.

This week I worked on the post performance feedback component on the web application. This part involved highlighting areas where the musicians are out of sync based on data given by the timing algorithm. To do this I took the audiveries sheet.xml output which contains the location of each and combined parts of it with the audiveries music.xml output to create a list of notes that contains there duration in beats as well as location. From this I’m able to specify a beat range and highlight it. For example if say beats 8 to 15 are out of sync this is the output is:

https://github.com/mathiast270/18500/tree/main/backend

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

On Schedule

What deliverables do you hope to complete in the next week?

I hope to polish the UI and link this functionality to the UI

Most of my time this week was spent making a prototype of the frontend web application. Familiarized myself with Expo a react native framework and made an extremely crude prototype(https://github.com/mathiast270/InSyncAPp). The prototype has three major pages the main page having a buttons to navigate to the other pages. The play_song page contains a drop down to select songs(currently just hard coded values) it also contains a text field to let a user select bpm. The other page allows a user to take a picture through the expo camera library. Currently all the functionality is not connected to the back end at all

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

On Schedule

What deliverables do you hope to complete in the next week?

Parse the XML output from Audiveries to get note location and use that to implement the post performance feedback component.

Most of this week was spent working on the design report. I worked on the mostly on the web application/ sheet music scanning sub systems of the report as well as parts of the introduction and design considerations. The major thing that has changed since last report is the OMR I will be using, instead of Mozart I will be using Audiveris. The report details many of the design considerations between the two but as a summary of why, Audiveries not only contains a more robust output that gives duration as well as the position of each note in regard to the sheet music image. This also means the backend will change from a flask backend to a spring boot backend since the library is in Java.

 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

A bit behind since I will have to redo certain parts of the to have it be in spring boot instead of flask.

 

What deliverables do you hope to complete in the next week?

Backend with full ability to scan in sheet music as well as be start working on the ability to color in certain areas of an image for the post performance feedback.

This week I’ve mostly spent trying to polish the sheet music conversion. I had an issue last week where for many of sheet music files after splitting and passing through Mozart the final output would have no output. First step I made in debugging was changing realize that my resizing algorithm had a mistake. To split the sheet music by line I used the character that appears at the start of every measure. To handle the case that every image may not be the perfect size for the template I’m using I resize the image from a scale of .1x to 10x. In the previous implementation of the algorithm we only scaled from .1 to 1 due to a bug. Fixing this bug allowed for various size images to work however it did not fix the Mozart output. To address this I tried to split the sheet music image based on the squiggly character at the beginning of each measure and pass that result into Mozart however that didn’t work as well. The last thing I tried was to resize and sharpen the final image result to a similar size to the one that worked however this approach did not work as well. Due to this I spent some extra time this week lightly checking solutions outside of Mozart as well.

 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Progress is on schedule so far.

 

What deliverables do you hope to complete in the next week?

Most of the time next week will be spent on the design report. Outside of that I’ll spend more time trying to get Mozart to work and if that fails look into a different solution.

 

 

This week in addition to assisting gathering data from the school of music I spent most of my time working on converting the sheet music to an independent file format. Before I was mentioning how there was an issue where I needed to split the image for the library to work.  I tested that this week and confirmed that the sheet music needed to be split at least by measure for the library to work correctly. I then looked into methods to split a sheet a piece of sheet music by measure anf decided on splitting it based on the special character that appears at the beginning of the sheet music. I wrote a python script that used opencv to check if the special character appears in a sheet music image, highlight the character, then parse based on the position of the character. Initially testing this by passing the result of the parsing into the Mozart library showed somewhat successful results but the edge cases need to still be handled.

 

 

 

 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule.

On schedule

 

What deliverables do you hope to complete in the next week?

I hope to work mostly on the design presentation and design report next week. I also hope to iron out some of the edge cases with the conversion.

 

I also want to look into whether we need to convert to MIDI. I  From the conversion to Mozart’s format we are already given the notes played which should be the key information we would need so I would like to look into whether we could bypass the midi conversion entirely.

This week assisted in preparing for the project presentation. For the project I was assigned to work on the sheet music scanning as well as the web app. I did some research on other methods to convert sheet music outside of the previously suggested Mozart library and starting on setting up a skeleton for the web application. I tested the Mozart library with sheet random sheet music images from the internet to check if it would work with arbitrary sheet music however the library produced empty outputs for these files. I looked into other methods to convert sheet music but most of them were not maintained well or dont do what I need i music21. Luckily in the github issues for Mozart there is a explanation saying that its necessary to split the image into multiple sections so I’ll be looking into that later. For the web app I got a sample file upload endpoint implemented

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule.
On Schedule

What deliverables do you hope to complete in the next week?

I hope to finalize my research into the sheet music scanning and have working stub endpoints the the web application.