Posted on

Mathias’ Status Report 12/7/2024

This week I focused on making the backend more robust. Previously the backend only worked when tested with correct inputs so if I sent in an invalid input there would be undefined behavior/system crashes. To do this I first changed the scanning portion of the code so that if not musicXML was produced we would return a error back to the caller instead of failing. I also changed the highlighting code to check if the beats lie within a specified range before attempting to highlight.

 

This week I’ll be mostly working on integration, I’ll be working with integrating Aakash’s solution into my component and having that all run on the Raspberry Pi.

Posted on

Mathias’ Status Report 11/30

This week I focused on creating a functional UI that links to the backend. The UI has three major components a component that allows for the upload of a sheet music pdf to the backend, a component that allows for a user to play a song that they uploaded, and lastly a component that allows a users results from one of their sessions. The upload component is simply a button that allows a user to select a file from their device. The play song and view session component are separate pages that have a drop-down to allow users to select a song/result respectively.

I also spent time fixing a bug in the highlighting logic. Previously the logic always assumed that a quarter note was one beat however this wasn’t always the case so I changed this to be based on the value in the time signature.

In regards to testing I was able to quantify the accuracy of the sheet music scanning more formally. For a intermediate piece such as Fly Me to the Moon the sheet music scanning is at 99% accuracy for both timing onset and pitch detection.

In terms of new knowledge I learned about the basics of OpenCV as well as some general music knowledge. My learning method for most of the new content I had to learn was to learn incrementally meaning I would only learn about what I would need at the time. If I needing a specific piece of functionality from OpenCV or Audiveris instead of front loading all the learning I would just learn about that component I need.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

The project is on track for the final presentation.

What deliverables do you hope to complete in the next week?

Polishing up the UI and feedback as well as working to better integrate all the project components.

Posted on

Mathias Status Report for 11/16

This week I polished the post performance feedback as well as created an endpoint for a user to upload a piece of sheet music, running the sheet music conversion during the upload.

For the post performance feedback, I extended it such that the feedback would work if the sheet music was multiple pages. I did this by maintaining a global beat counter in each element of the position list. I also made the highlighting more accurate. Before the y-position of the highlighting was based off the y-position of the corresponding note. Now the y-position is based on the y-position of the staff lines. This helps since previously if the y-positions of the start and end note were similar the box around that area would be very thin which is no longer an issue. I also updated how I do the highlighting from a simple rectangle around the area to a proper highlight . This was done by adding a weighted yellow numpy array to the specified area of the image.

Example of changes(Note in the test I ran for this two areas were out of sync)

 

For the upload it takes in a file from a request which is then saved locally. A process running audiveris is then run on the file to produce the mxl and omr files which are then moved to a designated folder. These files are then unzipped and deleted.

 

For validation I have mostly been testing these components with Postman to simulate API requests. I’ve also created a short python script to simulate the input from the timing algorithm for the post performance feedback section. So far the tests have been generally successful with both the  post performance and file upload however most of my tests have been happy case tests and more tests need to be done to see how the system handles failure and unexpected inputs.

 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

The project is on track for the interim demo this upcoming week.

What deliverables do you hope to complete in the next week?

Time will largely be spent working towards the demo and then I will continue work on the UI.

Posted on

Mathias’ report for 11/9

I spent this week integrated the post performance feedback component into an API which would allow me to call it from the web application. I created the API using flask and set it up as a POST endpoint which takes in a song name in the request. On receiving a request I then spawn a process to handle receiving data from the timing algorithm and marking the sheet music. The process uses regular python sockets to receive timing information

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

On Schedule

What deliverables do you hope to complete in the next week?

I hope to polish the UI and link this functionality to the UI

Posted on

Mathias’ Status Report 11/2

This week I worked on the post performance feedback component on the web application. This part involved highlighting areas where the musicians are out of sync based on data given by the timing algorithm. To do this I took the audiveries sheet.xml output which contains the location of each and combined parts of it with the audiveries music.xml output to create a list of notes that contains there duration in beats as well as location. From this I’m able to specify a beat range and highlight it. For example if say beats 8 to 15 are out of sync this is the output is:

https://github.com/mathiast270/18500/tree/main/backend

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

On Schedule

What deliverables do you hope to complete in the next week?

I hope to polish the UI and link this functionality to the UI

Posted on

Team Status Report for 11/2

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The risks to the project as similar to what they are before. We are planning to have a basic demo prepared for the musicians on Wednesday. From this demo we will try to see how much the imperfect accuracy of each component effects the overall systems ability to provide feedback and adjust the specifics of the system accordingly.  A contingency plan if adjustments don’t fix the issues would be to simplify the pieces played to pieces where we know we can have a high accuracy for.

 

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

Changes made: None

Posted on

Mathias’ Status Report for 10/26

Most of my time this week was spent making a prototype of the frontend web application. Familiarized myself with Expo a react native framework and made an extremely crude prototype(https://github.com/mathiast270/InSyncAPp). The prototype has three major pages the main page having a buttons to navigate to the other pages. The play_song page contains a drop down to select songs(currently just hard coded values) it also contains a text field to let a user select bpm. The other page allows a user to take a picture through the expo camera library. Currently all the functionality is not connected to the back end at all

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

On Schedule

What deliverables do you hope to complete in the next week?

Parse the XML output from Audiveries to get note location and use that to implement the post performance feedback component.

Posted on

Ben’s Status Report 10/20

During the week my primary focus was creating a connection between the Scarlett USB device (one which I own) and linking it into the skeleton of the pitch detection script from earlier. This was in preparation for a similar device we are purchasing. I was able to successfully create a program using the pyaudio library, and the pysudev library that detects when a Scarlett audio interface is connected and finds a path that can be used in the audio processing script with Aubio. The current difficulty is integrating these two programs as the device detection is buggy at times and requires consistent polling to function. I also need to add edge cases for if the Scarlett device becomes disconnected during operation.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

In terms of schedule, everything seems to be on track. The majority of the tasks ahead are things I have to learn so my concept of how long they will take is purely an estimate. With that being said, my hope is that a MVP could be reached in about 2 weeks. This would look like a working note detection program which outputs an event list which will be used by the timing algorithm. I imagine most of the time over the next few weeks will be spent debugging and optimizing the systems.

What deliverables do you hope to complete in the next week?

In terms of deliverables for the end of the week I’d like to have these successfully integrated and to a point where I can begin testing the pitch and note onset detection libraries. This would keep me on the expected schedule.

Posted on

Mathias Status Report for 10/20

Most of this week was spent working on the design report. I worked on the mostly on the web application/ sheet music scanning sub systems of the report as well as parts of the introduction and design considerations. The major thing that has changed since last report is the OMR I will be using, instead of Mozart I will be using Audiveris. The report details many of the design considerations between the two but as a summary of why, Audiveries not only contains a more robust output that gives duration as well as the position of each note in regard to the sheet music image. This also means the backend will change from a flask backend to a spring boot backend since the library is in Java.

 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

A bit behind since I will have to redo certain parts of the to have it be in spring boot instead of flask.

 

What deliverables do you hope to complete in the next week?

Backend with full ability to scan in sheet music as well as be start working on the ability to color in certain areas of an image for the post performance feedback.

Posted on

Mathias Status Update 10/5

This week I’ve mostly spent trying to polish the sheet music conversion. I had an issue last week where for many of sheet music files after splitting and passing through Mozart the final output would have no output. First step I made in debugging was changing realize that my resizing algorithm had a mistake. To split the sheet music by line I used the character that appears at the start of every measure. To handle the case that every image may not be the perfect size for the template I’m using I resize the image from a scale of .1x to 10x. In the previous implementation of the algorithm we only scaled from .1 to 1 due to a bug. Fixing this bug allowed for various size images to work however it did not fix the Mozart output. To address this I tried to split the sheet music image based on the squiggly character at the beginning of each measure and pass that result into Mozart however that didn’t work as well. The last thing I tried was to resize and sharpen the final image result to a similar size to the one that worked however this approach did not work as well. Due to this I spent some extra time this week lightly checking solutions outside of Mozart as well.

 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Progress is on schedule so far.

 

What deliverables do you hope to complete in the next week?

Most of the time next week will be spent on the design report. Outside of that I’ll spend more time trying to get Mozart to work and if that fails look into a different solution.