Weekly Status Reports

Posted on

Ben’s Status Report 11/16

This week was spent getting the DSP in a good place for the interim demo. This meant using slightly more complex music (the example being tested this week is Fly Me to the Moon) which has chords, different note timings, and imperfect singing to see if the audio could be properly parsed.

In tandem, features to interface with user input were added. In the final product, these signals will ideally come directly from the web app but, for now, can be directed through command line inputs. The program can process existing sound files or attempt to record from scratch, save, and process the audio. This allows the program to mimic the full functionality we hope to have once all parts are combined.

Time was also spent manually slicing and cleaning up audio files from the recording sessions which are located in the Recordings folder of the project. This was necessary since each take needed to be broken up seperately and the vocal and piano tracks needed to be isolated into their own files respectively. 

Aubio is producing more audio events than there are notes in the piece. This makes sense, as deviations in playing (particularly singing) seem to cause multiple notes to be generated. Remove erroneous data from the stream (i.e. midi values that correspond to octaves not being reached). This removes any of these spikes but also results in some missing data. More testing is needed to determine which spikes are just noise and which are valid events (with wrong pitch data).

In addition, a method of “pitch stitching” is in progress but needs work. This would combine sequential pitches of very close or equal values into one event. So far this has yielded mixed results with increasing error as the complexity of the piece increases.

Updates to the DSP test files and main process can be found here

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

The project is on track for the interim demo this upcoming week.

What deliverables do you hope to complete in the next week?

Time will largely be spent working towards the demo and then refining the pitch detection and timing output even more.

Posted on

Team Status Report for 11/16/2024

Now that you have some portions of your project built, and entering into the verification and validation phase of your project, provide a comprehensive update on what tests you have run or are planning to run. In particular, how will you analyze the anticipated measured results to verify your contribution to the project meets the engineering design requirements or the use case requirements?

Verification is usually related to your own subsystem and is likely to be discussed in your individual reports.
Validation is usually related to your overall project and is likely to be discussed in your team reports.

For validation tests, we want to ensure that when the system is given the same inputs, we get the same outputs every time. This can be done by giving the system the same sheet music and audio data and ensuring the music highlighting is the same.

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

So for the most significant risks are that we cannot properly process complex pieces of music. This risk is being managed by slowly increasing the complexity of the music we process to ensure that we don’t have any surprises as we test more complicate pieces. The contingency plans we have are to just stick with simpler pieces of music because it is working so far so if there are complex pieces of music we have trouble processing, we will treat them as edge cases and focus our energy on polishing what we know works.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

No changes made.

 

Posted on

Aakash’s Status Report for 11/16/2024

This week I spent a lot of time this week making sure my system is robust and we are ready for the demo next week. We are going to demo on a simpler piece of music that doesn’t have complex chords, so am making sure the system is reliable. The current state of the system has it so it parses the sheet music data from the music xml and gives an output as so in the form of (pitch, onset time, duration)

It then takes this data and compares it to the audio data to find where the similarities of the two. This is done using a modified dynamic time warping algorithm and prioritizes onset time, then duration, then pitch when comparing two notes. This then outputs a list of (sheet music index, audio index) which are indexes where the audio matchest the sheet music. This looks like this:

This still needs a little work because when the audio data isn’t edited, there is a lot of noise that can disrupt the algorithm. This can be mitigated by doing pre processing on the data and to allow the algorithm to be more selective when matching indexes. Currently every index has to be match to another, but it can be modified so this is not the case. Once these changes are implemented it should be way more accurate. The preprocessing is going to be done by implementing a moving average to get rid of the erratic note anomalies. This should be relatively easy to implement. I will have to do more research on dynamic time warping and see if there is a signal processing technique I can implement to help with matching the erroneous notes, and if there isn’t,  I will manually iterate through the list and remove audio data notes that match with sheet music data more than once which should increase accuracy.

I have worked on increasing the accuracy of the matching audio by changing parameters and fine tuning the distance function within my custom dynamic time warping implementation. I am still learning the details about how this works mathematically in order to create an optimized solution, but I am getting there. Getting better data from Ben will certainly help with this as well but I wan’t to make sure the system is robust even with noise.

It does this for both the singer portion of the sheet music and the piano portion. It then finds where there are simultaneous notes in both portions in order to check there for whether they are in sync of out of sync. It gives an output of (singer index, piano index) of where to check in the sheet music data for similar notes. This is as so:

It the compares the audio data to the shared note points and sees if they are similar to a certain threshold. This threshold is currently .01 seconds. It then sends the sheet music time to the frontend which then highlights that note.

 

Verification:

Some of the verification tests I have currently run is when processing the sheet music data, manually comparing that data to the sheet music pdf to ensure it looks correct. This isn’t a perfect way to go about this but it works good enough for now and I don’t know of any automated way to do this because I am creating the data from the MusicXML. Perhaps a way to automate this verification is to create a new MusicXML file from the data I parsed and compare the two to ensure they look correct.

Some other tests I plan on doing is

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am on schedule. There is still a lot of refining to do before the final demo, but the core functionality is there and its looking very promising.

What deliverables do you hope to complete in the next week?

I want to keep working on improving the system so we can processing more complicated systems. For our demo we have it working with simple pieces, but I want to improve the chord handling and errors.

Posted on

Mathias Status Report for 11/16

This week I polished the post performance feedback as well as created an endpoint for a user to upload a piece of sheet music, running the sheet music conversion during the upload.

For the post performance feedback, I extended it such that the feedback would work if the sheet music was multiple pages. I did this by maintaining a global beat counter in each element of the position list. I also made the highlighting more accurate. Before the y-position of the highlighting was based off the y-position of the corresponding note. Now the y-position is based on the y-position of the staff lines. This helps since previously if the y-positions of the start and end note were similar the box around that area would be very thin which is no longer an issue. I also updated how I do the highlighting from a simple rectangle around the area to a proper highlight . This was done by adding a weighted yellow numpy array to the specified area of the image.

Example of changes(Note in the test I ran for this two areas were out of sync)

 

For the upload it takes in a file from a request which is then saved locally. A process running audiveris is then run on the file to produce the mxl and omr files which are then moved to a designated folder. These files are then unzipped and deleted.

 

For validation I have mostly been testing these components with Postman to simulate API requests. I’ve also created a short python script to simulate the input from the timing algorithm for the post performance feedback section. So far the tests have been generally successful with both the  post performance and file upload however most of my tests have been happy case tests and more tests need to be done to see how the system handles failure and unexpected inputs.

 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

The project is on track for the interim demo this upcoming week.

What deliverables do you hope to complete in the next week?

Time will largely be spent working towards the demo and then I will continue work on the UI.

Posted on

Ben’s Status Report 11/9/24

This week, I was not as productive as I would have liked do to factors outside of my control, however progress was made. As it stands, the audio processing component successfully takes in a basic audio file, and produces a correctly formatted, and correctly populated output. Seen below, the output is broken into three values per row in the form:

[midi value, start time (seconds), duration (seconds)]

This was generated after running a basic audio test file created by the midi output of this simple score:

It can be seen, however, there is some slight error in the process of determining exact duration as is expected. It can be seen that all notes started almost exactly 0.5 seconds after each other. Since the audio file was simply a uniform midi output, the duration of each note should be the exact same though the test yields some variance. Duration is a difficult value to accurately assess as it requires primitively determining the silence threshold, the point at which notes are considered not-playing or “silent.” In this case, the value is set at -40dB which is the standard value for the Aubio library and other recording standards.

All files have been added to the DSP folder in the group git linked here.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am behind schedule due to external circumstances. With that being said, I have concentrated my efforts and am making back up that time but have accounted for delays by using up some of the slack planned out from the beginning of the project

What deliverables do you hope to complete in the next week?

The system is in the early stages of integration and will hopefully be able to run in a limited demo mode by the end of the week. This would look like running the system with a very basic score and midi audio.

Posted on

Mathias’ report for 11/9

I spent this week integrated the post performance feedback component into an API which would allow me to call it from the web application. I created the API using flask and set it up as a POST endpoint which takes in a song name in the request. On receiving a request I then spawn a process to handle receiving data from the timing algorithm and marking the sheet music. The process uses regular python sockets to receive timing information

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

On Schedule

What deliverables do you hope to complete in the next week?

I hope to polish the UI and link this functionality to the UI

Posted on

Aakash’s Status Report for 11/09/2024

This week I spent a lot of time making sure the MusicXML parsing is accurate. There was some bugs when parsing the piano portion of the MusicXML as the notes that played at the same time were being interpreted as being sequential notes. I was able to fix this and ensure that the MusicXML is being parsed accurately from just manually cross checking the note list with the sheet music pdf.

I also was able to produce a list of where there were timing errors. This can be used to send to Mathias in order to highlight it on the sheet music. I also worked on cleaning up the code.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am slightly behind schedule as I wasn’t able to test on the data produced from the audio were recorded, but this isn’t a major hurdle as we still have ample time to make sure everything is working.

What deliverables do you hope to complete in the next week?

I hope to be able to start doing system integration with Ben and Mathias to make sure we have a working demo ready so we have ample time to make sure we are ready for the interim demo.

Posted on

Team Status Report for 11/9

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

We still have the risk of system integration as we are all having success with our individual parts of the project, but we haven’t spent much time integrating them and seeing if there are any weird bugs or edge cases. This can be mitigated by starting to to integration soon so we have ample time before the interim and final demo to make sure our system is stable and reliable.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

No changes made.

 

Posted on

Mathias’ Status Report 11/2

This week I worked on the post performance feedback component on the web application. This part involved highlighting areas where the musicians are out of sync based on data given by the timing algorithm. To do this I took the audiveries sheet.xml output which contains the location of each and combined parts of it with the audiveries music.xml output to create a list of notes that contains there duration in beats as well as location. From this I’m able to specify a beat range and highlight it. For example if say beats 8 to 15 are out of sync this is the output is:

https://github.com/mathiast270/18500/tree/main/backend

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

On Schedule

What deliverables do you hope to complete in the next week?

I hope to polish the UI and link this functionality to the UI

Posted on

Team Status Report for 11/2

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The risks to the project as similar to what they are before. We are planning to have a basic demo prepared for the musicians on Wednesday. From this demo we will try to see how much the imperfect accuracy of each component effects the overall systems ability to provide feedback and adjust the specifics of the system accordingly.  A contingency plan if adjustments don’t fix the issues would be to simplify the pieces played to pieces where we know we can have a high accuracy for.

 

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

Changes made: None