Posted on

Deeya’s Status Report for 4/19/25

This week I focused on finishing up all the functionality and integration of the website. With Shivi I worked on adding the key and time signature user input and integrating in the backend for generating the midi file.

 

I also worked on the Past Transcriptions page where the user can view each transcription they have made with the web app including the composition name which the user names when they input/record an audio, the bpm of the piece, the generated sheet music, the date it was created, and the audio controls to playback the audio. 

The only thing left is integrating the edit functionality which is a paid API call. We are waiting to hear back if it is possible to get reimbursed and we will explore another option which is to redirect the user to the Flat.io page to edit the composition directly there instead of on the web app.

 

Tools/knowledge I have learned throughout the project:

  • Developing a real-time collaborative web application leveraging WebSockets and API calls – Youtube videos and reading documentation
  • How to get feedback from the flute students and then be able to incorporate it to make the UI more intuitive
  • How to incorporate the algorithms Shivi and Grace have worked on with the frontend and being able to update it seamlessly whenever they make any updates – developing strong communication within the team
Posted on

Shivi’s Status Report for 4/19/25

After the live testing session with flutists on Sunday, we found that when performing two-octave scales, certain higher octave notes were still being incorrectly detected as belonging to the lower octave; after making some adjustments to the HSS pitch detection, they seem to be working correctly now. I also modified the MIDI encoding logic to account for rests. On the web app side, I worked with Deeya to incorporate time and key signature user inputs, and our webapp now supports past transcriptions as well. We also expired ways we could make the sheet music editable directly within the webapp. Since Flat.io API only supports read-only display with the basic subscription and we still have not heard back regarding access to the Premium version, we are planning to redirect users to the full editor in a separate window for now. Finally, I worked on the final presentation that is scheduled for next week.

In terms of the tools/knowledge I’ve picked up throughout the project:

  1. I learned to implement signal processing techniques/algorithms from scratch. Along the way, I learned a lot about pitch detection and what the flute signal specifically looks like and how we can use its properties to identify notes
  2. Web app components such as websockets and integration with APIs like Flat.io
  3. Familiarity with collaborative software workflows with version control, documenting changes clearly, and building clean/maintainable web interfaces with atomic design. We encountered some technical debt in our codebase, so a lot of time also was spent in refactoring for clarity and maintainability
  4. Conducting user testing for our project and collecting data/feedback to iterate upon our design
Posted on

Team Status Report for 4/19/25:

This week, we tested staccato and slurred compositions and scales with SOM flute students to evaluate our transcription accuracy. During the two-octave scale tests, we discovered that some higher octave notes were still being misregistered in the lower octave so Shivi worked on fixing this in the pitch detection algorithm. Deeya and Shivi also made progress on the web app by enabling users to view past transcriptions and input key and time signature information. Grace is working on improving our rhythm algorithm to better handle slurred compositions by using Short-Time Fourier Transform (STFT) to detect pitch changes and identify tied notes within audio segments. However, we’re still working on securing access/reimbursement for the Embed AP in Flat.io, which is needed to allow users to edit their compositions. For this week we are preparing for our final presentation, planning on doing two more testing sessions with the flute students, and cleaning up our project.

Posted on

Grace’s Status Report for 4/19/25

This week has been continuing to fine tune the algorithm for slurred notes and rests. We made the switch last week to STE from RMS and we saw better results as it kept the near zero values closer to zero, however with slurred notes the values still do not get close enough.

I created an algorithm that would look through the segments from the original algorithm and if it has any times, then it would “flag” the segment as a possible slurred note. Then I would check for pitch changes. I first used spectrogram but found that iw would miss some notes or incorrectly identify where the notes actually change, so I switched to using STFT and having a ratio varying based on BPM to detect smaller note changes with faster tempos with better success. Here is a picture of using the spectrogram

and using the STFT

While there are still some inaccuracies, it is much better than before. Currently working on fine tuning the rest detections as right now it has a tendency to over detect rests. Also looking into using CNNs for classification of slurred notes

While implementing this project, I learned more about signal processing and how some things are much easier to identify manually/visually than coding it up. Additionally, reading up on how much people research into identifying segmentations in music and how different types of instrument can add to more slurs as they tend to be more legato. For this project, I needed new tools on identifying new notes like using STFT and STE. To learn more about these, I would read research papers from other projects and universities on how they approach it and tried to combine aspects of them to get a better working algorithm.

Posted on

Deeya’s Status Report for 4/12/25

This week I focused on making the web app more intuitive. Shivi had worked on the metronome aspect of our web app which allows the tempo to change as you change the metronome value using Websockets. I integrated her code into the web app and took the metronome value into the backend so that the bpm changes dynamically based on the user. I also tried to get the editing feature of Flat IO but it seems that the free version using iframe doesn’t work. We are thinking of looking into the Premium version so that we can use the Javascript API. The next step is to work on this and add a Past Transcriptions page.

Posted on

Grace’s Status Report for 4/12/2025

Worked on the issues mentioned during interim demo, which were not being able to accurately detect slurs and not picking up the rests in songs.

First, experimented with modifying the code to Short Time Energy (STE). This helped the code become more “clear” as there were less bumps and more clear near zero values, essentially eliminating some of the noise that stayed with RMS. Should make amplifying the signal a lot easier now. However, still having some difficulty seeing the differences in slurred notes, so doing some additional research in onset detection to detect slurred notes specifically rather than look for the separation of notes in segmentation.

 (forgot to change label for the line, but this is a graph for STE – this was taken using audio from a student in the school of music)

For rests, modified my rhythm detection algorithm to instead look for zero values after reaching the peak (means the note is done playing) and taking the additional length after the note to count as a rest. Sometimes takes slight moments of silence to distinguish notes as a rest though so need to do some experiments to make it less sensitive.

Posted on

Team Status Report for 4/12/25

Last week, we had a successful interim demo where we had our transcription pipeline working for a recording of Twinkle Twinkle Little Star. We also met with a flutist from the School of Music to get her feedback on our pipeline and obtain some sample recordings. She found the interface intuitive and easy-to-use, though we did run into some bugs with audio file formats and found that our note segmentation struggled a little with slurred notes. 

This week, we focused on the following items:

  1. Fix the denoising step
  2. Set up websockets for real-time BPM adjustment
  3. Any remaining frontend-backend integration remaining for the web app. (for e.g., earlier we had some bugs with audio file formats and with recording audio directly via the web app)
  4. Using a Short Time Energy approach instead of RMS to perform note segmentation. This helped to better account for rests/slurs in the music.

Later this weekend, we are meeting again with another flutist from the SoM to obtain more audio and to see if our note segmentation performs better this time. Our new audio interface and XLR cable also arrived this week, so we will hopefully be able to collect better audio samples as well. In the upcoming week, we will focus on:

  1. Polishing our STE/note segmentation
  2. Fixing the issues with making our sheet music editable via the Flat API
  3. Collecting user metrics such as their transcription history
  4. Deploying our web app
  5. Preparing our final presentation/demo
  6. Thorough testing of our pipeline

Below is our plan for verification, which we already started last week.

Posted on

Shivi’s Status Report for 4/12/25

 

This week, I first worked on fixing the denoising step so that the note octaves would be accurate. Earlier, the notes would sometimes come out an octave higher because the bandpass filter was cutting out some of the lower frequencies, so I adjusted the frequency range to prevent this from happening. I also set up the websocket for real-time adjustment of the metronome, so the user is now able to adjust the tempo of the composition. Deeya and I integrated all of the webapp code and have been trying to figure out how to make the generated composition editable via the Flat API; unfortunately, we have been running into a lot of issues with it but are going to continue debugging this this week. I am also adding inputs for the user to be able to specify a key signature and time signature. Overall, my progress is on track. Pitch detection and MIDI encoding is largely done, and in the upcoming week, I will be focusing on resolving the issues with editing the sheet music directly through our web app using the Flat API and adding the key/time signatures. 

Posted on

Shivi’s Status Report for 3/29/25

This week, I worked on preparing for the interim demo. I refined my pitch detection to account for rests and ensure that the generated notes were accurate (i.e. earlier, some notes were incorrectly being marked as flat/sharp instead of natural). Then, I worked with Deeya to set up the Flat.io API, as we were running into several errors with authorizing and formatting sending/receiving requests and responses. However, we were able to figure out how to send our generated MIDI files to the API for processing into sheet music. Finally, Grace and I worked on ensuring compatibility between our code, and I finished modularizing all our existing code and integrating it into a single pipeline that gets triggered from the web app and runs in the backend. Pitch detection is mostly done, and for next steps, I will be working on:

  1. Tempo detection
  2. Setting up websockets for our webapp for real-time adjustment of the metronome + assisting Deeya with making the displayed sheet music editable
  3. Working with Grace to refine audio segmentation (ex: rests and incorporating Short-Time Energy for more accurate note duration detection)

I am also finding that when I incorporate the denoising step into the pipeline, the detected pitches are thrown off a bit, so I’ll have to look more into ensuring that the denoising step does not impact the pitch detection.