Author: shivij

This week, I first worked on fixing the denoising step so that the note octaves would be accurate. Earlier, the notes would sometimes come out an octave higher because the bandpass filter was cutting out some of the lower frequencies, so I adjusted the frequency range to prevent this from happening. I also set up the websocket for real-time adjustment of the metronome, so the user is now able to adjust the tempo of the composition. Deeya and I integrated all of the webapp code and have been trying to figure out how to make the generated composition editable via the Flat API; unfortunately, we have been running into a lot of issues with it but are going to continue debugging this this week. I am also adding inputs for the user to be able to specify a key signature and time signature. Overall, my progress is on track. Pitch detection and MIDI encoding is largely done, and in the upcoming week, I will be focusing on resolving the issues with editing the sheet music directly through our web app using the Flat API and adding the key/time signatures. 

This week, I worked on preparing for the interim demo. I refined my pitch detection to account for rests and ensure that the generated notes were accurate (i.e. earlier, some notes were incorrectly being marked as flat/sharp instead of natural). Then, I worked with Deeya to set up the Flat.io API, as we were running into several errors with authorizing and formatting sending/receiving requests and responses. However, we were able to figure out how to send our generated MIDI files to the API for processing into sheet music. Finally, Grace and I worked on ensuring compatibility between our code, and I finished modularizing all our existing code and integrating it into a single pipeline that gets triggered from the web app and runs in the backend. Pitch detection is mostly done, and for next steps, I will be working on:

1. Tempo detection
2. Setting up websockets for our webapp for real-time adjustment of the metronome + assisting Deeya with making the displayed sheet music editable
3. Working with Grace to refine audio segmentation (ex: rests and incorporating Short-Time Energy for more accurate note duration detection)

I am also finding that when I incorporate the denoising step into the pipeline, the detected pitches are thrown off a bit, so I’ll have to look more into ensuring that the denoising step does not impact the pitch detection.

This week, I focused on being able to write the detected rhythm/pitch information to a MIDI file and also looked into APIs for displaying the generated MIDI information as sheet music. Using the pitch detection I did last week, I wrote another script that takes in the MIDI note numbers and note types and creates MIDI messages. Each note is associated with a message that encodes its frequency, duration, and loudness, and the script generates a .mid file with all the notes and their corresponding attributes. I tested this on a small clip of Twinkle, Twinkle Little Star; for the generated .mid file, I then uploaded this to the music notation platform, flat.io to see if the .mid file contained the correct notation. Below is the generated sheet music. For now, all the note pitches were generated by my pitch detection script, but all the notes are hard coded as quarter notes for now as our rhythm detection is in progress. The note segmentation –> pitch detection –> MIDI generation pipeline seems to be generating mostly correct notes for basic rhymes like Twinkle Twinkle.

Earlier this week, I also did some research into APIs that we could use to display the generated sheet music on our web application in a way that is similar to MuseScore, a popular music notation application. While MuseScore doesn’t have an API that we can use, flat.io has a developer guide that will allow us to display the generated sheet music. Next week, I will be looking more into the developer guide and working with Deeya to set up/integrate the Flat API onto our web app. I will also work with Grace to refine/test our note segmentation more and ensure it is accurate for other notes and rests. We will also potentially be meeting one of the flutists this week so that we can collect more audio samples as well. Overall, my progress is on schedule, and hopefully we will have our transcription pipeline working on simple audio samples for our interim demo.

This week, I met with Grace to test the audio segmentation algorithm she wrote. We tested it on a sample of Twinkle Twinkle Little Star, as well as Ten Little Monkeys. We found that for each of the two samples, we needed to adjust the RMS threshold to account for differences in the maximum amplitude of the signal; as a result, we realized that we will need to add some way to either standardize the amplitude of our signal or dynamically change the RMS threshold based on the signal’s amplitude. 

I also worked to integrate our preprocessing and audio segmentation code all together. Our current pipeline can be found on this GitHub (Segmentation/seg.py for note segmentation, and Pitch Detection/pitch.py for pitch detection) along with some of our past experimentation code.

Furthermore, now that we have audio segmentation, I was able to get pitch detection to work, at least on Twinkle Twinkle.  To do so, I used FFT and comb filtering to find and map the fundamental frequency to the MIDI note. I plan to test the pitch detection on more audio samples next week and work with Grace and Deeya to integrate all the stages of our project that we have implemented so far (web app and triggering the preprocessing/audio segmentation/pitch detection pipeline). 

Last week, I mainly focused on working on the design review document with Deeya and Grace. Incorporating the feedback we received during the design presentation, I worked mostly on the preprocessing/calibration, pitch detection, and design trade studies aspects of the design document. Additionally, Professor Dueck connected us with Professor Almarza from the School of Music, and Deeya and I met with him and the flutists from his studio. This helped us confirm our use case requirements, get their opinion on our current user workflow, and solicit their availability for testing out our pipeline in a few weeks. The flutists were excited about the project as a composition tool such as the one we are developing would greatly aid them in writing new compositions. Grace and I also discussed how to implement the audio segmentation; as of now, we are planning to apply RMS over 10 ms windows of the signal and use spikes in amplitude to determine where the new note begins. Based on our research, similar approaches have been used in open-source implementations for segmenting vocal audio by note, so we are optimistic about this approach for flute audio as well. We are currently on schedule with our progress, but I anticipate issues with audio segmentation this week, so we plan to hit the ground running for this aspect of our project on Monday so that we can have the segmentation working, at least for recordings of a few quarter notes, by the end of the week.

This week, I spent most of my time working on the design review presentation and design review document. I also thought more about our current noise suppression method, for which we are using a Butterworth filter, spectral subtraction, and adaptive noise filtering. However, based on Professor Sullivan’s advice and my own experimentation with hyperparameters and various notes, the latter two methods do not make a significant improvement in the resulting signal. To avoid any redundancy and inefficiencies, I removed the spectral subtraction and adaptive noise filtering for now. Additionally, I looked more into how we can perform audio segmentation to make it easier to detect pitch and rhythm and found that we may be able to detect note onsets by examining , though this might not work for different volumes without some form of normalization. I will be working with Grace this week to combine our noise suppression and amplitude thresholding code, and more importantly, to work on implementing the note segmentation. Some of the risks with audio segmentation are as follows: noise (so we may need to go back and adjust noise suppression/filtering based on our segmentation results), detecting unintentional extra notes in the transition from one note to another (can be mitigated by setting a rule that consecutive notes must be, say, 100ms apart), and variations in volume (will be mitigated by Grace’s script for applying dynamic thresholding and normalizing the volume).  This week, we are also visiting with Professor Almarza from the School of Music to solicit flutists to test our transcription pipeline within the next few weeks.

We are currently on schedule, but we might need to build in extra time for the note segmentation, as detecting note onset and offset is one of the most challenging parts of the project.