Posted on

Shivi’s Status Report for 2/15/25

This week, I worked on setting up our codebase/environment, running experiments with various noise suppression/filtering methods implementation details for our final design.

With regards to noise suppression, Grace and I recorded some clear and noisy flute audio. I then experimented with some filtering/noise reduction techniques as well as the open-source Demucs deep learning model by Facebook, which separates music tracks. 

In my experiments, I used a combination of Butterworth bandpass filtering, adaptive noise reduction, and spectral gating. The Butterworth bandpass filter was applied first to ensure that only frequencies within the frequency range of the flute (261.6-2093.0 Hz) were captured in the audio. Then, I used spectral gating, which first estimates the noise profile from a specific part of the audio and subtracts it from the audio signal.

denoised = magnitude – reduction_factor * noise_stft

Currently, my script estimates the first second of audio to be noise, but this is not entirely accurate/true for all cases. This is why we introduced a calibration step into our pipeline, so that we can get a more accurate estimate of the noise in a particular environment as well as ensure that the user is playing loud enough (as the signal can always be reduced later). 

Then, the signal undergoes adaptive noise reduction to account for unpredictable fluctuations in background noise. I also experimented with various parameters such as prop_decrease (value between 0-1 that determines the degree of noise suppression), finding that 0.5 produced the best result. Below is a graph comparing the original and denoised signal:

Though this noise suppression module did eliminate much of the noise in the original signal, you could still hear some muffled speaking in the background, though this didn’t seem to interfere much with the detection of harmonics in the D note that was being played. My experimental code for the noise suppression and Fast Fourier Transform for harmonic detection is linked in the following repo.

The second approach I tried was using Demucs, a deep learning model open-sourced by Facebook to perform music track separation. However, since it is used mainly to separate vocals and percussion, it did a great job at filtering out everything except the metronome noise, as opposed to keeping only the flute noise. 

Given these results, I think the best route is to experiment more with a calibration step that allows the pipeline to take in both a noise signal and a flute+noise signal to be able to perform spectral gating more effectively. My current progress is on schedule. Next week, my plan is to run more experiments with the calibration and work with Grace to figure out the best way to segment the audio before performing the rhythm/pitch detection.

Posted on

Team Status Report for 2/15/25

This week, we focused on working on our high-level design for our design presentation. After discussing with Ankit and Professor Sullivan, we placed an order for some hardware to begin working with: Behringer CB 100 Gooseneck Condenser Instrument Microphone, and an XLR to USB-C Adapter. This will allow us to improve our current experiments, as we will be able to obtain clearer audio recordings. Based on our discussions this past week, we also decided to move our entire implementation into software. Additionally, we determined that it would be best for us to provide users with a metronome (which will be a sound outside the frequency range of the flute so that it can be filtered out later) set to a default 60 BPM, which the user will be able to adjust in real-time using a slider on our web app. Previously, we had recorded single notes from the B flat major scale to experiment with harmonics, but this week we met up to also record some noisy signals to experiment with noise reduction and work on encoding information for single notes into a MIDI file and uploading it to Musescore to see if we could translate it into sheet music (see individual team member reports). After a lot of discussion, we also concluded that a real-time transcription is not relevant for our use case, since a user would only need to see the transcribed output once they are done playing. 

Our new pipeline will work as follows:

  1. User logs into their account.
  2. User calibration: Ensure the user is playing at some minimum threshold before they upload a recording.
  3. User is prompted to record and upload an audio file of their background noise.
  4. User is prompted to record flute audio. (Consideration: have a play/pause recording in case the user needs to pause in between the recording?) To do so, they turn on the metronome on the web app. Metronome is set to 60 BPM by default, but they can adjust it in real-time using a slider.
  5. The audio is saved in the website’s database, and the pitch/rhythm detection pipeline is triggered in the backend. 
    1. Noise suppression via Butterworth filter and adaptive noise filtering. 
    2. Audio segmentation: Spectral Flux (how much the spectrum changes over time) and Short-Time Energy (STE) (detect sudden amplitude increases) to determine onset of a note
    3. For each segment (we can parallelize this with threading so multiple segments can be processed at once):
      1. Use note length to determine its type (eighth, quarter, half, whole, etc)
      2. Use FFT to determine frequency/pitch and classify which note it is
  6. Encode the info from all the segments into a MIDI file
  7. MIDI file gets uploaded to the web database and MuseScore API converts MIDI into sheet music
  8. Newly generated file is stored along with that user’s previous transcriptions that they can view
  9. IF time remains: we can add an editing feature where the user can adjust transcribed notes and add additional notation like crescendos, etc.

The biggest risk/challenge as of now is verifying if the methods that we are planning to use for noise suppression, pitch/rhythm detection will work or not. For instance, based on this week’s experiments with noise suppression, we experimented with a variety of filters (see Shivi’s status report), but found that many times, the flute audio would also get suppressed or that the background noise would not be suppressed enough. We would like to run more experiments, and our contingency for this is a calibration step that gives us a noise sample that we can then subtract from the flute audio signal. Similarly, note onset detection will probably be quite challenging as well, because it may be difficult to determine the exact moment a note ends. This is why we are deciding to segment the audio as our initial processing step, and then “round” the duration of each segment to the nearest eighth of a beat based on the BPM. 

Despite these challenges, we are on-track with our schedule; over the next week, we plan to have an even more detailed design while simultaneously working on setting up our web app, experimenting more with signal calibration/noise suppression, and starting on audio segmentation.

Week-specific status report questions:

Part A (Shivi): Our flute transcription system enhances public health by supporting creative expression through music learning. With an attachable microphone and a software-only pipeline, it is affordable and safe to use. Our system also promotes welfare by lowering barriers to music production, as it can be made accessible online for musicians, students, and educators to use.

Part B (Grace): Write on Cue aims to make music transcription more accessible to diverse musical communities, including amateur musicians, educators, composers, and students from various cultural and social backgrounds. This benefits people who may not have the technical skills or resources to manually transcribe music and allows individuals to better engage with music across a variety of cultural contexts. For example, in communities where formal music education is less accessible, our project can provide a more equitable way for musicians to preserve and share traditional flute music, irrespective of whether they are classically trained. Additionally, socially, this allows musicians from different backgrounds to contribute their musical expressions and allows for easier preservation of musical heritage. 

Part C (Deeya): Traditional methods of music transcription are time-consuming and require specialized knowledge, creating a barrier for learners who want to review their performances or for educators who need to provide detailed feedback. By streamlining the transcription process, our project reduces the dependency on costly manual transcription services, which lowers the overall cost of producing sheet music. Also, we are designing our project on a web app, which maximizes accessibility and encourages a cheaper and more widespread music education.

Posted on

Shivi’s Status Report for 02/08/25

This week, I worked with Grace and Deeya to finish our proposal slides, where we included a very high-level workflow of our design. Completing the proposal slides gave us a better idea of the amount of work we need to do, and the three of us met up to generate some flute audio recordings. Since I am tasked with pitch detection, as an experiment, I wrote a basic Python script that performs a fast Fourier Transform on singular notes so that we could examine the frequencies associated with a few notes from the B flat major scale:

Here, we can see the fundamental frequencies/harmonics associated with each note, a property that we will leverage to determine which note is being played in the audio. After proposal presentations, we thought about some feedback from our TA (Ankit) and realized that we need to think more about software-hardware tradeoffs in our design. Initially, we were keen on having a hardware component in our project (having taken/taking 18341 and 18349 as well as seeing similar projects from the past doing this), but it seems that it may be cleaner/more efficient to simply perform certain tasks purely in software. For instance, our initial design included performing FFT using the microcontroller, but it will definitely be more efficient to perform it on a laptop CPU. These are some of my thoughts for a revised design (at least on the signal processing side) based on some independent research:

  • Signal Processing 
    • Use microphone to capture flute audio
      • Suggested mic: InvenSense ICS-43434, a MEMS microphone with digital output. Can be mounted close to the flute’s embouchure hole and does not require any sort of PCB soldering. We also have the option to 3D print a custom clip to attach it to the flute for optimal placement.
      • Send audio to microcontroller via I2S (Inter-IC sound interface)
      • Microcontroller converts PDM (Pulse Density Modulation) to PCM (Pulse Code Modulation). Some suggested microcontrollers with built-in PDM support: RPi RP2040, STM32 (more suited for high-end tasks and higher performance so might not be necessary)
    • In software, do pitch detection: 
      • Apply additional digital filtering to the PCM signal: noise suppression, bandpass filtering, adaptive filtering
      • Apply Fast Fourier Transform to detect flute frequencies, map frequencies to flute notes
      • Use moving average filter (ex: Kalman filter) to smooth out pitch detection
    • In software, do note length detection:
      • Use Peak Tracking in Frequency Domain (more computationally expensive than methods like time-domain envelope filtering and requires harmonic filtering to avoid detecting overtones, but less sensitive to volume variations and more accurate in noisy environments)
      • Detect note length: note is considered ongoing if the peak frequency remains stable. If the peak disappears or shifts significantly, the note has ended.
    • MIDI: store the note frequencies, durations in a MIDI format. Then, generate a MIDI Note On message when note starts (0x90 message), MIDI Note Off message when note ends (0x80). Use duration to check note type (eighth, quarter, half, whole note, etc)
    • Use MuseScore API to upload MIDI file and display sheet music on web app

For the coming week, we plan to flesh the design out more and work on our low-level design with other important details such as BPM detection, metronome, and integration with webapp. We also aim to make a list of any inventory/purchase items we will need.