This week I focused on making the web app more intuitive. Shivi had worked on the metronome aspect of our web app which allows the tempo to change as you change the metronome value using Websockets. I integrated her code into the web app and took the metronome value into the backend so that the bpm changes dynamically based on the user. I also tried to get the editing feature of Flat IO but it seems that the free version using iframe doesn’t work. We are thinking of looking into the Premium version so that we can use the Javascript API. The next step is to work on this and add a Past Transcriptions page.
Deeya’s Status Report for 03/29/25
This week I was able to show sheet music on our web app based on the user’s recording or audio file input. Once the user clicks on Generate Music, the web app is able to call the main.py file that integrates both Shivi and Grace’s latest rhythm and pitch detection algorithms, generate a midi file and store it, convert it into MusicXML, make an API POST request to Flat.io to then display the generated sheet music. The integration process is pretty seamless now so whenever there are more changes made in the algorithms it is easy to integrate the newest code with the web app and have it functioning properly.
In the API part once I convert the MIDI to MusicXML, I use the API method to create a new music score in the current User account. I send a JSON of the title of the composition, privacy (public/private), and data which is the MusicXML file:
new_score = flat_api.ScoreCreation(
title=’New Song’,
privacy=’public’,
data=musicxml_string
)
This then creates a score and an associated score_id, which enables the Flat.io embedding to place the generated sheet music into our web app:
var embed = new Flat.Embed(container, {
score: scoreId,
embedParams: {
mode: ‘edit’,
appId: #,
branding: false,
controlsPosition: ‘top’.
}
});
Flat.io has a feature that allows the user to make changes to the generated sheet music including the key and time signatures, notes, and any articulations and notations. This is what I will be working on next, which then should leave good amount of time for fine-tuning and testing our project with the SOM students.
Team Status Report for 03/22/2025
This week we came up with important tasks for each of us to complete to make sure a working prototype for the interim demo. For the demo we are hoping that the user would be able to upload a recording, the key, and the BPM. The web app should then trigger the transcription pipeline and then the user can view the score. We are aiming to have this all finished for a rhythmically very simple music piece.
These are the assigned tasks we each worked on this week:
Grace: Amplify Audio for Note segmentation, Review note segmentation after amplification, Rhythm Detection
Deeya: Get the Metronome working, Trigger backend from the webapp
Shivi: Learn how to write to MIDI file, Look into using MuseScore API + allowing user to edit generated music score and key signature
After the demo we are hoping to have the user be able to edit the score, refine the code to better transcribe more rhythmically complex audio, do lots of testing on a variety of audio, and potentially add tempo detection.
Also, all of our parts (microphone, xlr cable, and audio interface) have arrived and this week we will try to get some recordings with our mic from the flutists from SOM.
This week we completed the ethics assignment, which made us think a little bit more about plagiarism and user responsibilities when using our web app. We came to the conclusion that we might need to include a disclaimer for the user that they need to be careful and pay attention to where they are recording their piece so that someone else can’t do the same. Also, we have decided not to work on our stretch goal into a project anymore after reflecting on our conversation with Professor Chang. We instead will be focusing on making sure the web app interface is as easy as possible to use and that the user can customize and edit the sheet music our web app generates. Overall we are on track with our schedule.
Deeya Status Report for 3/22/2025
This week I focused on getting the metronome working to play at various speeds the user puts in. Also we don’t need to focus on the pitch of the metronome or making sure that it gets recorded by the microphone because the detection algorithms are using the user inputted BPM instead. Also I focused on getting the backend integrated with the web app. I was able to get the python file Shivi and Grace having been working on to start running when the user clicks on the Generate Music button on the web app. I first started off with a hardcoded recording inputted into the file, and then was able to use the uploaded recording as the input to the python file to do the detection algorithm on. Overall my progress with the web app is on schedule.
Deeya Status Report for 3/15/25
This week I finished up the UI on our website for being able to record the flute and background separately, and I added the feature of being able to hear a playback of the audio being recorded. If a user decides to upload a recording, the ‘Start Recording’ button gets disabled, and if a user presses the ‘Start Recording’ button then the ‘Upload’ button gets disabled. Once the user starts recording they can stop the recording and then replay it or redo their recording. Also the user is able to adjust and hear the tempo of the metronome that gets played at a default of 60bpm. There are still two modifications I need to figure out 1. In the recording, the metronome can’t be heard because the recording only catches sound coming externally from the computer so I am playing around with some APIs I found that are able to catch internal and external recordings from a computer 2. I need to adjust the pitch of the metronome to a value that isn’t in the range a flute can be played at. For the Gen AI part there is a Music Transformer implementation available online using the MAESTRO dataset that focuses on piano music. I am thinking of using this instead of creating this process from scratch. I downloaded the code and tried to understand the different parts of the code. I was able to take a flute midi file and convert it into a format that the transformer can use. I want to continue learning and experimenting with this and see if I can fine tune the model on flute midi files.
Deeya’s Status Report for 3/8/25
I mainly focused on my parts for the design review document and editing it with Grace and Shivi. Shivi and I also had the opportunity to speak to flutists in Professor Almarza’s class about our project, and we were able to recruit a few of them to help us with recording samples and providing feedback throughout our project. It was a cool experience to hear about their thoughts as well as understand how this could be helpful for them during practice sessions they have. Specifically for my parts of the project I continued working on the website and learned how to record audio and store it in our database to be used later. I will now be starting to put more of efforts in the Gen AI part. I am thinking of utilizing a Transformer-based generative model trained on MIDI sequences and I will need to learn how to take MIDI files and convert them into a series of token encodings of musical notes, timing, and dynamics, so that it can be processed by the Transformer model. I will also start compiling a dataset of flute MIDI files.
Team Status Report for 2/22/25
This past week we presented our Design Review slides in class and individually spent time working on our respective parts of this project. Deeya is almost done completing the basic functionality of the website and has so far completed the UI of the website, user profiles and authentication, and navigation of different pages. Grace and Shivi are planning to meet up to combine their preprocessing code and figure out the best way to handle audio segmentation. They want to make sure their approach is efficient and works well with their overall pipeline that they are each creating on their own.
This week we will focus on completing our design report, with each of us working on assigned sections independently before integrating everything into a cohesive report. We are planning to finish the report a few days before the deadline so that we can send it to Ankit and get his feedback. This Wednesday we will also be meeting with Professor Almarza from the School of Music and his flutists to explain our project and to come up with a plan on how we would like to integrate their expertise and knowledge into our project.
Overall we each feel that we are on track with our respective parts of the project and we are excited to meet with the flutists this week. We haven’t changed much to our overall design plan and there aren’t any other new risks we are considering besides the ones we laid out in our summary report last week.
Deeya’s Status Report for 02/22/2025
This week I finished setting up the user authentication process for our website so that each user will have an associated profile to their account. This will help keep track of what transcriptions belong to which user and which transcription to upload in their respective Past Transcriptions page. I also started looking into how to record live audio through the website and store that in our database so that it can be used by the pitch and rhythm algorithm being designed by Grace and Shivi. Overall I am on track with the website and should be done with its overall functionality this week. One thing I still want to figure out is how to take what is most recently stored in our database of either the uploaded or live recorded audio files and automatically put that through the pitch and rhythm algorithms so that when it is time to integrate the process should be smooth. For the Gen AI portion of the project it looks like I might need to create a labelled dataset myself which I will have time to focus on once I finish up the website this week. Also for this week I will be working on my portions of the design review report.
Deeya’s Status Report for 2/15/25
This week, I made progress on our project’s website by setting up a Django application that closely follows the UI design from our mockup using HTML and CSS. I am finishing up implementing the user authentication process using OAuth, which will allow users to easily register and log in with their email addresses. User profile information is being stored in an SQL database. I am currently on track with the website development timeline and will be focusing next on being able to upload files and storing them in our database. Also I will begin working on the “Past Transcriptions” web page, which will show the user’s transcription history along with the dates each transcription was created.
Regarding the generative AI component of the project, I am still searching for a large enough labeled dataset for training our model. I found the MAESTRO dataset of piano MIDI files, which would be ideal if a similar dataset existed for the flute. If I am unable to find a large labeled dataset within the next few days, I am planning on creating a small dataset myself as a starting point. This will allow me to start experimenting with model training and fine-tuning while continuing to look for a better dataset.
Deeya’s Status Report for 02/08/25
- I am tasked with working on the web/mobile application part of our project as well as with implementing the Gen AI aspect of our project
- We first were trying to assess whether a web application or a mobile app would work better for our project and its use cases. We decided to use a web app instead because it is easier to access, upload and store files, and authenticate users, and we overall have more experience working with Python, Javascript, HTML, CSS than with Swift for iOS apps.
- I designed a very basic UI for the website and will be starting a Django project that has the UI and basic functionality like being able to upload/save files to a database and has user profiles to allow users to login and out.
- For the Gen AI component the first step is to find a large enough dataset of flute music of different genres. I spoke with Professor Dueck to ask her if there was CMU archival of flute music or any resources that she recommends to look through. She recommended looking at classicalarchives.com and specifically for solo or duet flute sonatas or anything unaccompanied. Looking through this website there are a lot of flute compositions that can be useful for this project. However I still need to figure out what would be the best way to compile together a large dataset and categorize/label each piece based on its genre, tone, pace. This will be a time consuming process so I will still continue researching for more flute labelled datasets.