Posted on

Aditya’s Status Report 4/8

This week I started working out how to format information within the note data structures so that it’s always readable by Vexflow. I wrote functions in the back-end that went through the output of the integrator of the pitch and rhythm processors and modified the duration of each element, so that it would always be a multiple of 4. This is because our STFT was designed such that 4 windows would correspond to the length of an eight-note in sheet music.

I then needed to make sure each duration is a power of 2. That’s because each subsequent note type (eighth, quarter, half, etc…) is 2 times the duration of the previous type. I iterated through the data and if any value was not a power of 2, I would split it into smaller notes. For example, if a notes was 24 units long, it would be split into notes of length 16 and 8. Then a Tie was added, which in music notation indicates that the durations of two notes are combined.

Here is a demonstration of ties in the music:

As you can see, the first measure has been properly split so that the duration of the measure doesn’t exceed 4 beats (or 32 units in the back end). One bug is the tie connecting the notes at the end of Row 1 and the start of Row 2. This week I plan to fix this by implementing dotted notes, which are another way we handle non-power of 2 note sizes that don’t result in this graphical error.

Testing

To check the results of the formatting algorithms, I manually generated several small note lists that were meant to represent the output of the integrator, then ran it through the formatting methods. For example, a list containing only a note of duration 12 would output a new list of length 2, which notes of duration 8 and 4.

In the future, I plan to implement tests that test data of varying tempos and time signatures, such as 3/4 and 6/8. I want the testing code to be robust and able to handle all these time signatures the same way, so that the same function can be used to generate notes from 6/8 as in 3/4, with the parameters of the function being changed.

Posted on

Alejandro’s Status Report for 4/1

This week I implemented the SNR-rejection system of our project. It will display a red alert in the front-end of the app if the audio is rejected due to it having too much noise. If the SNR-rejection system requirements are satisfied, the audio will be processed as usual (see here).

I also focused on testing our systems. I first created a bunch of piano files in Garageband containing different audios of monophonic piano files. The audios at first were pretty simple and short for us to be able to use these and see how well our systems work initially. Then, I made a couple of longer and more complex audios to be able to test our systems with something more realistic. 

 

I run these files on our systems and they seem to work fine on our rhythm and pitch processors but not in the integrator. At first one issue was the integrator was outputting all rests. I realized the issue was that we changed code to normalize the signal before putting it through the rhythm process and therefore I had to modify a parameter that sets the height required of a signal for something to be a peak to a much lower value since the signal output normalized ranges only from 0 to 1. 

 

After fixing that running the integrator is not being too accurate as of right now. It seems like some notes detections are innacurate in terms of pitch. We also discussed with Tom that the way we are currently integrating the systems might not be the best so we might have to re-implement the integration system. 

 

This week I will be focusing on the testing part and try to find what issues are going on with our integrator as well as possible fixes. I think we are on track.

 

Posted on

Kumar Status Report 4/1

This week I also worked on the Vexflox API and node.js integration in the webapp. This was needed so that we can generate a pdf of the output music score. It required installing and backend integration setup with Django. This was my primary focus other than helping Aditya with the Stave Note functionality. There was significant debugging that hasn’t been resolved yet but should be this week. The code for the pdf generation was in the Vexflox Tutorial but required some changes and modifications that I worked on. 

 

Next week I will complete this debugging and work further in assisting in the rhythm processor integration – I would say I’m slightly behind schedule but once this bug is resolved it should be back on track.

Posted on

Aditya’s Status Report for 4/1

This week I worked on using the Vexflow API to transcribe the output of the pitch and rhythm sub-processors. A lot of this involved translating the data from our own Note design structure to fit into the StaveNote class provided by Vexflow. I also had to determine a robust method of dividing the processor outputs into smaller sets of data, because Vexflow is implemented in a stave-by-stave method, meaning you only draw 4 beats at a time before rendering that section of the sheet music. There’s a lot of in-between calculation here as I need to determine whether or not the 4 beats have been completed within a number of notes ranging from 1 to 8. Getting just one calculation error means the whole chart will be off.

By next week I hope to have the rhythm processor fully integrated, as I’m falling behind on that front due to the learning curve of the Vexflow API. I’ve figured out the library now, so things should be smoother from here. I also hope to be able to have a proper song, such as “twinkle twinkle little star” as input instead of brief sequences of notes.

Posted on

Team Status report for 4/1

Design Changes:

We discovered some needed changes to our design in the process of implementing it. We noticed that even signals with little dead air before the music begins can have a lot of needless rests at the beginning of the transcription. To account for this, we truncated the audio signals to remove any silent audio from before the user starts playing their instrument. We also added a feature that gives the user an option of selecting the tempo at which the audio is played, as we found that attempting to automatically detect the tempo was incredibly complex and unreliable. However, to keep our target users’ limited means in mind, we keep this feature optional because many will not have access to a metronome to ensure they stay on tempo.

Risks:

The largest risks currently posed to our project is the error present in our calculations of the duration of each note and it’s placement within the sheet music. We find ourselves having to modify the note durations calculated by the rhythm processor in order to have the data fit into Vexflow’s API. This leaves a lot of room for error in the output; for example, 3 eighth-notes could be transcribed as 3 quarter-notes due to compounding change in each note’s length in the process of sending information from back-end to front-end.

Another problem is that the transcription of a very short piece tends to result in a very long output, resulting in a file that people won’t be able to read conveniently as you can’t scroll down a computer screen while playing an instrument.

 

Our current status is that we are able to display the pitches of each note with very high accuracy, and we are able to accurately detect and transcribe rests, but the rhythm of each note is currently treated as every note being an eighth note.

(Image of transcription being displayed)

 

Posted on

Kumar Status Report 3/25

While Alejandro is familiar with Bootstrap, he focused on converting the previously written CSS to this as it helps  modularize our project better.

I continued working on the project, specifically the cruz of a major bug. The bug was that if the length of the notes array was greater than 0, then we weren’t able to draw the notes using the Vexflow library. We continued to encounter an uncaught syntax error in our JavaScript file. After digging into various forums and documentation for the JavaScript library Vexflow,  we ended up with the resolution to this – implement a try and except case and sort of hard code it in. I then tested it with various files and arrays and ensured it doesn’t fail.

 

As this tooka few days to sort out, I was mainly assisting Alejandro and Aditya on higher level discussions surrounding the integration files we had to alter, as outlined in the team status report. I also took the lead on the ethics assignment for our group.

I’d say I’m on track with the front-end nearly finalized and contributing mostly on debugging and code structure discussions on the backend.

 

I’ll be taking a lead on SNR rejection and adding more front end functionality such as saving and downloading files next week and perhaps creating user profiles.

Posted on

Aditya’s Status report for 3/25

My schedule changed this week as after we built the Note design structures, we realized we would have to implement the integrator step of the project in the front-end instead of the back-end, as discussed in our team status report. Because of this, I couldn’t use the output of the integrator in my Vexflow code like I’d originally planned.

Instead, I worked on determining how long each transcription will need to be based on the number of notes in each audio. Vexflow’s library is setup so that you have to manually instantiate each measure of the musical piece before adding notes to it. So, I wrote code that took the length of the input audio and using the baseline time signature of 4/4 determines how many measures (defined as Staves in the Vexflow API) would be required in the transcription.

Once the number of Staves was known, I could setup a 2-d array of locations, to track which measure goes where on the output PDF. I chose that there would always be 4 Staves in each row, so the 2-d array was an N-by-4 matrix where 4*N is the total number of Staves. If the number of Staves isn’t divisible 4, there are “padding” Staves added to the ending so the output still looks neat.

Once the Staves are instantiated, I iterate through the piece and write each note (only the pitches currently, as the integrator is incomplete); the row and column of the desired Stave is determined  based on the index of the Note in the list recieved from the backend; for example, the 5th item in the list will correspond to the 2nd row, 1st Stave of the transcription.

Posted on

Alejandro’s Status Report for 3/25

First, I had to work on the ethics assignment with my team.

After that, I decided to make the front end of the website look even better. I made sure we were only using Bootstrap and barely any CSS so that resizing the window would not affect the look of the website. I also had to made sure to add code so that our backend is able to read in the values set by the user in the form when selecting a clef, time signature and audio file. Before, our code was not able to read in these values correctly and now this should be fixed. This information will be sent to the rhythm and pitch processors. I also added the copyright footer to our website.

 

Finally, I had to write code in the views.py file that allows the backend to send all the correct information to the front end so that we can utilize it with VexFlow to display the music sheet. This required me to make some changes to the integrator like we talked about in the team weekly status report. I changed the integrator from being in python to happening in the javascript part of our code since there was a bug where apparently sending a list containing a class from python to javascript would not work. Therefore, now we just send to the javascript the pitches and the rhythm output and call our integration function in javascript.

I also made sure that Vexflow is able to display correctly in the front-end the clef and the time signature. 

Finally, I made sure that the integration system was working properly. It seems that it is now able to produce an output of notes, which means we should be able to test it next.

I would say my progress is on schedule.

 

Next week we will be focusing on testing the integration system as well as the other systems. We should also get started on the SNR rejection system if time allows. 

 

Posted on

Team Weekly Status Report for 3/25

Risks

It seems that when entering an audio file to our system it takes a little long to transcribe with short audios. For example, it takes 24 seconds to transcribe an audio containing 4 notes that lasts around 14 seconds. Therefore, this could be a bigger issue with even longer audios. 

 

Design Changes

Our design has two sub-processors, one for determining the pitches of each note in the audio and one for determining the rhythm of each note, followed by an integration engine. The sub-processors are implemented in Python, and we initially planned to implement the integrator in Python as well, generating a list of Note Structs and sending them to the front-end to be transcribed. However, we found out that sending information packaged within a design structure meant that the front-end could not effectively parse through the information within the structures. We realized we would have to send to the front-end information that contained primitive data types such as Strings or ints. So, we decided to integrate the pitch and rhythm processors’ outputs after the information was sent to the front end. This is because we can instead send to the front-end the output from the rhythm and pitch processor separately, since they are a list of integers and strings. The method of integration is unchanged, the primary difference is that the HTTP Response contains two outputs instead of one.

 

Progress

[Front-End Updated]

[Integrator Change]

Posted on

Kumar Status Report 3/18

My role this week was more of that of assisting my team-mates with their tasks. Our front-end app is coming along  nicely, so I transitioned to helping my team-mates with the backend work.

I planned out the HTTP response containing note information that we needed, so Aditya was able to use this while starting to work with Vexflow. I tried what Aditya already outlined, downloading the library with npm and manually, but this didn’t work for either of us, so we used a script tag.

I resolved a major bug on the front-end where we were having an issue with the MIME type of the Javascript, as the browser was unable to read the file being sent to it when trying to display the notes in Vexflow. This was an issue with the typescript and javascript integration, and was able to resolve it by simply moving the factory class in our javascript file to the bottom of the file.

I would say my progress is on schedule. I am going to be helping Alejandro with the rhythm and pitch processor integration next week, and taking a lead again on the CSS and look of the web app interface.