Posted on

Anja Status Report #10

This week I worked on improving pitch detection. I am going with Power Spectral Density function and peak detection within that. I have cleaned the signal up a little more, since I noticed many issues with spurious peaks and also the recording picking up harmonics.

I have passed the signal through a human vocal range band pass filter.

I have also thresholded based on amplitudes.

I had a meeting with Professor Stern’s team to talk about potentially improving it further. Their team has an approach that uses autocorrelation functions and then a histogram of peak differences to pick the best peaks. He suggested modifying which peaks I select.

So I still need to smooth out the signal, this job has proven more difficult than imagined. I will attempt to do more with better peak selection from the PSD plot.

All I have left to do is to improve the signal smoothing like such and to collect more statistics on the heuristics. We will make our final presentation very soon and also the poster for the poster session. Finally, I will place my scripts in my public AFS space so that our website can interface with it and query it.

I seem to be in a tight spot, but am hopeful things will be on time.

Posted on

Nolan: Status Report 10

I’ve got the pre-processing and creation of cross-similarity matrices down to a point where I feel happy with it. This week I created a set of .wav files of songs to train a model off of. I have 66 songs selected, so with two team members singing each song, there will be almost 9000 cross-similarity matrices to train off of: 132 matches and several thousand mismatches. This week I will be training the model.

Posted on

Team Status Report 10

We all made significant progress this week with the challenges mentioned last week and are pushing forward to the end. While we are still concerned that our system will be slow or not work very well, we are hopeful for decent results.

We are still lagging a bit behind but obviously the clock is ticking, so we are doing our best to reach the optimal state possible come public demo.

Posted on

Wenting: Status Report 10

This week I did more work on the app in both the visual design, adding pages, and connecting to the backend. My challenge this week was finding an audio recording library to use on the web app and integrating it with what I’ve already done. I also need work on how to trigger Matlab to run when the audio has been captured and then return the results back to the front end.

I am currently behind and have a lot to do before the final demo. I know the resources that I can use and reach out to for help though, so I am confident in my ability to get it done.

Posted on

Wenting: Status Report 9

This week I worked on putting together the web app and refining visualizations. I currently plan on having the following visualizations:

For a matched song

  • A “percent” representing how close the match was based on a weighted score from melody contour analysis and the similarity matrix
  • The sung input plotted on top of the original song
  • Histogram showing the distribution of distances (difference between original and input)

For a non-matched song

  • List of top match(es)
  • The same visuals as for the matched song but for each song

Possible additional features

  • Link to the song on Youtube/Spotify
  • A Yes/No button to determine whether Earworm was correct or not (good for feedback?)

As of the coming demo, I will be on track and just need to do further refinement based on feedback and integration.

Posted on

Team Status Report 9

This week we each ran into some obstacles with our parts. Pitch detection and finding patterns cross similarity matrices were challenges that we foresaw but are still formidable, and none of us have much experience with web development.

We are a bit behind schedule but are working hard to overcome it!

Posted on

Anja Status Report #9

This week I processed the MIDI library by hand. The general procedure included opening the MIDI into Musescore and then converting the melodies into a dry piano tone. Then exporting the clip as a wave.

When I started processing the MIDIs, I noticed some flaws in my processing for recorded audio too, since comparing recorded audio to preprocessed audio was starkly different.

My algorithm for comparing the series remained them same, that is the DTW. However, I realized I had to come up with a better pitch detection algorithm. I went with one based on Power Spectral Density Functions with a Hamming window of about 80ms.

So far this is ok, I still have a lot of work left to do with cleaning up the signal.

This set me back a little because the pitch detection is rather complex.

I will continue cleaning the signal and trying to extract the melody as best as possible from the recorded objects in the following week.

Posted on

Anja Status Report #8

This week I was actually unable to have the meeting with Zheng. Instead, however, I have devised the following plan to create the library:

I will look up MIDIs for all of the songs we want in our library with this link:

http://www.midiworld.com/files/

From this website I can get MuseScore arrangements of all the pieces, from which I can specifically select the melodies of interest.

Musescore allows me to export mp3s of pure pitch tones of melodies of interest, which I will do for all the tracks/staffs within the songs we intend to put in our library.

Finally, I will process the .mp3s or .wavs in MATLAB the same way I process vocal input (record vocal input), so that the list of tones is the same data format as that used for recorded vocal queries.

From here, the matching should be same.

 

I think I am a little behind, I hope to be able to create a substantial library by the middle of next week.

Posted on

Wenting Status Report 8 + Team Status

Wenting:

This week I started playing with the histogram idea to show the distribution of distances to the actual melody. I am debating whether I want to show absolute distance or both positive/negative as that may determine whether someone was singing flat or sharp, or simply which way they were off by.

Again, I am behind schedule but will need to actually have the website built by the end of next week since the clock is ticking till final demo, and I need to work on integrating the data visualization into the website and with other parts.

Team:

As predicted, all of us were very busy with carnival commitments this past week and were not able to do that much. However, we will be majorly kicking into gear this week for each of our parts.

Although we are a little behind, we are confident about finishing what we need to get done.

Posted on

Wenting: Status Report 7 + Team Status

Wenting:

This week I demo’d and received helpful feedback for the data visualization aspects. I was plotting the original song against the sung input (although both are still contrived examples and coloring the space between them with some color in the gradient from red to green, where red meant it was not a close match and green meant it was a close match. However, I neglected to include a scale which made it hard to interpret the graph. Since it is a gradient, the colors are not necessarily very comparable so it will be hard to discern particularly as the length of the sample grows and there are more colors in the gradient.

Suggestions from course staff included plotting the distances between notes and making a histogram to show the distribution of distances. Also, coloring the actual line of the melody as red/yellow/green is more indicative, and I think with that I could add toggle options to display just the sung input, the sung input and the melody it was matched to, or just the matched melody. This is an option that I have considered and am still tinkering with.

While I did most of my work for the demo in Python, I may port it over to Matlab for easier interfacing with the other parts. Also, I would like to start working on the pipeline from the two branches of audio analysis to data visualization, since that will be a key part of my portion.

I think I am a little behind schedule since I wanted to spin up a basic website by this week but that has not been done, and I would like to be further along in prototyping data visualization techniques.

Team:

Post interim demo, we are all working on pushing forward with the next steps on each of our parts. The suggestions and comments made by course staff were helpful for us to work off of for the next few weeks, and we feel confident that we will have enough time to finish what we have to do.

The risks that we are facing have not changed much, though there may be a big push at the end to put all the parts together into one system, which is to be expected. As this coming week is carnival and we all have commitments to carnival events, we will all be busy but will try to still be on track.

Some parts have not moved as quickly as originally estimated, but overall our team is in a good place schedule-wise – we know what we need to get done and have a good sense of time about it.