Category: Anja

The pitch contours are looking fairly good. I cleaned them further by trying to remove spurious troughs in the contour. I did this by taking max-series and then removing spurious peaks.

For improvement, I have been fine tuning the analysis. I have considered different options instead of just DTW:

For every alignment that looks really good, the autocorrelation would be completely symmetric, since the stretched function would look just like the one matched against it. So if I could instead get a measure of symmetry, or rather skewness, of the autocorrelation, this would show that for whatever amount of stretching needed (even if it were a lot), if it were very closely aligned then that would be the optimal match.

Besides the analysis, I have considered the complexity as well. Currently it matches within about 3 minutes for a 5 songs database, so this will need to be sped up completely. I am going to play around with the hop size for the DTW windowing. I have already bucketized the database contours and the query contours by about 0.1%, which seems to be the highest that still preserves information.

pictured above are some alignment plots. The blue is the sung query, and the contour of a song in the database at a segment in the song range that seemed to be the closest to what was sung.

Overall, I’m still going to be bumping up accuracy and complexity until the demo.

This week I worked on improving pitch detection. I am going with Power Spectral Density function and peak detection within that. I have cleaned the signal up a little more, since I noticed many issues with spurious peaks and also the recording picking up harmonics.

I have passed the signal through a human vocal range band pass filter.

I have also thresholded based on amplitudes.

I had a meeting with Professor Stern’s team to talk about potentially improving it further. Their team has an approach that uses autocorrelation functions and then a histogram of peak differences to pick the best peaks. He suggested modifying which peaks I select.

So I still need to smooth out the signal, this job has proven more difficult than imagined. I will attempt to do more with better peak selection from the PSD plot.

All I have left to do is to improve the signal smoothing like such and to collect more statistics on the heuristics. We will make our final presentation very soon and also the poster for the poster session. Finally, I will place my scripts in my public AFS space so that our website can interface with it and query it.

I seem to be in a tight spot, but am hopeful things will be on time.

This week I processed the MIDI library by hand. The general procedure included opening the MIDI into Musescore and then converting the melodies into a dry piano tone. Then exporting the clip as a wave.

When I started processing the MIDIs, I noticed some flaws in my processing for recorded audio too, since comparing recorded audio to preprocessed audio was starkly different.

My algorithm for comparing the series remained them same, that is the DTW. However, I realized I had to come up with a better pitch detection algorithm. I went with one based on Power Spectral Density Functions with a Hamming window of about 80ms.

So far this is ok, I still have a lot of work left to do with cleaning up the signal.

This set me back a little because the pitch detection is rather complex.

I will continue cleaning the signal and trying to extract the melody as best as possible from the recorded objects in the following week.

This week I was actually unable to have the meeting with Zheng. Instead, however, I have devised the following plan to create the library:

I will look up MIDIs for all of the songs we want in our library with this link:

http://www.midiworld.com/files/

From this website I can get MuseScore arrangements of all the pieces, from which I can specifically select the melodies of interest.

Musescore allows me to export mp3s of pure pitch tones of melodies of interest, which I will do for all the tracks/staffs within the songs we intend to put in our library.

Finally, I will process the .mp3s or .wavs in MATLAB the same way I process vocal input (record vocal input), so that the list of tones is the same data format as that used for recorded vocal queries.

From here, the matching should be same.

I think I am a little behind, I hope to be able to create a substantial library by the middle of next week.

This week I performed a demo of the first version of the dtw matching algorithm. I then refined the algorithm even past the demo. I tried out a few more test cases that I hadn’t considered last week, including

prioritizing pattern over pitch (transpose invariant)

prioritizing majority pattern over accurate pattern.

My very next task is to do MIDI processing so that we can begin library synthesis. I will work with getting these in a MATLAB format, I have already contacted Professor Dannenberg’s Master’s student who does melody contouring into MIDIs by hand and with some automated tools. We have a meeting scheduled for Monday to explore the options and figure out how it works.

My goal is to finish this part and then immediately get things in the right format so my algorithm can start matching against the library. I’m sure that’s when interesting bugs will come up, and I’ll save further refinement for the week after!

I might be like half a week behind schedule, since finding a time to metope for MIDI processing took a bit longer than hoped. But hopefully I can make up for the future debugging with the exploration of other test cases I did in the waiting period.

Team C6

Anja Kalaba

This week I looked through a lot of available libraries for audio microphone processing in C, most notable is PortAudio. Additionally, MATLAB was found to have a stable processing scheme.

I was able to obtain input and represent user input in the desired format. Still have to look into if the method currently is compatible for the app our team desires to make. Next, I’ll need to make sure that MIDI files can be placed in this format too, and actually do the time warping this final created data format. Which is an array of pitches based on the sampled input that gets bucketed for each time stamp.

I would say I am on schedule.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I finished the finalized our design concepts, created the design slides and have been working on the design review. I had to review presenting and practice the slides. I was also in communication with Professor Dannenberg and have settled we cannot contact the team members of Theme Extractor but will be in contact with his grad students to borrow melody extraction from MIDI files. We also finalized a division of labor, so next week I can start on the input discretization and type warping algorithm.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Definitely on schedule as long as implementation starts next week!

• What deliverables do you hope to complete in the next week?

A database of theme extracted songs, probably about 3 songs to start. Some humming samples from all 3 team members. The ability to discretize vocal input. A sketch in C for how to begin the time warping algorithm on the data.

This week I began working on my arm of the project, the chroma feature similarity matrix analysis. Since the first step is building chroma features (also known as chromagrams), I’ve started looking into available toolboxes/code for creating these. Most of the existing work seems to be in MATLAB, so if I want to use an existing chromogram library I’ll have to decide between working in matlab and compiling to c++ or simply drawing inspiration from the libraries and building my own implementation.  Even within chroma feature extraction, there are lots of design parameters to consider. There will be a choice between how the chroma vector is constructed (a series of filters with different cutoffs, or fourier analysis and binning are both viable options). On top of this, Pre and post-processing can dramatically alter the features of a chroma vector. The feature rate is also a relevant consideration: how many times per second do we want to record a chromagram?

Some relevant pre-and post-processing tricks to consider:

accounting for different tunings. The toolbox tries several offsets of <1 semitones and picks whichever one is ‘most suitable’. If we simply use the same bins for all recordings we may not need to worry about this? but also, a variation of this could be used to provide some key-invariance.

Normalization to remove dynamics–dynamics might actually be useful in identifying a song. We should probably test with and without this processing variant.

“flattening” the vectors using logarithmic features–this accounts for the fact that sound intensity is experienced logarithmically, and changes the relative intensity of notes in a given sample.

logarithmic compression and a discrete cosine transform to discard timbre information and attempt to get only the pitch info

Windowing different samples together and downsampling to smooth out the chroma feature in the time dimension–this could help obscure some local tempo variations, but its unclear right now if that’s something we want for this project. This does offer a way to change the tempo of a chroma feature, so we may want to use this if we try to build in tempo-invariance.

As it turns out, these researchers have done some work in audio matching (essentially what we’re doing) using chroma feature, and suggest some settings for their chroma toolbox that should lead to better performance, so that’s a great place for us to start.

an important paper from this week:

https://www.audiolabs-erlangen.de/content/05-fau/professor/00-mueller/03-publications/2011_MuellerEwert_ChromaToolbox_ISMIR.pdf

http://resources.mpi-inf.mpg.de/MIR/chromatoolbox/

http://resources.mpi-inf.mpg.de/MIR/chromatoolbox/2005_MuellerKurthClausen_AudioMatching_ISMIR.pdf