Weekly Status Reports

This week,  I was able to connect a live pipe between the Scarlett audio interface and Aubio to generate a real-time stream of pitch values . From initial testing I was able to generate a mapping to understand the pitch value output. It follows that middle C (C4) maps to a value of 60, with every semi-tone increasing or decreasing the value by one point. I did notice some potential issues from the live audio testing.

1. Any elongated or plosive consonants (“ssss”, “ttt”, “pppp”) caused a spike into the 100+ range which is out of the bounds we are considering for expected pitch as it would put the sound in and above the 7th octave (above what a piano can play). Accounting for that is important but since it is so high, it might be easiest to simply ignore values outside the expected range.
2. Chords behave abnormally. When two notes or more are played simultaneously, the output value was either lower than any of the pitches played or simply read “0.0” which is the default no-input or error value output. I believe there is a specific way to handle chords but this requires further digging into the documentation.
3. Speaking seems to consistently generate an output of “0.0” which is good, however some quick transitions from speaking to singing yielded mixed results. Sometimes the pitch detection would work immediately and other times it took a second to kick in.
4. Lastly, the pitch value provided has a decimal that corresponds to the number of cents each pitch is off by relative to the fundamental pitch. Accounting for notes that are within +/- 0.5 from a just pitch will be important. Vibrato varies from person to person but for me, at least, it seemed to be within that tolerance which is a good thing.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am just about on schedule. This is an ongoing learning progress but being able to sing live or play live and have a pitch value output is promising.

What deliverables do you hope to complete in the next week?

I hope to refine and create a more robust output that takes the average pitch over a short duration to create a pitch event list that is more parseable for the timing algorithm.

During the week my primary focus was creating a connection between the Scarlett USB device (one which I own) and linking it into the skeleton of the pitch detection script from earlier. This was in preparation for a similar device we are purchasing. I was able to successfully create a program using the pyaudio library, and the pysudev library that detects when a Scarlett audio interface is connected and finds a path that can be used in the audio processing script with Aubio. The current difficulty is integrating these two programs as the device detection is buggy at times and requires consistent polling to function. I also need to add edge cases for if the Scarlett device becomes disconnected during operation.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

In terms of schedule, everything seems to be on track. The majority of the tasks ahead are things I have to learn so my concept of how long they will take is purely an estimate. With that being said, my hope is that a MVP could be reached in about 2 weeks. This would look like a working note detection program which outputs an event list which will be used by the timing algorithm. I imagine most of the time over the next few weeks will be spent debugging and optimizing the systems.

What deliverables do you hope to complete in the next week?

In terms of deliverables for the end of the week I’d like to have these successfully integrated and to a point where I can begin testing the pitch and note onset detection libraries. This would keep me on the expected schedule.

This week, my focus was attempting to set up a connection between my existing audio interface (Scarlett Solo) and a python script that could process the input. I found three Python libraries to support this.

Firstly, PyAudio which is one possible route for input/output for our system. It can read existing sound files (like .wav) which is useful since we are starting by piping in recordings instead of live audio at first. It also has basic streaming capabilities for a possible shift to live audio input.

A second option I researched for  I/O is SoundDevice. Similar to PyAudio it can parse an existing music file and/or stream in real-time audio. It works in conjunction with numpy and has existing starting templates for various projects found here. 

Aubio is a library with pitch onset, duration, and separation routines; Audio filtering, FFT, and even beat detection. All of these features are essential components to our sound processing workflow.

Using templates from these libraries I made a basic, untested python script as a starting point.

In terms of scheduling, I am slightly behind as I expected to be working on the filtering component starting this week. I have realized that the event list generation can and should happen first as it goes hand in hand with the audio path creation. As such the scheduling has been changed to start that this upcoming week with filtering being pushed back until week ten as ensuring a working pipe for MVP needs higher priority.

By the end of next week I hope to have a python script that can take in an audio file and break it down into a tuple structure that contains (pitch, onset time, and a guess at duration). For my purposes I do not expect the generated list to be accurate, instead focussing on producing one to be tested in the first place. I’d also like to have a partial method of reading in input from a Scarlett audio interface. That is to say, a script that successfully transfers some data from the microphone into a buffer I can parse. I imagine that the live connection will take more time and may need to be shared over the next week as well, the schedule has been updated to reflect this (See group status update, Conversion Algorithm subsection).