What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours). 

I spent the majority of my hours this week working on our design report. We spent monday going over the different sections of the report, penciling in an outline for each section, and assigning sections among our group members. I was given the following sections: Architecture and/or Principle of Operation, Design Trade Studies, System Implementation, Test Verification and Validation, and Bill of Materials. 

I started with the Architecture and/or Principle of Operation section, which required several detailed diagrams.

One such diagram was the web application diagram that took me a great deal of formatting before I was satisfied with it. I wanted to prioritize comprehension within this diagram and so I included captures of our UI mockups at each screen to better convey what the User experience would look like from start to end. Additionally, this inspired me to make additional screen mockups (login, signup) as well as revise previous screens (song in progress) which can all be found on our figma. After talking through what was feasible with my teammates, we landed on a song screen with horizontal bars to represent notes, a chunking lyric approach, and a moving arrow to represent the users pitch. Web latency is not trivial and so having a scrolling system was clearly infeasible at a .25 second latency. Therefore, we pivoted to an approach that loads up per each chunk of lyrics (i.e. In the song Happy Birthday “Happy Birthday to you” is one screen and “Happy Birthday Dear ____” is another) and in this way we only have to update the users pitch position on the screen via an arrow. 

Another diagram I spent my time on was the system diagram. I struggled a lot with this diagram being either too wordy, not including enough information, or having a hard to read navigation system. 

For Design Trade Studies, I ended up testing an additional 2 aubio pitch detection algorithms in order to ensure our choice of “yinfft”. These can be seen in the spreadsheet here under the ‘schmitt’ and ‘specacf’ tabs. 

For System Implementation, I made yet another diagram to show the pitch detection logic. 

Additionally, I spent a bit of time looking into exactly how the new addition of MediaStream will connect with our Pyaudio backend.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule? 

According to the Gantt chart, I should be finishing up the UI, testing the pitch detection on a demo song, starting to integrate the pitch detection with the web app, and finishing up the design report. Overall, I finished the UI as well as the design report. However, I’m not concerned about the other two tasks as we just recently added the MediaStream API into our system that will be communicating with the Pyaudio backend and we’ll need to ensure this communication works how we intend before moving forward with this integration. This week I plan to focus solely on this communication between MediaStream and Pyaudio in order to ensure this unforeseen change doesn’t set us back too far.

What deliverables do you hope to complete in the next week? 

As stated above, I hope to integrate the web app with the Pyaudio pitch detection backend for this coming week. 

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours). 

This week, I looked into midi files for all of our songs, started preliminary testing of pitch detection algorithms, and began thinking about the design report.

First and foremost, I scoured the web for midi files that would both provide a melody line for us to compare to as well as provide a proper instrumental for a user to sing to. This proved more difficult than I had anticipated as most of the midi files that I was able to find ran into one of the following issues:

1. Too complex of an instrumental 
2. Strange interpretation of the melody
3. No separation between the melody and the backing

As I am getting these midi files for free on the web, I suppose I should have expected the quality to not be up to my standards and therefore decided to cut my losses and revise the free tracks into what I wanted. Through this process, I was able to scrape together both backing tracks and vocal only tracks for the following:

1. Itsy Bitsy Spider
2. Happy Birthday
3. Twinkle Twinkle Little Star
4. I’m Yours 

Additionally, I made a fast and slow recording of the C major scale to test out our pitch tracking algorithms with. All of these mp3 files will be attached in this post.

The order is as follows: Twinkle Twinkle Little Star – Vocals, Twinkle Twinkle Little Star, Itsy Bitsy Spider – Vocals, Itsy Bitsy Spider, I’m Yours – Vocals, I’m Yours, Happy Birthday – Vocals, Happy Birthday, C Major Scale – Slow, C Major Scale – Fast.

The midi files “Twinkle Twinkle Little Star”, “Itsy Bitsy Spider”, and “Happy Birthday” were quite easy to convert as I only had to duplicate the recorded track and manually separate out the melody from the backing track. Attached below is an example of this process:

Here we have the track as it came with the melody attached to the rest of the track.

Here we have both the melody and the rest of the track separated out. This is more convenient as now I am able to mute the rest of the track and only listen to the melody, providing me with a vocal only track that I can use on the pitch detection algorithm for testing. This separation of melody and vocal only tracks is crucial for testing our pitch detection algorithm as it gets rid of any background noise and provides us with the most accurate signal we could hope to measure.

The file “I’m Yours” was a bit more challenging, as one might imagine, as it’s a much longer song and includes quite a few vocal runs, which are quite hard to mimic on an instrument while still providing a sharp signal. Therefore, this track took a lot of liberties with the melody, especially near the end where Jason Mraz is mostly ad-libbing and not necessarily following a strict melody. While I have separated out the vocals of this track, there are a few problems that I have not yet tackled. As these problems involve me manually going through the track and replacing the notes to better fit the actual melody, it will be quite an arduous task and I’m aiming to complete it within the next week. Another problem I have focuses around the fact that the melody was recorded on a flute rather than a piano, but this is just personal preference, so I’m willing to give up on this dream if I must. 

Finally, I hand crafted the C Major Scale audio tracks in a fast and slow version. With the slow version, I’m hoping to calibrate the pitch detection algorithm on slow notes to see how well it fares when it’s given a signal for an extended duration and expect the confidence to go up. With the fast version, I’m hoping to capture more of these quick note changes that happen in songs as well as make sure that the pitch detection algorithm doesn’t drop too low in confidence while the speed of note changes increases. 

Next, I started some preliminary pitch algorithm testing and quickly found a pitfall I wasn’t expecting. First, I recorded some dummy vocals on a garageband track, just to hear the quality of the audio as well as see how the computer was interpreting the signal and realized one key fact that I was not testing for: mouth breathing. As I grew up participating in choir as well as musical theatre, I have been trained to not breathe through my mouth at all costs and when I must, it better be silent. However, our webapp is marketed to beginners who more than likely have not gone through this same training and so upon confirming with Anita and Anna that they mouth breathe when they sing, I knew I had to test the audio peaking. As the headset microphone is right in front of the mouth and the recorded audio from it is also being played back through the headphones, I wanted to make sure that our user would not be scared by their own breaths as this could definitely mess up their user experience. Thus, I set out to sing a whole bunch until I was out of breath and audibly gasped for air to see just how loud I would be hearing myself back. At the volume I was singing, even with a gasping breath rather than a normal breath, I was not able to jumpscare myself, but I’ll have people who aren’t used to hearing themselves test it next.

Next, I proceeded to use my aubio pitch tracking code to test different pitch tracking algorithms under more or less the same conditions. I would wear the headset microphone and have the interface plugged into my computer, then I would do the following:

1. Play the “C Major Scale – Slow” on my computer (so the microphone doesn’t pick it up) and sing along with it 
2. Be quiet and hold my phone playing the “C Major Scale – Slow” next to my microphone 
3. Repeat 1 + 2 with “C Major Scale – Fast”

The algorithms I tested were “YIN”, “YINFFT”, “fcomb”, and “mcomb”. I pasted the results of these pitch tracking in a spreadsheet and compared them side by side. The spreadsheet is linked here. 

Generally, I think the “fcomb”, “mcomb”, and “YIN” algorithms have spatterings of unreliable values (i.e. a 1000 amidst 260) and thus are not fit for our project. The “YINFFT” algorithm reported the most stable numbers and detected pitches within the correct ranges as well (200-600 Hz, give or take). Therefore, I will be moving forward with the “YINFFT” algorithm. 

In terms of the design report, I’ve read the feedback presented to us by our peers as well as our fellow section D peers and started zoning in on what people have questions about in an effort to clarify these for our report.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule? 

According to the Gantt chart, this week I am meant to test out pitch tracking algorithms, work on the UI, and put some thought into the design report. I did all of these things except for UI. In between lectures this week, I plan to get several peoples’ opinions on our UI mockups and refine them from there based on their suggestions. 

What deliverables do you hope to complete in the next week? 

Next week I hope to have refined UI mockups, preliminary testing done in a noisy environment, report sections written, and a start on pitch tracking for a song.