Team D4: EGGceptional Vocals

This week, I learned more about the EGG and its setup and ground truth research methods with the rest of the term. I also continued to work on building my Tkinter skills and think about the frames needed for the desktop app.

School of Music Conversations

Before getting into technical stuff, I wanted to note some new thoughts from conversations with School of Music collaborators, particularly with regards to options for sheet music syncing and microphone recording. One thing that we hadn’t thought much about is the need of an accompanist to help our singers to stay on key, if we’re doing anything beyond basic warmups (where it would be relatively easy to play ascending/descending starting notes). Fortunately, there’s accompanists in the SoM who would be willing to help with this. I suppose one of the possible complications could be getting a clear microphone signal when multiple ins. However, a strong possible advantage of this setup is that it might actually help us with sheet music syncing– if the accompanist uses a MIDI keyboard, the direct MIDI input will be comparatively easy to match to the MusicXML file. This would be a helpful way of syncing to sheet music without forcing our vocalists to stay on a super strict tempo.

Speaking of the tempo issue, our vocalists expressed mixed feelings about singing to a metronome. On the one hand, both of them appreciate, to an extent, the opportunity to do more practice with a metronome. However, they also worry about having the room to properly and fully form notes as they sing, which (as I understand it) could be impeded by an overly rushed/strict tempo. I think we could definitely sacrifice some level of matching accuracy to give some leeway here. Also worth noting that both vocalists strongly preferred the idea of an audible metronome to a visual one. However, I’ve come to tentatively prefer the MIDI idea (if that ends up being possible) given that an accompanist will be necessary in any case, and adding an accompanist would make the metronome option more complicated besides.

That being said, we’ve decided that sheet music syncing is a stretch goal for now– even though, as a musician, I think this will be one of the most useful extra things we can do, and I’d definitely like to prioritize it if possible! However, it is secondary to simply recording and displaying the EGG/audio signals themselves, so I’m going to put a pin on this rabbithole until I get more of the basics done.

Tkinter Learning

I’d never used Tkinter before, so I started by following parts of this Real Python tutorial to get started, with particular focus on learning about frame widgets (which I plan to use for switching between windows in the application). 

Based on my experience with building web applications, and given the number of frames in the application, I decided to follow a Model-View-Controller (MVC) paradigm for the desktop app to keep things organized. After some trial-and-error, I worked through this tutorial by Nazmul Ahsan to learn how this can be done in Tkinter. 

Progress Status

I am still on schedule with my parts of the project according to the plan. That said, I haven’t really made progress in the code for the application itself, and I think I ought to have set apart more explicit time in the schedule for simply getting to know the technologies that I’m using for this project. 

Goals for Next Week

• Implement the basic MVC application with homepage
• Create GitHub repo
• Create baseline audio recording page functionality
• Create baseline audio playback page functionality (with room for adding a future EGG signal)
• Help Melina with creating wireframes for application

While we’re still waiting to actually acquire the EGG sensor, my goal is to get a head start on planning out the recording feature + sheet music tracking. Ideally, the user will be able to input sheet music and a recording of them singing, and when they analyze their feedback in the app, the two will sync up. While the sheet music element wasn’t initially something we deemed important, it became a priority due to how this would dramatically increase usability, allowing the user to visually interact with the app’s feedback rather than be forced to relisten to a potentially lengthy audio file in order to get context for the EGG signal they see. It’s also a significant way of differentiating our project from our main existing “competitor,” VoceVista, which doesn’t have this feature. Additionally, if for some reason the EGG thing falls through, this mechanism will still be useful if we end up measuring a different form of vocal health.

General Brainstorm

Here’s what I’m envisioning, roughly, for this particular view of the app. Keep in mind that we don’t yet know how the feedback is going to look, or how we want to display the EGG signal itself (is the waveform showing vocal fold contact actually useful? Maybe we just want to show the closed quotient over time?). But that’s not my problem right now. Also note that there ought to be an option for inputting and viewing an audio/EGG signal without a corresponding sheet music input, as that’s not always going to be possible or desirable.

The technical challenges here are far from trivial. Namely:

1. How do we parse sheet music (what should be sung)?
2. How do we parse the audio signal input (what’s actually being sung)?
3. How do we sync these things up?

Researching how, exactly, this synchronization could be achieved, the prospect of a complicated system using pitch recognition and onset detection is daunting (though, we do hope to incorporate pitch detection later on as one of our analytic metrics). Doing something like fully converting the input signal into a MIDI file is difficult to achieve accurately even with established software, so, basically a project in its own right. 

So here’s an idea: what if the whole synchronization is just based on tempo? If we know exactly how long each beat is, and can detect when the vocalist starts singing, we can match each measure to a specific timestamp in the audio recording without caring about the content of what’s actually being sung. This means that all the app needs to know is:

1. Tempo
   1. This should be adjustable by user
   2. Ideally, the user should be able to set which beats they want to hear the click on (eg, full measure vs every beat)
2. Time signature
   1. Could be inputted by user, or extracted from the score
3. Where measure divisions occur in the score 
   1. This should be detectable in a MusicXML formatted file (the common file format used by score editors, and probably what we’ll require as the format for the sheet music input)
   2. However, if this is the only thing that needs to be extracted from the score, reading just the measure divisions off of a PDF seems more doable than the amount of parsing originally assumed necessary (extracting notes, rhythms, etc)
      1. But let’s not overcomplicate!
4. When the vocalist starts singing
   1. Onset detection
   2. Or, the recording page could already include the sheet music, provide a countdown, and scroll through the music as it’s being sung along to the click track. That could be cool, and a similar mechanism to what we’d be using for playback.

Potential difficulties/downsides:

1. Irregularities in beats per measure
   1. Pickup notes that are less than the full measure
   2. Songs that change time signature partway through, repeats, DC Al Coda, etc
      1. We just won’t support these things, unless, maybe, time signature changes come up a lot
2. A strict tempo reduces the vocalist’s expressive toolbox 
   1. Little to rubato, no tempo changes
   2. Will potentially make vocalists self-conscious, not sing like they normally would
   3. How much of an issue is this? A question for our SoM friends
      1. Sacrifice some accuracy in the synchronization for the sake of a bit of leeway for vocalists? 
      2. If an individual really hates it, or it’s not feasible for a particular song, could just turn off sheet music sync for that specific recording. Not necessarily a deal breaker for the feature as a whole.
3. Click track is distracting
   1. If it’s in an earpiece, could make it harder to hear oneself
      1. Using something blinking on screen, instead of an audio track, is another option
      2. And/or, will singing opera make it hard to hear the click track?
   2. If it’s played on speaker, could be distracting on playback, plus make the resulting audio signal more difficult to parse for things like pitch
      1. This could be mitigated somewhat by using a good microphone?
      2. Again, the visual signal is an option?
4. Latency between producing click track and hearing it, particularly if using an earpiece

Despite these drawbacks, the overall simplicity of this system is a major upside, and even if this particular mechanism ends up insufficient in some way, I think it’ll provide a good foundation for our sheet music syncing. My main concerns are how this might limit vocalists expression, and how distracting a click track might be to the singer. These are things that I will want to bring up at next week’s meeting with School of Music folks.

Workflow

In the meantime, this will be my order of operations:

1. Finalize framework choice
   1. We’ve decided that we (almost certainly) will be building a desktop app with a Python backend (the complications of a webapp seem unnecessary for a project that only needs to run on a few devices in a lab setting, and Python has very accessible/expansive libraries for signal processing)
   2. The frontend, however, is still up in the air– this is going to be my main domain, so it needs to be something that I feel comfortable learning relatively quickly
2. Set up basic “Hello World”
   1. Figure out a basic page format
      1. Header
      2. Menu
3. Create fundamental music view
   1. Both the recording and playback pages will use the same foundational format
   2. Parse & display the MusicXML file input
      1. This could be very complicated. I have no idea.
   3. Separate music into lines, with room for modularity (to insert EGG signal data etc)
   4. Figure out where measure separations are
      1. Measures will likely need to all be equal widths for the sake of signal matching ultimately? Not sure if this is crucial yet
   5. Create cursor that moves over score in time, left to right, when given tempo
      1. On the record page, tempo will be given explicitly by the user. On the playback page, the tempo will be part of the recording’s metadata
      2. How the cursor aligns with the measures will depend on the MusicXML file format and what is possible. Could just figure out amount of space between measure separation lines + move cursor smoothly across, given the amount of time per measure
   6. Deal with scrolling top-to-bottom on page
4. Create record page
   1. Set tempo
   2. Record audio 
      1. While music scrolls by, or by itself
   3. Countdown
   4. Backing click track
5. Create playback page
   1. Audio sync with measures + playback
   2. Play/pause button
   3. Drag cursor

Picking Framework & Technologies

I considered several options for a frontend framework for the Python desktop app. I was initially interested in using Electron.js, as this would work well with my existing knowledge of HTML/CSS/JavaScript, and would allow for a high level of customizability. However, Electron.js has very high overhead, and would introduce the complications of basically having to run an entire web browser instance in order to run the app. I also considered using PyQt, but I haven’t had experience with this framework, and it seemed like its idiosyncrasies would mean a pretty steep learning curve. So, I’ve decided to proceed with using TKinter, the standard Python GUI– in my small experiments, it seems to so far be straightforward, and since I’m starting out with some of the more complicated views of my GUI, I think I’ll be able to tell fairly quickly whether or not it’ll be sufficient for my purposes.

Finally, I wanted to learn how to parse and display MuseScore files in a Python program. I’ve started by learning how to use music21, a package developed in collaboration with the MIT music department, that offers a wide range of functionalities for dealing with MusicXML files. I’ve only really had the time to get this library set up, so I have yet to actually do much with it directly. Till next week!