Author: sbritton

Revisiting last week’s goals

• Use sample data to more clearly demonstrate the layered table rows in the playback view (for interim demo)
  • Done– there were some glitches here that I wasn’t expecting, but I basically just added a sine wave overlaid with the original audio signal. Tyler is working on syncing up our audio signal and EGG signal from our singer session on Thursday of this week, and once that’s complete I’ll simply use the average CQ data as the graph here instead.
• Enable cursor scroll
  • Done– oddly, the most challenging part of this was getting DearPyGui to make the separate cursors have proper their colors. This step is easier shown than talked about, so check out this video if you’d like to see the cursor scroll in action!
• Expand page menus and navigation
  • Partially done – This is sort of a vague task and I should have fleshed out more of what I was expecting it to encompass. I have created menus to link to the different instructional/playback pages, but since those pages aren’t completed yet, it’s not much of an accomplishment to note here.
• Hammer down specific parameters for data analytics views, and complete one view (eg, average CQ over time graph) using arbitrary sample data
  • Partially done – Melina and I worked on establishing explicitly what we want some of our data views to look like, particularly with allowing the user to view multiple CQ waveforms simultaneously and specifically how to display CQ per pitch over time.
  • Partly still in progress – I haven’t actually coded the sample view itself. It’s a bit of a chicken and egg scenario between gathering data and coding data views, but I was hesitant to dive in head first as I don’t actually know what form the pitch data will look like once Melina has processed it.

Repertoire Library Wireframe

Additionally, I worked on the actual flow of the application as the user navigates it, with some help from our School of Music friends, since we realized that some of our initial plans were a bit hand-wavey. Specifically, how do vocalists view their library of repertoire that they’ve recorded and navigate it easily, especially given that this is a key element of what sets our application apart? My solution was to sort the repertoire at the high level by the title of the piece, and then at a lower level by the date each instance was recorded. Perhaps this is another case where it’s easier to see what I mean than for me to try to describe it. This is the Repertoire Library view, where clicking on each recorded instance will open up the Playback View that we know and love:

Verification

Our verification testing for the frontend will be done manually. My main goal is to ensure that the playback page functions given various parameters (files that have varying lengths, files that are missing either the EGG signal or the audio signal) and that unexpected inputs are handled gracefully with the app’s error handling. Specifically:

• Input 5 songs of various lengths. Successfully play through, replay, and scroll through each song.
• Input 2 songs without EGG signal. Successfully play just the audio.
• Input 2 songs without audio signal. Successfully view the signal without playing audio.

Goals for next week

• Use real EGG output data on playback page
• Implement basic repertoire library page
• Complete one data analytics view

Revisiting last week’s goal’s:

• Add Visual Cursor
  • I did this by inserting an additional vertical line onto the graph that displays the audio waveform (see the red line in the image below), corresponding to where the time index is while playing. The cursor is built into the runtime loop so that it moves smoothly while the audio is playing, and it’s triggered by the play/pause button.
  • 
• Add Restart Button
  • I set up this button to return the timestamp of the given song to 0, and to pause it if it’s currently playing. It also resets the cursor position.
  •  
• Work on parsing VoceVista output file & CQ waveform
  • I got somewhat stymied by this, because we don’t actually have an example of a CQ waveform for a full piece. Ideally, we’ll be able to have a clean version of the waveform in spreadsheet form, similar to what we have currently for the pitch detection data. Unfortunately, it will currently only export this for one cycle of the CQ waveform. Tyler has been in contact with VoceVista about this, and there should be a way of exporting a full file. However, it’s possible that this won’t actually end up working. We’re prepared for this possibility, as there’s also an option of using a python scraper to compile a sheet out of the discrete chunks of data that VoceVista gives us. As long as the data is formatted well, it should be extremely simple to display in plot form using DearPyGui, just like I did for the audio signal.
  • In the meantime, I’ve made progress on restructuring the way I initially formatted the page to make it straightforward to insert new graph lines once the CQ data comes.This might seem like a minor change, but it ended up taking me some time to figure out how to do properly due to my unfamiliarity with DearPyGui. Basically, I turned the structure of the page into a table with one column. Within each row of the table, multiple graph lines can be inserted. The x-axis running along each row will be time, and the cursor will go over any graph lines in the given row. This structure works well because the table rows delineate which part of the data the cursor should be focused on, while also allowing the flexibility to add or subtract different lines inside of the row (eg: EGG signal, audio signal, potential future sheet music).
• Overall
  • Since I don’t currently have CQ data to work with, the playback view is a bit difficult to demo, but the basic version is now complete (though, I’d like to add the cursor scroll function). Additionally, we’ve had some shifts in our thinking about how to connect VoceVista to our application (see the team update), so the recording page planning is also still in flux. In the original plan, I was working on data analytics views this week, but I haven’t yet started that, so I’ve pushed that to next week. I’m also pleased with my gradual progress learning about DearPyGui in general. It was a bit of a gamble for us to use, and while I certainly have encountered outdated documentation and a lack of online resources for more granular adjustments I want to make, it’s been immensely helpful for things like automatic scaling and graph functionalities, and I’ve enjoyed getting to learn a new framework.

Goals for next week:

• Use sample data to more clearly demonstrate the layered table rows in the playback view (for interim demo)
• Enable cursor scroll
• Expand page menus and navigation
• Hammer down specific parameters for data analytics views, and complete one view (eg, average CQ over time graph) using arbitrary sample data

This week I worked on implemented a better version of the previous audio player, and doing so in DearPyGui. I also spent some time working on the pitch analysis algorithm with Melina.

Here’s the basic window for the audio player, with a simple menu, button for file selection, and the window that’s going to be filled with the audio visualization once a file is selected:

Once the user selects an audio file (WAV or mp3), it’s turned into a waveform visualization (normalized so that all audio files are displayed on a y-axis from -1 to 1). If the waveform is longer than ten seconds, it gets split up into multiple lines, making it easier to see the details of longer recordings. This window is scrollable, with a hovering bar at the bottom containing a play/pause button that toggles playing of the audio file. The top menu, file selector button, and name of the selected file also hover so that the user has easy access to navigation.

The window also auto-scales when the user adjusts its dimensions:

I’m still running a bit behind due to the switch to the new framework and getting very thrown off by my sickness coming back from spring break. At this point, I was planning to have started integrating the CQ data, which I haven’t done. However, my progress this week has pretty much finished the basic audio playback (I just want to add a visual cursor and a restart button), which means I’ll be able to endeavor to add the CQ waveform next week.

Goals for next week:

• Add Visual Cursor
• Add Restart Button
• Work on parsing VoceVista output file & CQ waveform

I’ve unfortunately been knocked out with illness this entire week, and wasn’t able to make project progress. This puts me further behind on my goals. I will more thoroughly assess what needs to be done once I fully recover, hopefully in the next couple of days.

Well, I significantly overestimated what my bandwidth would be this week before spring break. I spent a lot of time working on our design report, and given that my spring break travel started on Thursday, I didn’t end up having time for much else. 

MVC vs MVVM

One thing that I did more research into this week was the model for our software architecture. Originally, I had conceived of our backend as MVC (Model View Controller) architecture due to my familiarity with this paradigm from my experience developing web applications. However, looking into it a bit more, it turns out that the standard for desktop applications is MVVM (Model View Viewmodel).

Basically, MVVM introduces complete separation of concerns between the GUI (View) and the database and program logic (Model) with the Viewmodel acting as a mediator. This will make it easy for us to develop these elements in separation. Plus, it’s a reactive architecture: the Viewmodel automatically responds to changes in the Model, and likewise, the View responds to changes in the Viewmodel, which will be useful for real-time updating, like the animation of the cursor that scrolls over the music. MVC is a similar paradigm, but less strictly enforced, and more suitable to web development. Of course, for either paradigm, it’s up to us how strict we’re going to be, and there’s always options to customize. Tentatively, I think this will be a helpful general framework for us to follow.

Last Week’s Goals Revisited

• Write my parts for the Design Review report – COMPLETE
• Research & make a decision about DearPyGUI (ASAP) – COMPLETE
  •  Given the sorts of data analytics we will need in the application, and our desire for a flexible and engaging user interface, we decided that DPG is our best option for
• Better documentation for code – IN PROGRESS 
  • started README file, but haven’t tested it on other devices
• Record Screen File Saving – NOT STARTED
• (Optional) Playback Screen Upgrades – NOT STARTED

Goals For Next Week

• Better documentation for code
  • Finish README file for setting up dependencies on different OS
• Re-implement basic menu, record screen, and playback in DearPyGui
• Record Screen
  • User can choose to either save/discard a recording instead of auto-saving
  • User can customize naming a recording upon saving
• (Optional) Playback Screen
  • Add cursor that moves while playing
  • Let user drag cursor to various points in the playback
  • Add separation of signal into rows + scrolling

Week Goals Revisited

• Implement the basic MVC application with homepage
  • COMPLETE – It’s not much to look at, but the homepage exists:

• Create GitHub repo
  • IN PROGRESS – I created a repo, but still need to update the README with instructions for using the app on different operating systems (and testing this with my teammates to make sure it actually works)
• Create baseline audio recording page functionality
  • COMPLETE – I created a simple page that can record an audio signal. However, given that we know now that the microphone signal will be going through the EGG directly, I don’t think this version will be used in the final product, though it’s useful for testing now (instead, I think that the audio signal will come from VoceVista– though, the record page could still be useful for giving the singer visual/auditory cues to go through given warmups)
• Create baseline audio playback page functionality 
  • COMPLETE (ish?) – I didn’t exactly give a definition for “baseline” here, but I do have a basic version of the page, including a visual of the amplitude of the audio waveform. There’s still some things left to do, though: I’ll need to splice the audio signal into rows, allowing the user to scroll through a longer recording and syncing more seamlessly once we add the EGG signal. I also haven’t yet gotten the cursor to work (showing where the user is in playback)

• Help Melina with creating wireframes for application
  • COMPLETE – we have designed basic versions of the main application views, which was very helpful for clarifying some things in the design (namely, the separation of recording pages from analytic pages, and we’re planning to have separate recording views for recording repertoire (the original vision) and for recording guided warmups (which ends up being more useful for formally tracking CQ over time)). Below, for example, is a view of the general idea for the CQ warmup page.

Overall, I’m pleased with this progress, as it gives us some good ground for developing the app further, and I’m on track with what we have on the schedule.

DearPyGUI

As noted in our team update, we’re seriously considering switching frameworks from TKinter to DearPyGUI for the sake of its more seamless mechanisms for data visualization. I’m a bit torn on this, because TKinter has very good documentation and is more of a standard choice– plus, I’ve already spent some time learning about it and getting the application started. On the other hand, I don’t want to be knee-deep into a TKinter application and suddenly realize that it doesn’t give us the functionality that we need– and I’d love to have a more modern look and feel for our app, though it’s not a top priority.

Goals for Next Week

• Write my parts for the Design Review report
• Research & make a decision about DearPyGUI (ASAP)
  • Copy a sample DearPyGUI app and gauge its intuitiveness
  • Determine a more specific list of plots that we’d like to have for our final project, and see if there’s anything that DearPyGUI clearly does better
  • Rewrite what I’ve done so far if we do decide to switch
• Better documentation for code
  • README file for setting up dependencies on different OS
  • Better code comments
  • Better MVP formalization
• Record Screen
  • User can choose to either save/discard a recording instead of auto-saving
  • User can customize naming a recording upon saving
• (Optional) Playback Screen (if we do switch to DearPyGUI, I don’t think I’ll have time for this)
  • Add cursor that moves while playing
  • Let user drag cursor to various points in the playback
  • Add separation of signal into rows + scrolling

This week, I learned more about the EGG and its setup and ground truth research methods with the rest of the term. I also continued to work on building my Tkinter skills and think about the frames needed for the desktop app.

School of Music Conversations

Before getting into technical stuff, I wanted to note some new thoughts from conversations with School of Music collaborators, particularly with regards to options for sheet music syncing and microphone recording. One thing that we hadn’t thought much about is the need of an accompanist to help our singers to stay on key, if we’re doing anything beyond basic warmups (where it would be relatively easy to play ascending/descending starting notes). Fortunately, there’s accompanists in the SoM who would be willing to help with this. I suppose one of the possible complications could be getting a clear microphone signal when multiple ins. However, a strong possible advantage of this setup is that it might actually help us with sheet music syncing– if the accompanist uses a MIDI keyboard, the direct MIDI input will be comparatively easy to match to the MusicXML file. This would be a helpful way of syncing to sheet music without forcing our vocalists to stay on a super strict tempo.

Speaking of the tempo issue, our vocalists expressed mixed feelings about singing to a metronome. On the one hand, both of them appreciate, to an extent, the opportunity to do more practice with a metronome. However, they also worry about having the room to properly and fully form notes as they sing, which (as I understand it) could be impeded by an overly rushed/strict tempo. I think we could definitely sacrifice some level of matching accuracy to give some leeway here. Also worth noting that both vocalists strongly preferred the idea of an audible metronome to a visual one. However, I’ve come to tentatively prefer the MIDI idea (if that ends up being possible) given that an accompanist will be necessary in any case, and adding an accompanist would make the metronome option more complicated besides.

That being said, we’ve decided that sheet music syncing is a stretch goal for now– even though, as a musician, I think this will be one of the most useful extra things we can do, and I’d definitely like to prioritize it if possible! However, it is secondary to simply recording and displaying the EGG/audio signals themselves, so I’m going to put a pin on this rabbithole until I get more of the basics done.

Tkinter Learning

I’d never used Tkinter before, so I started by following parts of this Real Python tutorial to get started, with particular focus on learning about frame widgets (which I plan to use for switching between windows in the application). 

Based on my experience with building web applications, and given the number of frames in the application, I decided to follow a Model-View-Controller (MVC) paradigm for the desktop app to keep things organized. After some trial-and-error, I worked through this tutorial by Nazmul Ahsan to learn how this can be done in Tkinter. 

Progress Status

I am still on schedule with my parts of the project according to the plan. That said, I haven’t really made progress in the code for the application itself, and I think I ought to have set apart more explicit time in the schedule for simply getting to know the technologies that I’m using for this project. 

Goals for Next Week

• Implement the basic MVC application with homepage
• Create GitHub repo
• Create baseline audio recording page functionality
• Create baseline audio playback page functionality (with room for adding a future EGG signal)
• Help Melina with creating wireframes for application