{"id":45,"date":"2025-02-08T23:07:25","date_gmt":"2025-02-08T23:07:25","guid":{"rendered":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s25-teamd4\/?p=45"},"modified":"2025-02-09T19:44:47","modified_gmt":"2025-02-09T19:44:47","slug":"susannas-status-report-for-2-8-25","status":"publish","type":"post","link":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s25-teamd4\/2025\/02\/08\/susannas-status-report-for-2-8-25\/","title":{"rendered":"Susanna’s Status Report for 2\/8\/25"},"content":{"rendered":"

While we\u2019re still waiting to actually acquire the EGG sensor, my goal is to get a head start on planning out the recording feature + sheet music tracking. Ideally, the user will be able to input sheet music and a recording of them singing, and when they analyze their feedback in the app, the two will sync up. While the sheet music element wasn\u2019t initially something we deemed important, it became a priority due to how this would dramatically increase usability, allowing the user to visually interact with the app\u2019s feedback rather than be forced to relisten to a potentially lengthy audio file in order to get context for the EGG signal they see. It\u2019s also a significant way of differentiating our project from our main existing \u201ccompetitor,\u201d VoceVista, which doesn\u2019t have this feature. Additionally, if for some reason the EGG thing falls through, this mechanism will still be useful if we end up measuring a different form of vocal health.<\/span><\/p>\n

General Brainstorm<\/b><\/p>\n

Here\u2019s what I\u2019m envisioning, roughly, for this particular view of the app. Keep in mind that we don\u2019t yet know how the feedback is going to look, or how we want to display the EGG signal itself (is the waveform showing vocal fold contact actually useful? Maybe we just want to show the closed quotient over time?). But that\u2019s not my problem right now. Also note that there ought to be an option for inputting and viewing an audio\/EGG signal <\/span>without<\/span><\/i> a corresponding sheet music input, as that\u2019s not always going to be possible or desirable.<\/span><\/p>\n

\"\"<\/p>\n

The technical challenges here are far from trivial. Namely:<\/span><\/p>\n

    \n
  1. How do we parse sheet music (what <\/span>should <\/span><\/i>be sung)?<\/span><\/li>\n
  2. How do we parse the audio signal input (what\u2019s <\/span>actually <\/span><\/i>being sung)?<\/span><\/li>\n
  3. How do we sync these things up?<\/span><\/li>\n<\/ol>\n

    Researching how, exactly, this synchronization could be achieved, the prospect of a complicated system using pitch recognition and onset detection is daunting (though, we do hope to incorporate pitch detection later on as one of our analytic metrics). Doing something like fully converting the input signal into a MIDI file is difficult to achieve accurately even with established software, so, basically a project in its own right.\u00a0<\/span><\/p>\n

    So here\u2019s an idea: what if the whole synchronization is just based on tempo? If we know exactly how long each beat is, and can detect when the vocalist starts singing, we can match each measure to a specific timestamp in the audio recording without caring about the content of what\u2019s actually being sung. This means that all the app needs to know is:<\/span><\/p>\n

      \n
    1. Tempo<\/span>\n
        \n
      1. This should be adjustable by user<\/span><\/li>\n
      2. Ideally, the user should be able to set which beats they want to hear the click on (eg, full measure vs every beat)<\/span><\/li>\n<\/ol>\n<\/li>\n
      3. Time signature<\/span>\n
          \n
        1. Could be inputted by user, or extracted from the score<\/span><\/li>\n<\/ol>\n<\/li>\n
        2. Where measure divisions occur in the score\u00a0<\/span>\n
            \n
          1. This should be detectable in a MusicXML formatted file (the common file format used by score editors, and probably what we\u2019ll require as the format for the sheet music input)<\/span><\/li>\n
          2. However, if this is the only thing that needs to be extracted from the score, reading just the measure divisions off of a PDF seems more doable than the amount of parsing originally assumed necessary (extracting notes, rhythms, etc)<\/span>\n
              \n
            1. But let’s not overcomplicate!<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<\/li>\n
            2. When the vocalist starts singing<\/span>\n
                \n
              1. Onset detection<\/span><\/li>\n
              2. Or, the recording page could already include the sheet music, provide a countdown, and scroll through the music as it\u2019s being sung along to the click track. That could be cool, and a similar mechanism to what we\u2019d be using for playback.<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

                Potential difficulties\/downsides:<\/span><\/p>\n

                  \n
                1. Irregularities in beats per measure<\/span>\n
                    \n
                  1. Pickup notes that are less than the full measure<\/span><\/li>\n
                  2. Songs that change time signature partway through, repeats, DC Al Coda, etc<\/span>\n
                      \n
                    1. We just won\u2019t support these things, unless, maybe, time signature changes come up a lot<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<\/li>\n
                    2. A strict tempo reduces the vocalist\u2019s expressive toolbox\u00a0<\/span>\n
                        \n
                      1. Little to rubato, no tempo changes<\/span><\/li>\n
                      2. Will potentially make vocalists self-conscious, not sing like they normally would<\/span><\/li>\n
                      3. How much of an issue is this? A question for our SoM friends<\/span>\n
                          \n
                        1. Sacrifice some accuracy in the synchronization for the sake of a bit of leeway for vocalists?\u00a0<\/span><\/li>\n
                        2. If an individual really hates it, or it\u2019s not feasible for a particular song, could just turn off sheet music sync for that specific recording. Not necessarily a deal breaker for the feature as a whole.<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<\/li>\n
                        3. Click track is distracting<\/span>\n
                            \n
                          1. If it\u2019s in an earpiece, could make it harder to hear oneself<\/span>\n
                              \n
                            1. Using something blinking on screen, instead of an audio track, is another option<\/span><\/li>\n
                            2. And\/or, will singing opera make it hard to hear the click track?<\/span><\/li>\n<\/ol>\n<\/li>\n
                            3. If it\u2019s played on speaker, could be distracting on playback, plus make the resulting audio signal more difficult to parse for things like pitch<\/span>\n
                                \n
                              1. This could be mitigated somewhat by using a good microphone?<\/span><\/li>\n
                              2. Again, the visual signal is an option?<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<\/li>\n
                              3. Latency between producing click track and hearing it, particularly if using an earpiece<\/span><\/li>\n<\/ol>\n

                                Despite these drawbacks, the overall simplicity of this system is a major upside, and even if this particular mechanism ends up insufficient in some way, I think it\u2019ll provide a good foundation for our sheet music syncing. My main concerns are how this might limit vocalists expression, and how distracting a click track might be to the singer. These are things that I will want to bring up at next week\u2019s meeting with School of Music folks.<\/span><\/p>\n

                                Workflow<\/b><\/h3>\n

                                In the meantime, this will be my order of operations:<\/span><\/p>\n

                                  \n
                                1. Finalize framework choice<\/span>\n
                                    \n
                                  1. We\u2019ve decided that we (almost certainly) will be building a desktop app with a Python backend (the complications of a webapp seem unnecessary for a project that only needs to run on a few devices in a lab setting, and Python has very accessible\/expansive libraries for signal processing)<\/span><\/li>\n
                                  2. The frontend, however, is still up in the air\u2013 this is going to be my main domain, so it needs to be something that I feel comfortable learning relatively quickly<\/span><\/li>\n<\/ol>\n<\/li>\n
                                  3. Set up basic \u201cHello World\u201d<\/span>\n
                                      \n
                                    1. Figure out a basic page format<\/span>\n
                                        \n
                                      1. Header<\/span><\/li>\n
                                      2. Menu<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<\/li>\n
                                      3. Create fundamental music view<\/span>\n
                                          \n
                                        1. Both the recording and playback pages will use the same foundational format<\/span><\/li>\n
                                        2. Parse & display the MusicXML file input<\/span>\n
                                            \n
                                          1. This could be very complicated. I have no idea.<\/span><\/li>\n<\/ol>\n<\/li>\n
                                          2. Separate music into lines, with room for modularity (to insert EGG signal data etc)<\/span><\/li>\n
                                          3. Figure out where measure separations are<\/span>\n
                                              \n
                                            1. Measures will likely need to all be equal widths for the sake of signal matching ultimately? Not sure if this is crucial yet<\/span><\/li>\n<\/ol>\n<\/li>\n
                                            2. Create cursor that moves over score in time, left to right, when given tempo<\/span>\n
                                                \n
                                              1. On the record page, tempo will be given explicitly by the user. On the playback page, the tempo will be part of the recording\u2019s metadata<\/span><\/li>\n
                                              2. How the cursor aligns with the measures will depend on the MusicXML file format and what is possible. Could just figure out amount of space between measure separation lines + move cursor smoothly across, given the amount of time per measure<\/span><\/li>\n<\/ol>\n<\/li>\n
                                              3. Deal with scrolling top-to-bottom on page<\/span><\/li>\n<\/ol>\n<\/li>\n
                                              4. Create record page<\/span>\n
                                                  \n
                                                1. Set tempo<\/span><\/li>\n
                                                2. Record audio\u00a0<\/span>\n
                                                    \n
                                                  1. While music scrolls by, or by itself<\/span><\/li>\n<\/ol>\n<\/li>\n
                                                  2. Countdown<\/span><\/li>\n
                                                  3. Backing click track<\/span><\/li>\n<\/ol>\n<\/li>\n
                                                  4. Create playback page<\/span>\n
                                                      \n
                                                    1. Audio sync with measures + playback<\/span><\/li>\n
                                                    2. Play\/pause button<\/span><\/li>\n
                                                    3. Drag cursor<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

                                                      Picking Framework & Technologies<\/b><\/h3>\n

                                                      I considered several options for a frontend framework for the Python desktop app. I was initially interested in using Electron.js, as this would work well with my existing knowledge of HTML\/CSS\/JavaScript, and would allow for a high level of customizability. However, Electron.js has very high overhead, and would introduce the complications of basically having to run an entire web browser instance in order to run the app. I also considered using PyQt, but I haven\u2019t had experience with this framework, and it seemed like its idiosyncrasies would mean a pretty steep learning curve. So, I\u2019ve decided to proceed with using TKinter, the standard Python GUI\u2013 in my small experiments, it seems to so far be straightforward, and since I\u2019m starting out with some of the more complicated views of my GUI, I think I\u2019ll be able to tell fairly quickly whether or not it\u2019ll be sufficient for my purposes.<\/span><\/p>\n

                                                      Finally, I wanted to learn how to parse and display MuseScore files in a Python program. I\u2019ve started by learning how to use music21, a package developed in collaboration with the MIT music department, that offers a wide range of functionalities for dealing with MusicXML files. I\u2019ve only really had the time to get this library set up, so I have yet to actually do much with it directly. Till next week!<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

                                                      While we\u2019re still waiting to actually acquire the EGG sensor, my goal is to get a head start on planning out the recording feature + sheet music tracking. Ideally, the user will be able to input sheet music and a recording of them singing, and when they analyze their feedback in the app, the two … <\/p>\n

                                                      Continue reading “Susanna’s Status Report for 2\/8\/25”<\/span><\/a><\/p>\n","protected":false},"author":1120,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[4],"tags":[3],"_links":{"self":[{"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s25-teamd4\/wp-json\/wp\/v2\/posts\/45"}],"collection":[{"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s25-teamd4\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s25-teamd4\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s25-teamd4\/wp-json\/wp\/v2\/users\/1120"}],"replies":[{"embeddable":true,"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s25-teamd4\/wp-json\/wp\/v2\/comments?post=45"}],"version-history":[{"count":2,"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s25-teamd4\/wp-json\/wp\/v2\/posts\/45\/revisions"}],"predecessor-version":[{"id":72,"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s25-teamd4\/wp-json\/wp\/v2\/posts\/45\/revisions\/72"}],"wp:attachment":[{"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s25-teamd4\/wp-json\/wp\/v2\/media?parent=45"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s25-teamd4\/wp-json\/wp\/v2\/categories?post=45"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s25-teamd4\/wp-json\/wp\/v2\/tags?post=45"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}