Posted on

Jerry’s Status Report for 9/23

I personally helped do research for and complete the proposal presentation slides for our proposal presentation Wednesday.  I mostly focused my research around the app development frameworks and tools we wanted to use (such as React Native and Swift), schedule creation, risk mitigation, and our solution approach .  I also worked with Max to create speaker notes for our presentation, and presented on Wednesday when Max fell sick Tuesday night.  I also did research later in the week on how we wanted to set up our app environment, where I looked into tools for file management (settled on react-native-fs library), developing apps for iOS (settled on Xcode environment), displaying musicxml files in our frontend (opensheetmusicdisplay npm package), and looking into if we can meet the app latency requirements with iOS devices (we can, iOS devices can operate at a 120 HZ refresh rate which is 8.3 ms updates, below our 10 ms requirement).  I also looked into modifying our design to have the signal processing done in C/C++ and then imported as an npm package into our backend, instead of doing all of the signal processing done in javascript on the backend.  This should improve the speed of our app’s real time response as C/C++ is a much faster language than javascript, and should also improve developer productivity and quality of life, as Max has expressed that he has more experience working in C than javascript.  

I believe my progress is on schedule as I am ready to familiarize myself with musicxml which is scheduled for next week on our gantt chart and that I will be ready to start development when we finish our design presentation, provided I have some form of access to a MacOS system.  

Next week I plan to look more into parsing musicxml files for our app.  Help complete the design slides, and look more into some of the dependencies between the tools we want to use and if this will cause issues.  I will also look into how we want to setup our C/C++ environment and start looking into the nitty-gritty of how we want to compile and how we want to setup our Make file.  I am also wondering if we want to use Docker to have a container for our development environment as this seems to be the most convenient, but I am also not sure how necessary it is.  I also want to look into coming up with a design for how all of our tools are going to be integrated together.

Posted on

Max’s Status Report for 9/23/2023

This week I continued to research implementation options and tools for pitch detection and rhythm detection. I have narrowed down the process for pitch detection to involve the FFT of the input audio in combination with Autocorrelation and either HPS (Harmonic Product Spectrum) or HPSS (Harmonic Percussive Source Separation). Both Autocorrelation and HPS are common approaches to pitch detection, but have their own issues with potential harmonic disruption and a limited minimum range of around 40 Hz respectively. By using the two together, I plan to implement a pitch detection algorithm that uses the harmonic approach of HPS (or HPSS) to allow the algorithm to be used across a wide of variety of instruments (each with their own harmonics) while using autocorrelation to overcome the frequency limits of HPS. For rhythm detection, I have found that there is ongoing research into this topic at CMU under Chris Donahue and have reached out to him regarding the methods I have looked into. The scope of approach is much more limited than Donahue’s, as we are restricting measurement to a specific tempo metronome in western classical marches, practice book exercises, and allegro’s instead of a general test against all sheet music, but would still like to learn more about Donahue’s work.

 

Even with the health issues I experienced this past week, I am still on schedule to continue research into this next week and finish our design proposal. This week I am going to be wrapping up my rhythm detection research, extend that into our music tracking algorithm research and implementation, and finalize these three main algorithms for our design proposal.  I am also hoping to hear back from Chris Donahue.

Posted on

Team Status Report for 9/23

This past week, Max had a health issue that prevented him from participating in the proposal presentation after previously planning on presenting the team’s slides. The main concern from this health issue was whether or not a similarly large health event would prevent Max from participating in the class for an extended period of time later on during the semester, impacting the group’s ability to complete the project. Given that our team is only two people, this is a significant concern and led to meetings with our professors, Byron and Tamal, and our team’s TA, Lindsey, to discuss potential contingency plans such as breaking up our current team and making groups of four with some of the other teams in the class. We are still exploring this decision, but plan on having a concrete answer on whether our team will stay together or split by early this week. Even as we consider the option of breaking up our team, we have continued to follow our team’s schedule in the case we do stay together, preventing any large schedule or workload changes.

We made one change to the design in that we decided not to do the signal processing in javascript in the backend, we would instead write all of the signal processing in C/C++, and then make all of our signal processing into a npm package and import it into our backend.  This change was necessary because latency of the app in terms of how fast it could refresh and do computations was a big worry we had, and so doing the bulk of the calculation in C/C++ instead of javascript should speed up the app considerably as C/C++ is a much faster language than javascript.  Additionally, since Max is doing a significant amount of the signal processing, this helps us segment our code more so Max has his own environment to unit test his implementation without having to understand significant amounts of the app design while he is trying to design and implement our pitch detection and rhythm detection algorithms.  This change however will increase the complexity of our project, as we now have to integrate C/C++ code into our project and we will likely have to deal with Make files, which we do not have significant experience with.  We will manage this increase in complexity through additional research into how we want to set up the C/C++ Make environment and consulting our TA’s, as a lot of our assignments here at CMU have involved Make files and C/C++ so we can very easily consult someone with more experience than us in this domain.

Our schedule has not experienced any changes and we are still making steady progress as we continue to research our tools and implementation methods leading up to our design proposal and presentation. Even with the issues around Max’s health earlier this week, we were still able to stay on track with our research and goals to complete our design proposal this week.

One important factor to consider in our project is its limited scope in terms of the types of music it can be applied to, which impacts the reach of the product. The project is restricted to judging pitch relative to the 12-tone equal temperament system, which is the most common system used in the west, but not the only system used globally, with many cultures across the globe using different systems. This is particularly important to consider as it potentially isolates our product from being used to practice music present in other cultures, and could reinforce a euro-centric view of music for users of the tool.