This week, I worked primarily on integration and debugging. I added an audio component to the game loop itself, allowing the music and game to play simultaneously. I was primarily focused on integrating mine and Yuhe’s parts of the game, adding some way to transition from the menus into the game itself by selecting the song the user would like to play. On a song request, the game must read which song was selected and then fetch the associated audio file and json representation of the game, which I still need to find a way to efficiently store.
Next week I’ll finish integration and audio synchronization, allowing for a seamless transition from menu to game and ensuring that the game itself is exactly what it needs to be for the user request.
As far as testing and verification, since it’s a bit more difficult to quantitatively test a game like this, I’ll start by having some user playtesting – I’ll then gather feedback and try to deploy it within the game. I’ll then try to measure performance, primarily FPS, and ensure that it meets our desired 30+ outlined in our initial design requirements. Further, I’ll likely need to just play the game with a number of different audio files, ensuring that the notes and music are synced, and making sure that there are no errors in parsing the JSON file.
This week, I continued testing and refining the rhythm analysis algorithm. I tested out a second version of the algorithm that more heavily weights the standard deviation of the onset strengths into determining whether to count a peak as a note or not. This version is much more accurate across various tempos, as shown in the figure below. These are the results of testing a self-made single clef piano composition. The first version would have more false positives at a very slow tempo and false negatives at a very fast tempo, the missed notes typically being any 32nd notes or some 16th notes. The second version when tested on the same piece performs much better, only missing a few 32nd notes at very fast tempos.
The verification methodology involves creating compositions in MuseScore and generating audio files to test the processing algorithm on. This way, I have an objective truth of the tempo and rhythm and can easily manipulate variables such as the instrument, dynamics, time signature, etc. and see how these affect the accuracy. Additionally, I also test the algorithm using real songs, which often have more noise and more blended sounding notes. Using a Python program, I can run the analysis on a song I uploaded, and playback the song while showing an animation that blinks on the extracted timestamps and record any missed or added notes. To verify that my subsystem meets the design requirements, the algorithm must capture at least 95% of the notes without adding any extra notes of single-instrument songs between 50 and 160 BPM.
Comparing results of V1 and V2 on a piano composition created in MuseScore
I also tested an algorithm that uses a local adaptive threshold instead of a global threshold. This version uses a sliding window so it compares onset strengths more locally, which can allow the algorithm to be more adaptive over the course of a piece especially when there are changes in dynamics. The tradeoff with this is that it can be more susceptible to noise.
I am on track with the project schedule. I think the current version is sufficient for the MVP of this subsystem, so further work will just be more extensive testing and stretch goals for more complex music. I have begun creating more compositions with even more complex rhythms, including time signature changes, which I plan to test this V2 on next week. I also will test the algorithm on pieces with drastic dynamic changes. I plan to play around with the minimum note length more as well. Since V2 is experiencing less false positives, I may be able to decrease this from the current 100ms to accommodate more complex pieces. Additionally, I want to test out a version that uses median absolute deviation instead of standard deviation to see if this outperforms V2. This method will be less sensitive to extreme peaks.
This week I focused on finishing integration of my game with the signal processing by adding a functioning JSON parser that will allow for the output from the music analysis to be played as a game. I also added a result splash screen that displays at the end of each game, with a few more detailed statistics and graphs that I plan to add to later.
Next week I’d like to finish integration to allow for a gateway between the main menus and the game itself, ensuring that each song saved by the user can be clicked on and played. I’d also like to clean up some UI and make the game more visually appealing.
This week, I continued finetuning the audio processing algorithm. I continued testing with piano and guitar and also started testing voice and bowed instruments. These are harder to extract the rhythm from since the articulation can be a lot more legato. If we used pitch information, it may be possible to distinguish note onsets in slurs, for example, but this is most likely out of scope for our project.
Also, there was a flaw in calculating the minimum note length based on the estimated tempo because sometimes a song that most people would consider 60 BPM, librosa would estimate 120 BPM, which is technically equivalent, but then the calculated minimum note length would be much smaller and result in a lot of “double notes”, or note detections directly after one another that resulted from one more sustained note. For the game experience, I believe it is better to have more false negatives than false positives. I think having a fixed minimum note length will be a better generalization. A threshold of 0.1 seconds seems to work well.
Additionally, In preparation to integrate the music processing with the game, I added some more information to the JSON output that bridges the two parts. Based on the number of notes in for a given timestamp, the lane numbers are randomly chosen from which the tiles will fall from.
Example JSON output
My progress is on schedule. Next week, I plan to finalize my work on processing the rhythm of single-instrument tracks and meet with my teammates to integrate all of our subsystems together.
This week, I continued to make the game into an actual game utilizing the new engine. Like I (maybe) said last week, the engine doesn’t cover much of what Unity did, which means there’s a lot more tedious work related to drawing and keeping track of data structures containing info about game elements. I was able to add back things like timing, scoring, tracking multiple note blocks at the same time, and the beginnings of a JSON parser that will generate the actual beat map.
The game looks and feels a lot more like an actual game now, barebones as it is. Next week, I’ll finish up the JSON parser to fully integrate the signal processing aspect into the game and add more visual feedback to the game/UI elements that should make it more engaging for the player. I’ll also add a gateway between the main menus and the game itself, allowing for more seamless transitioning between parts of the game.
This week I continued testing my algorithm on monophonic instrumental and vocal songs with fixed or varying tempo. I ran into some upper limits with SFML in terms of how many sounds it can keep track of at a time. For longer audios, when running the test, both the background music and the generated clicks on note onsets will play perfectly for about thirty seconds before the sound starts to glitch and then goes silent and produces this error:
It seems that there is an upper bound of SFML sounds that can be active at a time and after running valgrind it looks like there are some memory leak issues too. I am still debugging this issue, trying to clear click sounds as soon as they are done playing and implementing suggestions from forums. However, this is only a problem with testing as I am trying to play probably hundreds of metronome clicks in succession, and will not be a problem with the actual game since we will only be playing the song and maybe a few sound effects. If the issue persists, it might be worthwhile to switch to a visual test. This will be closer to the gameplay experience anyway.
Next week I plan to try to get the test working again, try out a visual test method, and work with my team members on integration of all parts. Additionally, after having a discussion with my team members, we think it may be best to leave more advanced analysis of multi-instrumental songs as a stretch goal and focus on the accuracy of monophonic songs for now.
I started out this week with exploring how to leverage Librosa to analyze the beat of songs that have time-varying tempo. These are the results of processing, using a standard deviation of 4 BPM, an iPhone recording of high school students at chamber music camp performing Dvorak Piano Quintet No. 2, Movement III:
When running the a test that simultaneously plays the piece and a click on each estimated beat, the beats sound mostly accurate but not perfect. I then moved on to adding note onset detection in order to determine the rhythm of the piece. My current algorithm selects timestamps where the onset strength is above the 85th percentile. It then removes any timestamps that are within a 32nd note of each other, which is calculated based on the overall tempo. This works very well for monophonic songs that can have some variation in tempo. For multi-instrumental tracks, it tends to detect the rhythm of the drums if present, since these have the most clear onsets, and some of the rhythm of the other instruments or voices.
I also worked on setting up my development environment for the new game engine Yuhe built. Next week I plan to continue integrating the audio analysis with the game. I also plan to adjust the rhythm algorithm to dynamically calculate the 32nd note threshold based on the dynamic tempo, as well as experiment with different values for the standard deviation when calculating the time-varying tempo. I also would like to look into possible ways that we can improve rhythm detection in multi-instrumental songs.
This week, I continued to work on the core game loop in Unity. I was able to add in a scoring system, and began work on implementing a json controlled beat map instead of dropping randomly generated notes. I also spent considerable time debugging a few things that ended up (mostly) being due to misunderstanding some Unity stuff. I also worked on writing the design review, and had to write up a few sections for it.
Next week, I think I’ll have to spend most of my time integrating; switching my sections of the game to use the game engine Yuhe developed, and writing it up in C++. I’d also like to do some work on the game’s visuals, as right now it doesn’t look great and could use some UI updates.
This week, I continued working on the main game while taking some additional time to focus on the design review.
I began implementing some more core features like using a JSON file to generate our beat maps as opposed to just having them place note blocks randomly, and began some of the work on the scoring system. Next week I aim to finish up with these two features and hopefully begin working on some more front end details, to make the game feel a bit more like a game for the time being.
I also made the slides for and presented the design review, which means I took some time outlining our project in a slideshow and doing a bit of presentation preparation. I’ll need to continue this a bit by writing up some of the design review’s written portion this week.
Made progress on core game loop. Wrote some scripts controlling note blocks and note block destruction/timing, currently note blocks are only randomly generated. I’m on track and want to continue progressing on the game by allowing the generated beats to be controlled by a JSON file instead of random generation, and incorporate more accurate timing feedback and improve the UI.
