Category: Rishabh’s Status Reports

This week I was able to fully get the networking aspect of the game working with the hosting and joining pages fully working. Additionally, this week, I completed several of the utility pages such as the intro page and a summary page at the end of the song. I also added back buttons on several of the pages so that a user doesn’t have to keep relaunching the program to play a new song. The addition of both of these features significantly improved the UI/UX of the game.

On the integration side, this week we confirmed that both the networking 2 player changes work with the hardware changes made by Caleb, and after some debugging, we got the integration to be successful.

With this being the final week, my goal for this week is to fully complete the software side of our product. This primarily includes cleaning up the UI of the game and also improving the learning aspect of the game such as evaluating the difficult of songs and suggesting new songs based on the score achieved on the previous song.

This week I was able to get a good start on the networking/multiplayer part of the software side of our product, but there are still some bugs/architectural issues that I’m running into which are preventing me from fully completing the 2 player functionality. Here’s what the current interface looks like (in the image, you can see that the connection between the host and the client has been made and now the host has the option to select some song [which is where some of the issues are arising]):

I’ve also made some minor UI improvements such as making it clearer when there is a well timed hit or not.

My goal for next week is to get the multiplayer functionality fully working so that we can start the integration testing with the full set of pads and the 2 players functionality. Also, during this week, I would like to get some people to start testing out our product so that we can see if there are any ways that we can quality of life improvements that we can make.

This week, both Caleb and I spent much of our time working on integrating both the hardware and software systems together. After some debugging and adjusting each of our interfaces, we got the integration working, so we’ve gotten to the point where we have an end to end basic product working.

Additionally, during the process of testing, I noticed that the ML model that we’re I was previously using (Omnizart) was doing a poor job at handing songs where the tempo was high, specifically with residuals from cymbals, so instead, I added a new drum transcription model (ADTOF) and then only register a cymbal note if both models register a note at that timestamp.

Finally, I also worked on improving the UI of our game, making a formal entry screen and a screen for uploading new songs (although this is still incomplete). Also, I added signals indicating when a specific note was actually played and an indicator showing if the timing of the hit note was off or not.

By next week, my goal is to get the UI in a much better place with it starting to look more ready for our final by completing the uploading songs screen and just overall making the UI look cleaner. Also, I would like to begin planning out the networking and multiplayer aspect of our game since I anticipate that this will require a significant redesign of some of the code.

This week, I’ve worked a lot on the software integration to properly connect with the interface that Caleb’s hardware model presented. My software model is currently able to read the inputs from hitting the pads, then parse the information from the Daisy Seed and identify both which pad and where on the pad the hit was made. There is currently some issues that are going on with inconsistencies in the data that I receive from the Daisy Seed that we might try to fix in software.

Additionally, this week I’ve fully gotten the Omnizart ML model working and so we’re able to parse a wider range of drum sounds.

By next week, our combined goal is to get our end to end product working with a relatively high accuracy so that we can begin more thorough validation and user based testing.

This last week, I’ve spent my time debugging my build infrastructure to get Omnizart (the ML model that we’re planning on moving our note identification module to) build within our existing software design. After a few hours of parsing some incredibly dense error messages, I finally got Omnizart to build.

After messing around with Omnizart, I’ve started to move our current note identification to using Omnizart, but I haven’t gotten this new pipeline to fully work yet.

Another thing I’ve been working on is improving the user experience for the learning platform by creating a home page and a page for users to upload new songs to get parse. Additionally, some of these steps were taking to begin the integration with hardware.

On that note, it seems like Caleb is largely ready to begin integrating hardware and software together, so a large part of our work in the next weeks will be to start integrating these two components together and get an end-to-end design working. Also, specifically for me, I want to continue trying to get Omnizart fully working as that will improve our accuracy and note types.

These last two I spent my time developing both the game and the note from music file generation. I’ve moved the game development towards being a 3D like UI to improve the user experience. Here’s what the game currently looks like:

I’ve also started moving the note generation towards using an ML model to allow for identifying more types of notes (outside the current Kick, Snare, Cymbals). This ML model is based on the Omnizart music platform.
The ML model is currently not working, and my plan for next week is to get the ML model working an to get the integration between the game, the ML model, and the hardware system working.

I spent the beginning part of this week prepping for our design review presentation (which I think went pretty well). Concurrently, I was also improving our existing sound identification methodology to improve its accuracy, but during this process, I realized that our existing implementation wouldn’t scale beyond the 3 base sounds that I was currently identifying. The reason is that the 3 foundational drum sounds (kick, snare, and cymbals) can be identified solely using the frequency, but as we start incorporating other sounds such as tom-toms, rim shots, etc require more information than just frequency. Before I started fixing this issue by incorporating more parameters into my existing identification methodology, I thought that an ML model might already exist that handles this task for us. Because of Abi’s experience with ML, I tasked him with trying to get an ML model for identification working.

With the identification off my plate, I started working more on the game development side. Here, I’ve created a basic structure for the game, but there are still several improvements that need to be made. Currently, I feel as though I’m a little bit behind on the game design aspect, but because Abi seems to have started working with an ML model, I think he can help me get caught up this week, so overall with respect to the software, I think we’re in a good place.

This week, my priority changed from designing the interactive interface to constructing a proof of concept for the converting from an audio .WAV file to timestamped drum sounds. Thus, this week, I research the identification techniques behind the basic 3 drum sounds (kick, snare, and cymbal) and existing tools for parsing WAV files. Then using a WAV parser and FFT library, I was able to write a tool which is able to identify when and what drum sound is played (even if multiple are made at the same time). The accuracy of the tool is still somewhat lacking, but it achieves its goal as a proof of concept showing that we can take an audio file and isolate the various sounds that are made within it.

From the software side, I think that I’m slightly behind as I reprioritized my task of completing a basic implementation of the learning interface. Because of this, my workload this week may slightly increase in an effort to get caught up.

My deliverable for next week will likely be to get caught up with the learning interface and start integration with my two components. Therefore, I will have to build a minimal interactive “game” system and enable transferring the sound and time information generated from the identification program into the “game” system and update the system’s state based on that input.

I spent the first part of this week looking in to various tools that our group could use to build the game end of our design that worked with our available strengths. I looked into several C# based tools like Unity and Godot but decided that the overhead of learning C# plus the fact that C# requires a programming paradigm we weren’t as favorable towards led me in a different direction. I ended up selecting Bevy which is a Rust based game engine that focuses on highly modular designs.

After selecting Bevy, I spent the second half of the week learning how to use Bevy, both to confirm that the tool I selected was the right one and to start designing our game. After messing around for a little, I was able to get a simple static game frame to render. Because I’m on track to get started with the game, I think we’re right on track with respect to the software side.

By next week, I plan on getting a very rough game working which includes only or fundamental actions, like a simple fixed game loop and processing user input (although the processing of our user input will have to change later since I’m unfamiliar with the type of data we can read from our Raspberry Pi).