Rishabh’s Status Report for 2/21

I spent the beginning part of this week prepping for our design review presentation (which I think went pretty well). Concurrently, I was also improving our existing sound identification methodology to improve its accuracy, but during this process, I realized that our existing implementation wouldn’t scale beyond the 3 base sounds that I was currently identifying. The reason is that the 3 foundational drum sounds (kick, snare, and cymbals) can be identified solely using the frequency, but as we start incorporating other sounds such as tom-toms, rim shots, etc require more information than just frequency. Before I started fixing this issue by incorporating more parameters into my existing identification methodology, I thought that an ML model might already exist that handles this task for us. Because of Abi’s experience with ML, I tasked him with trying to get an ML model for identification working.

With the identification off my plate, I started working more on the game development side. Here, I’ve created a basic structure for the game, but there are still several improvements that need to be made. Currently, I feel as though I’m a little bit behind on the game design aspect, but because Abi seems to have started working with an ML model, I think he can help me get caught up this week, so overall with respect to the software, I think we’re in a good place.

Abishek’s Status Report for 2/21

Rishabh and I ran into a fairly interesting DSP problem while working on the drum extraction logic in the software architecture. Our original instinct was that drums are just certain frequencies, right? That’s why Rishabh was able to implement a deterministic FFT to extract the 3 main drum sounds.However, we realized we actually wanted the full spectrum of drum sounds (up to 13) and this introduces a lot more complexity. Real drum kits are actually quite messy since multiple sounds can be occurring simultaneously. This causes the waveform to be a sum of sources, and there is no way for us to know what the original sources were with just a simple filter. The entire problem of drum transcription becomes mathematically ill-posed. We need some way of getting strong priors to actually predict what the original sources should be. This is where machine learning comes in.

Omnizart is a python library that solves exactly this problem. This project includes a pretrained model that is meant to optimally transcribe drum sounds from an input .wav file. I was able to write some inferencing code that loads the models weights, takes in a .wav file, and outputs the models predictions for the drum sound at every timestep. However, I am still running into some issues running this code due to some environment conflicts.

Given this progress, I feel like I am on schedule. The next steps for me would be to integrate this into the rest of the codebase and finding out a way to streamline inferencing the model with running the rest of the game engine.

Team Status Report for 2/21

Progress has been going well this week. Since we just finished up with the design presentations, we feel really solid about all of the work we have ahead of us and feel on track.

Caleb has gotten to work on some of the hardware side of things and has written some transmitting logic in C++ that is working well within our latency budget. The biggest risk we see for ourselves right now is not getting all of our hardware in time and not being able to test it properly. We are confident that we can sink enough time into the software side and get it working, but if we don’t even have access to the critical hardware we need, we are most fearful about that. Once we get the piezos and other parts we need, we will be able to test our implementation out and feel more confident.

On the software side, we are still doing good. Rishabh has been working on implementing more of the game logic and has made good progress on that. Meanwhile, I have been doing research into potential ML solutions for extracting drum sounds from a .wav file.

There was one small change we made in our software architecture. Originally, Rishabh was using demucs and a Fast-Fourier Transform that he implemented to extract drum sounds from a song’s mp3 file. However, the issue with this is that FFT is limited to only extracting 3 sounds from the .mp3/.wav. Therefore, Abishek worked on getting an ML solution working for this. More details can be found in the individual status report.

Caleb’s Status Report for 2/21

I developed the nRF24L01 communication protocol on C++ for both the receiving logic and the transmitting logic that should be on both the  Ebyte E01-ML01DP5 and the transmitter. We had to quickly order a new set of transmitters because our order for ML01S were cancelled. I am currently testing the code on the pair of ML01DP5 modules I have right now and it seems as though the protocol is working with consistent 5 ms latency. I did this by taking timestamps of both transmission and reception of ack packets from the transceiver. My only concern would be the ability to supply power to the pads but for now it should be sufficient to plug in 9V batteries with a step down convertor to 5 or 3.3V which should last a couple of hours (2-3 hrs).

Furthermore, I am working on the assembly of the pad and will get to cutting up the wood backing, acrylic impact plate, both backing and padding foam.

One of the big bottlenecks is not having the receiving & central hub modules on hand to actually implement and test the code; additionally, we don’t have the piezo discs to actually test the implementation and threshold. So instead of waiting to develop the pad until the piezos come I started with the implementation of the wireless communication protocol allowing for us to still be on schedule. 

Abishek’s Status Report for 2/14

This week, my primary focus shifted toward refining the overall system and game design, as well as contributing to the design presentation to better articulate how users will interact with the platform. Rather than implementing features directly, I spent time thinking through the structure of the learning experience and how gameplay mechanics can meaningfully support skill development for beginner and intermediate drummers.

Specifically, I designed a level-based learning system in which users progress through increasingly difficult stages (easy, medium, and hard). Each level consists of a song that the user plays along with, while the system tracks performance by comparing expected drum hits against the detected input from the pads. A “miss” can be defined as either failing to hit a required note within a fixed timing window or hitting an incorrect pad. At the end of each song, the user is presented with a performance summary showing the number of correct hits versus missed notes, along with an overall accuracy percentage. If the user achieves greater than 95% accuracy, they are allowed to advance to the next difficulty level. This structure provides clear, measurable feedback while still keeping the experience game-like and motivating.

In addition to the core progression system, I also thought through extensibility of the platform. One important design goal is to allow users to upload their own .wav files so they can practice with songs they enjoy, rather than being limited to a predefined library. This introduces additional challenges around difficulty classification and timing extraction, which will need to be addressed in future weeks, but it aligns well with the project’s emphasis on flexibility and personalization.

Because my efforts this week were primarily focused on design and planning rather than implementation, I am slightly behind in terms of direct software development. However, I believe this time was well spent, as it clarified the requirements for the game logic and will make future implementation more structured and efficient.

My next steps will be to help decide and curate which songs belong in the easy, medium, and hard categories, and to begin translating this game design into a minimal interactive prototype. This will involve defining the exact criteria for hits and misses, implementing basic game state transitions, and preparing the interface to accept timing and sound information from the drum detection pipeline once integration begins.

Rishabh’s Status Report for 2/14

This week, my priority changed from designing the interactive interface to constructing a proof of concept for the converting from an audio .WAV file to timestamped drum sounds. Thus, this week, I research the identification techniques behind the basic 3 drum sounds (kick, snare, and cymbal) and existing tools for parsing WAV files. Then using a WAV parser and FFT library, I was able to write a tool which is able to identify when and what drum sound is played (even if multiple are made at the same time). The accuracy of the tool is still somewhat lacking, but it achieves its goal as a proof of concept showing that we can take an audio file and isolate the various sounds that are made within it.

From the software side, I think that I’m slightly behind as I reprioritized my task of completing a basic implementation of the learning interface. Because of this, my workload this week may slightly increase in an effort to get caught up.

My deliverable for next week will likely be to get caught up with the learning interface and start integration with my two components. Therefore, I will have to build a minimal interactive “game” system and enable transferring the sound and time information generated from the identification program into the “game” system and update the system’s state based on that input.

Team Status Report for 2/14

Overall, the project is moving along and we are adapting to the delay in the material acquisition well by developing our software side and some hardware code which we find hard to verify without the presence of the actual circuitry we are implementing on. The biggest risk is not being able to verify our coding constructs because we do not have the hardware aspect to build upon for our interfacing between all of our respective areas of design. We are trying to mitigate this risk by unit testing our functionality of software and hardware systems with similar products such as the Arduino Uno to mock that of the Arduino Pro Mini developing our data packaging algorithm. In the case we are unable to acquire the necessary parts for the project we can change from our miniature model to a bit larger model using the Raspberry Pi’s in the inventory instead of our small Daisy Seed for processing and changing it so that our logic can be moved between RPi and the computer rather than having it fully onboard as in the Daisy Seed.

We made a few changes to the hardware requirements/design choices of this design. We moved from ESP32 to the Ebyte E01-ML01 series because of the nRF24L01+ performances compared to the performances of the ESP32 on average latency. This article provided a comprehensive study comparing the costs of the different Wireless transmitting modules and I found that the Ebyte E01-ML01 most closely aligns with our hardware requirements for low latency. We just need an additional, inexpensive Arduino Mini Pro 3.3v chip to communicate with this transceiver module and send data and power accordingly. Also the E01-ML01DP5 comes with a multiceiver functionality where they can receive packages from multiple channels simultaneously.

Another change to the hardware design is the change from FPGA to the Daisy Seed Microcontroller. After listening to feedback on our proposal presentation, we found that the complexity in the code was not enough to offset the tradeoff of the low latency it provides. Furthermore, the Daisy Seed is specifically a microcontroller built for audio mixing with 64 MB of onboard RAM, an onboard Audio output pin, SPI pins for the receiver, FPU compliant, and usb port to communicate and receive power from the computer. This is a relatively fast microcontroller capable of handling some expensive or complicated operations which gives us the same advantages of an fpga but this microcontroller has a lot of RAM to allow us to store our audio samples. The SPI pins allow us to communicate with our receiver module and the usb port allows for communication with the computer to alter game state.

 

PART A:

From a psychological health and welfare perspective, the interactive learning platform promotes motivation, engagement, and sustained skill development through structured feedback and clear progression. The level-based system and performance summaries provide users with measurable goals and positive reinforcement, which can reduce frustration and performance anxiety often experienced by beginners. By allowing users to practice at home with portable, configurable equipment and even upload their own music, the system lowers barriers to musical participation and supports long-term learning regardless of access to formal instruction or dedicated practice spaces. Overall, the product aims to enhance user well-being by making music education safer, more approachable, and more inclusive.

Part A was written by Abishek

PART B:

As mentioned in our use case audience we are targeting beginner / intermediate musicians who want both the authenticity of certain instruments coupled with a portable and engaging learning experience to grow. So our game experience allows the user to be able to interact with our system in a manner conducive to education and improvement. The use of partitioning the system into distinctive pads, central hub, and computer allows for the user to configure different amounts of pads, instrumental sounds, and bring anywhere compared to the clunky electronic drums now. This will lower the barrier of entry for musicians into any field allowing there to be a greater intake and participation of the musical arts. I would say our system tries to provide the closest real world feedback while maintaining the benefits of the portable electronic pads.

Part B was written by Caleb

PART C:

When design our product and considering economic factors, we had the benefit of having several previously made products which implement various drum pad features at prices ranging from $60 to several hundreds of dollars. We want our product to improve the configurability and usability of the cheaper existing products, and thus many of our components had to available at low cost. This decision to use cheaper products caused us to make several component changes such as moving our processing from an FPGA to the Daisy Seed Microcontroller and moving our networking from an ESP32 to the Ebyte E01-ML01 series. Additionally, our mission to reduce cost also caused us to adjust our product solution to have each distributed pad be as simple as possible with the central hub being where most processing takes place. Since each pad is extremely simple, they can be made cheaply and thus most of our costs will come from the central hub. Here we’re also able to reduce costs since our interface is able to plug into commonly available devices such as a computer and audio speaker. We have our interactive learning platform can be hosted on any computer and after simple processing from our MCU, the output can be played from existing speakers. Since these are commonly available items, it’s unlikely that consumers will have to bear these costs, reducing the expense of our product. Thus, by using highly modular components and taking advantage of existing products and interfaces, we’ve been able to design our product to reduce costs while maintaining the fundamental learning experience.

Part C was written by Rishabh

Caleb’s Status Report For 2/14

This week I got (1) the BOM finalized and ordered the items for the very first prototype of both the pad and central hub system, (2) found online free drum sample libraries to load onto the central hub microcontroller for real time convolution to mimic real drum audio, and (3) developed the specific circuitry and connectivity of all hardware systems.

I don’t really think that the central hub will have too much variation in architecture so I feel confident that we don’t have to change the architecture/design. However, for the pads I ordered the very preliminary design which uses a rubber head and piezos to detect impact. This might not give the sensitivity we require to pinpoint location and impact characteristics so we might have to look for a different configuration but this requires the prototype to be tested using FSR, PVDF, and mesh head.

I started developing the base bootloader code for the Arduino Pro Mini  & Daisy Seed that we will be using for data packaging and central hub logic respectively. The Arduino Pro Mini can be flashed using an Arduino Uno so that should be fine but in the Daisy Seed I need to flash both the RAM with the audio samples on boot as well as the code I want to flash it with. So I am reading the datasheets currently to figure out the setup code for each of these.

I feel as though my process is only a couple days behind because we cannot start developing or testing the code until we have the hardware components and that is our biggest bottleneck in this project. However, the software side seems to be making some progress in their translation.

By next week we hopefully receive the materials we ordered so that we can develop a working prototype of our central hub ecosystem with the hardware and basic unit testing code for the audio output and the SPI interconnect with the receiver hub.

Abishek’s Status Report for 2/7

As Caleb mentioned in the proposal presentation, a lot of the work for this project can be parallelized across the hardware and software components. This week, I mostly brainstormed about possible game ideas and ways to implement it. I did some research on possible game engines and it came down to either Unity (C#) or Bevy (Rust).

Unity seems to be the more fleshed out project and will possibly have better community support, but learning C# might be a high barrier.

Rust on the other hand is less of an overhead since I have already used it before, however getting Bevy to read input from our hardware could potentially be more complicated based on my initial exploration.

By next week, I hope to have a clearer picture over the pros and cons of both game engines and finalize a decision (and also get aligned with Rishabh on the choice of what platform to use to design the game engine). Furthermore, I also want to start finalizing a plan for how we will process the user data to adjust game difficulty. This might be through some sort of simple ML model or other rule-based evaluation.

I would say we are on schedule.

Rishabh’s Status Report for 2/7

I spent the first part of this week looking in to various tools that our group could use to build the game end of our design that worked with our available strengths. I looked into several C# based tools like Unity and Godot but decided that the overhead of learning C# plus the fact that C# requires a programming paradigm we weren’t as favorable towards led me in a different direction. I ended up selecting Bevy which is a Rust based game engine that focuses on highly modular designs.

After selecting Bevy, I spent the second half of the week learning how to use Bevy, both to confirm that the tool I selected was the right one and to start designing our game. After messing around for a little, I was able to get a simple static game frame to render. Because I’m on track to get started with the game, I think we’re right on track with respect to the software side.

By next week, I plan on getting a very rough game working which includes only or fundamental actions, like a simple fixed game loop and processing user input (although the processing of our user input will have to change later since I’m unfamiliar with the type of data we can read from our Raspberry Pi).