Category: Team Status Reports

After our interim demo we met up again to consolidate our next step goals because we were a little ahead of schedule and moved on to integrate the multiplayer aspect of our games. We have 2 daisy seeds which allows us to split up our hardware of 6 pads into 3 pads for each player. There were some set backs in setting up the client server conection and communication while staying true to the latency of the game. Furthermore, on the hardware side I found some optimizations by reducing the size of our hit data packets which allows us to increase the transmission speed without suffering too much packet loss from 128kb to 256kb packages reducing our perceived latency.

Overall, we are on schedule and have the remaining few weeks to debug any problems that arise with our final implementation and we have the last few weeks to touch up on our presentation and continue to test for any hidden bugs.

Some next steps would be more on the software implementation side to be able to make the interactive gameplay more beginner friendly by adding checkpoints and a normalized scorebar to create more incentive to keep combos of hits going.

 

We successfully were able to demonstrate our product with a working UI for our guitar hero game with 3 pads or 3 inputs. We were able to adjust the difficulty of the song on the spot, reduce the number of double hits and mis-inputs, and overall have a positive user experience.

Hardware: Our hardware was mostly working as expected without much noise detected however we did experience some scenarios where we did run into potential issues. (1) Our initial drum pad design had a case that wasn’t flush to our impact plate and thus residual hits would tamper with the impact plate and cause multiple hits to be registered. Furthermore this could be an issue with the overall drum not being tight enough. (2) Our newest pad did have some issues with double hits and as I described in Caleb’s Status Report, it could be an inherent characteristic of the acrylic impact plate. This can be accommodated with some software fixes and maybe some dampening circuitry. (3) Lastly, our drum frames are in direct contact with the table that it is on which gives rise to some noises being picked up from table hits and I believe it can be lightened with rubber stop gaps between frames but also between screw and the floor.

Hardware Test: Caleb will then conduct a series of 25 consecutive hits in 3 sets: control represents dead center hits, group 1 which consists of hits to the area between the center of the drum and the rim, and group 2 which consists of hits only to the rim. Then I will check the outputs on my serial monitor from the daisy seed and find the percentage of hits that result in a double hit being registered. From these fixes I am hoping to lower the percentage of ghost hits to <8% (only 2 ghost hits for every 25 hits). One thing to note is that there are some instances were double hits are intentional, if the grip on the drum stick is very loose causing the bouncing of the stick head on the drum pad.

Secondly, latency was a big concern of ours so I will hook up one end of an oscilloscope to the output of a piezo pin on the drum then I will hook up the other end to the tx port of the daisy seed to be able to determine the difference in time of arrival of each packets and ensuring that the time difference is within our limits of 10-20 ms with 5% tolerance. (9.5-21 ms)

Software: We got the overall game functional, the extraction of the drums from a song work well in regards to any type of music file inserted, however the pipeline does take a lot of time. Due to us having to convert a song into wav file, pushing it through demucs which takes 2 ish minutes to parse the drums out of. We manually went through one song and compared the drum music sheet with the notes being outputted to find there was a 95% match rate which is great news.

Some difficulties we faced was moving our ML model from an x86 proprietary driver model to a more open model which has helped streamline our process a lot. And also receiving ghost screens very seldomly.

Steps we can take for the future of our software interface is (1) improving the UI so that players can easily tell the difference and arrangement of pads, (2) improving the gameplay so that players know how well they are doing (numerical feedback system and normalized score tracker) so that they don’t lose interest, (3) implementing the networking interface so that we can simultaneously run 2 independent pad systems playing a single game on 2 different computers.

Software Testing: Most of our software testing will probably take the form of user experience feedback in the early stages because we want to improve the gaming aspect with a point system that is incentivizing for our users. Including checkpointing system to accommodate for beginner drummers. And then we will move on to stress testing our networked model with multiple data packets to ensure that our timing between client and server is synced and we do so without a lossy network.

Overall, we would say that we are on track to being able to finish our product by the final deadline but also on track with our schedule.

We successfully added 3 pads to the system and will try to accommodate 6 pads on one system because the nRF24L01+ system can support up to 6 active pipes on the multiceiver functionality. We are currently using channel 108 which is on the border of the free legal 2.4 GHz RF spectrum. This is to avoid any confusion and noise on the main 76 channel for 2.4 GHz which is going to be very saturated during demo day. Furthermore, with new materials coming in we can continue production up to 6 pads so we can split into 3 pads half and half for networked gaming hopefully. The hardware side is trying to implement DAC onto the central hub instead of the software side so that there could be a standalone aspect to the product but the practicality will be determined soon. The software system needs a few more tweaks to be able to support user input for songs and such.

We are pretty on track and that gives us room to try and tackle our stretch goals of creating a networked game for 2 different sets of drums. There will most likely have to be some encoding on the daisy to switch between the different contexts but otherwise there will be not much hiccup to upscale our product and we can find other ways to improve the sensitivity and accuracy of our product.

We just need to add our location calculations on the code to be able to determine the different types of sounds to play on the output.

We have been able to debug the end-to-end connection of our overall system, with 2 fully functional pads with the central hub being able to control the 2.4GHz communication between the pads and allow for near simultaneous perception of the inputs. We successfully debugged our “multiceiver” functionality of the nordic nRF24L01+ modules. These packets are parsed by the central hub and sent to the computer via UART and serial output which the software is able to process and use as inputs to our game to get a viable scoring gaming mechanic. We definitely have a lot more features that we want to include but overall, the latency falls within our target latency and it provides good gameplay. Furthermore, the translation of the songs and parsing into the gaming format is pipelined and provides good integrity on the overall file.

Some improvements that can be made is the synchronization of audio and gaming inputs could be further improved. We can definitely add a lot more pads to the configuration. We still need to implement more sophisticated learning paradigms to our gaming design, such as the checkpointing mentioned within our reports. Our end-to-end assembly took some time but overall there is little to no bugs due to comprehensive testing and planning prior to assembly. There will need to be more iterative designing of the pad casing but overall we are almost on track.

We will meet over the week to continue working in lockstep to be able to reach our goals of more functionality added to our overall design by syncing each other on our progress every 2 days and combining the systems.

Still running into a few bugs on the digital to analog conversion on the daisy seed and will need to revisit the documentation to be able to debug the current problems.

We are almost done with a very rough end-to-end system from pad hit detection all the way to gameplay. Recently, we received our hardware parts and got most of our communications and developments rolled out for testing and further synchronization. Receiving our hardware parts almost 1.5 weeks late does push our schedule back slightly but with some of the work we did over the break we would say that we are almost on track. We just have to handle the interfacing between the central hub and the computer with a standardized communication protocol using MIDI format to convey our notes.

Some changes that occurred, is that we had to switch our transmitter modules from the Ebyte E01ML01S to the Aidepeen nRF24L01+PA+LNA modules because of some shipment hiccups but we were able to circumvent any large disruptions because these chips are all from Nordic and thus their libraries are mostly similar. Besides that we were still wondering if we should outsource our audio generation to our computer interface or through the Daisy Seed which has onboard DAC as well as SDRAM for storing our wav files for our DSP.

Part A: The main global impact of this project is how it reduces the barrier to learning drums for beginners. Typically, you would need to buy an expensive drum kit, an expensive teacher, and know how to read sheet music, etc. With PadL1, all that is needed are the few pads and access to a personal computer, which makes learning drums much more accessible on a global scale.

Part A was written by Abishek

Part B: This project attempts to accommodate cultural factors by having the learning platform not use any specific language and instead using universally understandable symbols and colors. For example, we identify the pad that needs to be hit matching each pad with a given color and the timing is shown by a note falling towards a designated hit zone. Also, in the case a missed note, we use an audio “error note” to designate this mistake. All of these features are culture agnostic and thus allows people for all cultures to enjoy our product.

Part B was written by Rishabh

Part C: This project does require a bit of resources from the environment for the hardware development of the pads such as rubber sheets, acrylic sheets, foam dampeners, and plywood backings. But in the future, with access to industry standard materials, I feel like we can make our system a lot more compact and thus avoid all the excess materials from prototyping. However, our system does not require further natural resources and is pretty compact without releasing any emissions thus presenting not much of a threat on the environment itself.

Part C was written by Caleb

Progress has been going well this week. Since we just finished up with the design presentations, we feel really solid about all of the work we have ahead of us and feel on track.

Caleb has gotten to work on some of the hardware side of things and has written some transmitting logic in C++ that is working well within our latency budget. The biggest risk we see for ourselves right now is not getting all of our hardware in time and not being able to test it properly. We are confident that we can sink enough time into the software side and get it working, but if we don’t even have access to the critical hardware we need, we are most fearful about that. Once we get the piezos and other parts we need, we will be able to test our implementation out and feel more confident.

On the software side, we are still doing good. Rishabh has been working on implementing more of the game logic and has made good progress on that. Meanwhile, I have been doing research into potential ML solutions for extracting drum sounds from a .wav file.

There was one small change we made in our software architecture. Originally, Rishabh was using demucs and a Fast-Fourier Transform that he implemented to extract drum sounds from a song’s mp3 file. However, the issue with this is that FFT is limited to only extracting 3 sounds from the .mp3/.wav. Therefore, Abishek worked on getting an ML solution working for this. More details can be found in the individual status report.

Overall, the project is moving along and we are adapting to the delay in the material acquisition well by developing our software side and some hardware code which we find hard to verify without the presence of the actual circuitry we are implementing on. The biggest risk is not being able to verify our coding constructs because we do not have the hardware aspect to build upon for our interfacing between all of our respective areas of design. We are trying to mitigate this risk by unit testing our functionality of software and hardware systems with similar products such as the Arduino Uno to mock that of the Arduino Pro Mini developing our data packaging algorithm. In the case we are unable to acquire the necessary parts for the project we can change from our miniature model to a bit larger model using the Raspberry Pi’s in the inventory instead of our small Daisy Seed for processing and changing it so that our logic can be moved between RPi and the computer rather than having it fully onboard as in the Daisy Seed.

We made a few changes to the hardware requirements/design choices of this design. We moved from ESP32 to the Ebyte E01-ML01 series because of the nRF24L01+ performances compared to the performances of the ESP32 on average latency. This article provided a comprehensive study comparing the costs of the different Wireless transmitting modules and I found that the Ebyte E01-ML01 most closely aligns with our hardware requirements for low latency. We just need an additional, inexpensive Arduino Mini Pro 3.3v chip to communicate with this transceiver module and send data and power accordingly. Also the E01-ML01DP5 comes with a multiceiver functionality where they can receive packages from multiple channels simultaneously.

Another change to the hardware design is the change from FPGA to the Daisy Seed Microcontroller. After listening to feedback on our proposal presentation, we found that the complexity in the code was not enough to offset the tradeoff of the low latency it provides. Furthermore, the Daisy Seed is specifically a microcontroller built for audio mixing with 64 MB of onboard RAM, an onboard Audio output pin, SPI pins for the receiver, FPU compliant, and usb port to communicate and receive power from the computer. This is a relatively fast microcontroller capable of handling some expensive or complicated operations which gives us the same advantages of an fpga but this microcontroller has a lot of RAM to allow us to store our audio samples. The SPI pins allow us to communicate with our receiver module and the usb port allows for communication with the computer to alter game state.

 

PART A:

From a psychological health and welfare perspective, the interactive learning platform promotes motivation, engagement, and sustained skill development through structured feedback and clear progression. The level-based system and performance summaries provide users with measurable goals and positive reinforcement, which can reduce frustration and performance anxiety often experienced by beginners. By allowing users to practice at home with portable, configurable equipment and even upload their own music, the system lowers barriers to musical participation and supports long-term learning regardless of access to formal instruction or dedicated practice spaces. Overall, the product aims to enhance user well-being by making music education safer, more approachable, and more inclusive.

Part A was written by Abishek

PART B:

As mentioned in our use case audience we are targeting beginner / intermediate musicians who want both the authenticity of certain instruments coupled with a portable and engaging learning experience to grow. So our game experience allows the user to be able to interact with our system in a manner conducive to education and improvement. The use of partitioning the system into distinctive pads, central hub, and computer allows for the user to configure different amounts of pads, instrumental sounds, and bring anywhere compared to the clunky electronic drums now. This will lower the barrier of entry for musicians into any field allowing there to be a greater intake and participation of the musical arts. I would say our system tries to provide the closest real world feedback while maintaining the benefits of the portable electronic pads.

Part B was written by Caleb

PART C:

When design our product and considering economic factors, we had the benefit of having several previously made products which implement various drum pad features at prices ranging from $60 to several hundreds of dollars. We want our product to improve the configurability and usability of the cheaper existing products, and thus many of our components had to available at low cost. This decision to use cheaper products caused us to make several component changes such as moving our processing from an FPGA to the Daisy Seed Microcontroller and moving our networking from an ESP32 to the Ebyte E01-ML01 series. Additionally, our mission to reduce cost also caused us to adjust our product solution to have each distributed pad be as simple as possible with the central hub being where most processing takes place. Since each pad is extremely simple, they can be made cheaply and thus most of our costs will come from the central hub. Here we’re also able to reduce costs since our interface is able to plug into commonly available devices such as a computer and audio speaker. We have our interactive learning platform can be hosted on any computer and after simple processing from our MCU, the output can be played from existing speakers. Since these are commonly available items, it’s unlikely that consumers will have to bear these costs, reducing the expense of our product. Thus, by using highly modular components and taking advantage of existing products and interfaces, we’ve been able to design our product to reduce costs while maintaining the fundamental learning experience.

Part C was written by Rishabh

We have developed our presentation and reviewing our design, adding new information from research papers that have developed augmented drum pads before and going to be developing our network protocol soon. We are finalizing our BOM over this weekend and having our design checked by the faculty and TA soon.

The largest risk at this moment is not getting our parts on time to get prototyping to allow significant leeway for our versions in due time. In order to combat this we need to get our design checked and BOM finalized as soon as possible.

Attached is the link to our BOM currently as we begin to make adjustments and order backups in case our design needs additional requirements. There are no additional changes to our design as of now.