Rishabh and I ran into a fairly interesting DSP problem while working on the drum extraction logic in the software architecture. Our original instinct was that drums are just certain frequencies, right? That’s why Rishabh was able to implement a deterministic FFT to extract the 3 main drum sounds.However, we realized we actually wanted the full spectrum of drum sounds (up to 13) and this introduces a lot more complexity. Real drum kits are actually quite messy since multiple sounds can be occurring simultaneously. This causes the waveform to be a sum of sources, and there is no way for us to know what the original sources were with just a simple filter. The entire problem of drum transcription becomes mathematically ill-posed. We need some way of getting strong priors to actually predict what the original sources should be. This is where machine learning comes in.
Omnizart is a python library that solves exactly this problem. This project includes a pretrained model that is meant to optimally transcribe drum sounds from an input .wav file. I was able to write some inferencing code that loads the models weights, takes in a .wav file, and outputs the models predictions for the drum sound at every timestep. However, I am still running into some issues running this code due to some environment conflicts.
Given this progress, I feel like I am on schedule. The next steps for me would be to integrate this into the rest of the codebase and finding out a way to streamline inferencing the model with running the rest of the game engine.