This past week, I mainly wanted to start drafting a bit more Computer Vision code that could potentially be used in our final implementation. At the time, we had not yet received our accelerometers and microcontrollers in the mail, so I wanted to figure out how to use PyAudio (one of the methods we are considering for playing the drum sound) by creating simple spacebar-triggered sounds from a short input .wav music file. I figured that this process would also help us get an idea of how much latency could be introduced by this process.
I ended up adding code to the existing red dot detection code that first opens the .wav sound file (using the wave module) and a corresponding audio stream (using pyaudio). Then, upon pressing the spacebar, a thread is spawned to create a new wave file object & play the opened sound using the previously mentioned modules/libraries, as well as threading.
Though simple, writing this code was a good exercise for me to get a decent feel of how PyAudio works, as well as what buffer size we should use for when the audio stream is being written (i.e. when the input sound is being played). During our recent meeting with Professor Bain and Tjun Jet, we discussed possibly wanting to have a small buffer size of 128 bytes or so instead of the usual ~4096 to reduce latency. However, I found that the ~(256-512) range is more of a sweet spot for us, as having a small buffer also means that it would get filled more often (which could introduce latency accordingly, especially when it comes to larger audio files). I also found that when the spacebar was mashed quickly (which could simulate multiple, quick, consecutive drum hits), the audio lagged a bit towards the end, despite a new thread being spawned for every spacebar-press detection. I suspect this was likely due to the aforementioned small buffer, as increasing the buffer size seemed to remedy the issue.
Our gantt chart indicates that we should still be working on CV code this week, so I believe I am on schedule. This week, I hope to work with the ESP32 and accelerometers to get a feel for how they output data in order to determine how to process it. Setting up the ESPs is not necessarily a straightforward task, however, so I plan to work with Elliot in order to get them working (which would be confirmed by at least having some sort of communication between them and the laptop over Bluetooth). From there, if we are able to transmit accelerometer data, I would like to graph it and determine a range for the ‘spike’ generated by creating a ‘hit’ motion with the drumsticks.