Over the last week I’ve split my time between two main topics: finalizing the webapp/local server interface, and implementing an audio playback module. I spent a considerable amount of time on both of these tasks and was able to get myself back up to speed on the schedule after falling slightly behind the week before.
The webapp itself was already very developed and close to being done. There was essentially just one additional feature that needed to be written, namely the button that triggers the user’s local system to identify the locations and radii of the drum rings at the start of a playing session. Doing this implies sending a trigger message from the webapp to the local server that initiates the ring detection process. To do this, I sent a post request to the local server running on port 8000 with a message saying “locate rings”. The local server needed a corresponding “locate-drum-rings” endpoint to receive this message, which also needed to be CORS enabled. This means I needed a pre-request and post-request endpoint that sets the request headers to allows for incoming POST requests from external servers. This is done as follows (only the pre-request endpoint is shown):
@app.route('/locate-drum-rings', methods=['OPTIONS'])
def handle_cors_prefilght_locate():
response = app.make_default_options_response()
headers = response.headers
headers['Access-Control-Allow-Origin'] = '*'
headers['Access-Control-Allow-Methods'] = 'POST, OPTIONS'
headers['Access-Control-Allow-Headers'] = 'Content-Type'
return response
Though the CV module for detecting the locations/radii of the rings isn’t fully implemented yet, once it is, it will be as easy as importing the module and calling it in the endpoint. This is once of the tasks I plan on getting to in this coming week. Both of the endpoints on the local server “locate-drum-rings” and “receive-drum-config” (which receives 4 sound files and stores them locally in a sounds directory on the users computer) work as intended and have been tested.
The more involved part of my work this week was implementing a rudimentary audio playback module with a few of the latency optimizations I had read about. However, before I explain the details of the audio playback functions, I want to explain another crucial aspect of the project I implemented: the system controller. During a play session, there needs to be one central program that manages all other processes. i.e. receiving and processing accelerometer data, monitoring for spikes in acceleration and spawning individual threads for object detection and audio playback after any given detected impact. Though we are still in the process of implementing both the accelerometer processing modules and the object detection modules I wrote controller.py in a was that simulates how the system will actually operate. The idea is that then when we eventually get these subcomponents done, it will be very easy to integrate them given a thought out and well structured framework. For instance, there will be a function dedicated to reading in stream accelerometer data called “read_accelerometer_data”. In my simulated version, the function repeatedly retruns an integer between 1 and 10. This value is then passed off to the “detect_impact” function which determines whether a reading surpasses a threshold value. In my simulated controller this value is set to 5, so half for the readings trigger impacts. If an impact is detected, we want to spawn a new thread to handle the object detection and audio playback for that specific impact. This is exactly what the controller does; it generates and starts a new thread that first calls the “perform_object_detection” function (still to be implemented), and then calls the “playDrumSound” function with the drum index returned by “perform_object_detection” function call. Currently, since the “preform_object_detection” function isn’t implemented, it returns a random integer between 1 and 4, representing one of the four drum rings.
Now having outlined the controller I designed, I will explain the audio playback module I developed and some of the optimizations I implemeted in doing so. We are using the soundDevice library inside the sound_player.y file. This file is our audio playback module. When the controller first starts up, it calls two functions from the audio playback module: 1.) “normalize_sounds”, 2.) “preloadSounds”. The first call ensures that each of the 4 sounds in the sounds directory have consistent sampling rates, sampling widths, and use the same number of channels (1 in our case). This helps with latency issues related to needing to adjust sampling rates. The second function call reads each of the 4 sounds and extracts the sampling frequency and data, storing both in a global dictionary. This cuts latency down significantly by avoiding having to read the sound file at play time, and instead being able to quickly reference and play a given sound. Both of these functions execute before the controller even starts monitoring for accelerometer spikes.
Once an impact has been detected, the “playDrumSound” function is called. This function takes an index (1-4) as a parameter and plays the sound corresponding to that index. Sounds are stored locally with a formatted name of the form “drum_{x}.wav” where x is necessarily and integer between 1 and 4. To play the sound, we pull the data and sampling frequency from the global dictionary. We dynamically change the buffer size based on the length of the data, ranging from a minimum of 256 samples to a maximum of 4096. These values will most likely change as we further test our system and are able to reliably narrow the range to something in the neighborhood of 256-1024 samples. We then use the “soundDevice.play()” function to actually play the sound, specifying the sampling frequency, buffer size, data, and most importantly the device to play from. A standard audio playback library like pyGame goes through the basic audio pipeline which introduces latency in a plethora of way. However, by interfacing directly with the WASAPI (Windows audio session api) we can circumnavigate a lot of the playback stack and reduce latency significantly. To do this I implemented a function that identifies whatever WASAPI speakers are listed on the user’s device. This device is then specified as an argument to the “sounddevice.play()” function at execution time.
The result of the joint controller and sound player is a simulator that continuously initiates and plays one of the 4 sounds at random. The system is set up so that we can easily fill in the simulated parts with the actual modules to be used in the final product.
As I stated earlier, I caught back up with my work this week and feel on schedule. In the coming week I plan to develop a module to initially detect the locations and radii of the 4 drum rings when the “locate-drum-rings” endpoint receives the trigger from the webapp. Additionally, I’d like to bring the audio playback latency down further and develop more rigorous tests to determine what the latency actually is, since doing so is quite difficult. We need to find a way to measure the time at which the sound is actually heard, which I am currently unsure of how to do.