Over the last week(s), I’ve focused my time on both creating a tool for dynamically selecting/tuning precise HSV values and conducting tests on our existing system in anticipation of the final presentation and the final report. My progress is still on schedule with the Gant Chart we initially set for ourselves. Below I will discuss the HSV tuning tool and tests I conducted in more detail.
As we were developing our system and testing the integration of our CV module, BLE module, and audio playback module, we quickly realized that the approach of finding uniform and consistent lighting was pretty much unattainable without massively constraining the dimensions to which the drum set could be scaled. In other words, if we used a bunch of fixed lights to try and keep lighting consistent, it would a.) affect how portable the drum set is and b.) limit how big you could make the drum set as a result of a loss in uniform lighting the further the drum pads move from the light source. Because the lighting massively impacts the HSV values used to filter and detect the drum stick tips in each frame, I decided we needed a tool that allows the user to dynamically change the HSV values at the start of the session so that a correct range can be chosen for each unique lighting environment. When our system controller (which starts and monitors the BLE, CV, and audio threads) is started, it initially detects the rings locations, scales their radii up by 15mm and shows the result to the user so they can verify the rings were correctly detected. Thereafter, two tuning scripts start, one for blue and one for green. In each case, two windows pop up on the user’s screen, one with 6 sliding toolbar selectors and another with a live video feed from the webcam with the applied blue or green mask over it. In an ideal world, when the corresponding blue or green drumstick is held under the camera, only the colored tip of the drumstick should be highlighted. However, since lighting changes a lot, the user now has the ability to alter the HSV range in real time and see how this affects the filtering. Once they find a range that accurately detects the drum stick tips and nothing else, the user hits enter to close the tuning window and save those HSV values for the playing session. Below is a gif of the filtered live feed window. It shows how initially the HSV range is not tuned precisely enough to detect just the tip of the drumstick, but how eventually when the correct range is selected, only the moving tip of the drumstick is highlighted.
Following building this tool, I conducted a series of verification and validation tests on parts of the system which are outlined below:
1.) To test that the correct sound was being triggered 90% of the time I conducted the following tests. I ran our whole system and and played drums 1, 2, 3, 4 in that order repeatedly for 12 impacts at a time, eventually collecting 100 samples. For each impact, I recorded what sound was played. I then found the percentage of impacts for which the drum pad hit corresponded correctly to the sound played and found this value to be 89%.
2.) To verify that the overall system latency was below 100ms, I recorded a video of myself hitting various drum pads repeatedly. I then loaded the video into a video editor and split the audio from the video. I could then identify the time at which the impact occurred by analyzing the video and identify when the audio was played by finding the corresponding spike in the audio. I then recorded the difference between impact time and playback time for each of the impacts and found an average overall system latency of 94ms. While this is below the threshold we set out for, most impacts actually have a far lower latency. The data was skewed by one recording which had ~360ms of latency.
3.) To verify that our CV module was running in less than 60ms per frame, I used matplotlib to graph the processing time for each frame and found the average value to be 33.2ms per frame. The graph is depicted below. 
I conducted several other more trivial tests, such as finding the weight of the drumsticks, the minimum layout dimensions, and verifying that the system can be used reliably within a 3m range of the laptop, all of which yielded expected results as outlined in our use case and design requirements.
In response to the question of what new tools or knowledge I’ve had to learn to progress through our capstone project, Id say that the two technologies I had to learn about and learn how to implement were CV (via openCV and skimage), and audio playback streaming (via pyAudio). I had never worked with either of them before so it definitely took a lot of learning before I was able to implement any strong, working code. For CV, I’d say I probably learned the most from reading other people (especially Belle’s) initial CV code. Her code used all the techniques I needed to use for building my HSV range selecting tool as well as the module I wrote for initially detecting the locations and radii of the drum pads. I read through her code as well as various other forums such as stack overflow whenever I encountered issues and was able to learn all I needed in order to implement both of these modules. In the case of audio playback streaming, I’d say I learned it mostly through trial and error and reading on stack overflow. I probably went through 6 iterations of the playback streaming module before I found a solution with low enough playback speed. Because many other applications such as drum machines or electronic synthesizers encountered many of the same issues as I was when trying to develop an efficient playback module, there was a large amount of information online, whether that be regarding using pyAudio streaming or overall concepts on low latency audio playback (such as preloading audio frames, or initializing audio streams prior to the session)
In the coming week the most pressing goal is to determine why playing with two connected drumsticks at once if resulting in sch a drop in performance. Once we figure out wat this issue is, I hope to spend my time implementing a system that can reliably handle two drumsticks at once. Additionally, I hope to start working on either the poster or video as to alleviate some stress in the coming two weeks before capstone ends.