This week, we mainly focused on integrating the different components of our project to prepare for the Interim Demo, which is coming up soon.
We first successfully integrated Elliot’s bluetooth/accelerometer code into the main code. The evidence of said success was an audio response (a drum beat/sound) being triggered by making a hit motion with the accelerometer and ESP32 in-hand.
We then aimed to integrate my drumstick tip detection code, which was a bit more of a challenge. The main issue concerned picking the correct HSV/RGB color values with respect to lighting and the shape of the drumstick tip. We positioned the drumstick tip on the desk (which we colored bright red), in-view of the webcam, and took a screenshot of the output. I then took this image and used a HSV color picker website in order to get HSV values for specific pixels in the screenshot. However, because of its rounded, oval-like shape, we have to consider multiple shadow, highlight, and mid-tone values. Picking a pixel that was too light or too dark would cause the drumstick tip to only be “seen” sometimes, or cause too many things to be “seen”. For example: sometimes the red undertones in my skin would be tracked along with the drumstick tip, or the tip would only be visible when in the more brightly-lit areas of the table.
In order to remedy this issue, we are experimenting with lighting to find an ideal setup. Currently we are using a flexible lamp that clamps onto the desk that the drum pads are laid on, but it only properly illuminates half of the desk. Thus, we put in an order for another lamp so that both halves of the desk can be properly lit, which should make the lighting more consistent.
As per our gantt chart, we are supposed to be configuring accelerometer thresholds and integrating all of our code at the moment, so we are surely on track. Next week, I plan to look into other object/color tracking such as CamShift, Background Subtraction, or even YOLO/SSD systems in case the lighting situation becomes overly complicated. I also would like to work on fine-tuning the accelerometer threshold values, as we are currently just holding the accelerometer and making a hit-like motion rather than strapping it to a drumstick and hitting the table.