Author: bconnaug

This past week, my time was mainly spent on creating a test to mimic a simplified version of our project. In MATLAB, I made a short video of a small red dot moving in a somewhat-square path over 4 colored rings (a still frame of the video is shown below, as I am not sure how to upload a gif here).

 

This is supposed to vaguely emulate the behavior of the tip of a drumstick (which we plan to paint red or some other bright color) moving over the drum rings. It is not exact, but the main goal was to just make the dot move around so that I could figure out how to detect it using CV later on. I also made the proportions approximately equal to the real drum rings we will be using.

Then, in VSCode, I wrote a short program using HoughCircles and other numpy and OpenCV functions to read in/process the video, then output one where the red dot is detected in every frame. Said “detection” is indicated by drawing a small neon blue dot over the targeted red one. One can also pause the video by pressing the spacebar to step through and analyze a given frame, or press ‘q’ to close/force quit the output window.

Since the main task for this past week was to work on the computer vision code to detect rings, I would say that I am on track.

In the next week, I would like to measure how long it takes for the red dot to actually be detected in each frame, which will give us a better idea about what latency ranges we can expect when processing the live video feed from the camera in the real-world implementation. I also want to get started on the sliding window that will house a preset number of the most recent frames from the live video feed. Eventually, locating the drumstick tip in each of these frames will help determine which drum sound to make when an accelerometer spike is detected (by making a hit-like motion with the drumsticks).

 

This past week, I discussed the purpose of a few components with Professor Tamal Mukherjee, mainly including how we plan to mount the camera that will have a top-down view of the drum rings and thus acquire data needed for CV processing. I also began to look at the pinout of the ESP32 microcontroller to determine which registers would be most relevant when interfacing with the MPU 6050 accelerometer, as well as found a few relevant OpenCV libraries and documentation that could be useful for the aforementioned processing. We did not have too much planned out for last week on our Gantt chart besides starting to research and potentially implement Computer Vision code, so I believe we are on schedule.

To remain on schedule, this upcoming week, I plan to put more time into narrowing down which OpenCV libraries are most relevant. I also will begin writing code to experiment with specific color and shape detection functions, and upload it to the group repository. This can potentially be accomplished by generating images of my own with varying levels of noise (to simulate potentially-blurry frames from the webcam) and ring sizes, and trying to detect those rings as well as filter out particular colors. I hope that this process will help us to determine color ranges when detecting the rings and drumstick tips from the camera’s video frames, as we would want to avoid having different lighting conditions affect the functionality of our project. For example, since the drumstick tips are relatively spherical, cast light/shadow on the edges and highest point will have different color values than the color we paint them in.