This week I spent a massive amount of time debugging and testing, as well as integrating the entire system together. The debugging was specifically for the cue stick detection model. After rigorous testing I realized that there were a number of edge cases that the cue stick detection completely failed at. Most of these specifically depends on how the user grips the cue stick. Varying grips cover different parts of the cue stick that the cue stick detection model relies on to detect the stick, which makes it hard to have consistent results. Further, skin tone also affects the rate of detection, with skin tones that are closer to the color of the cue stick confusing the model more.
I cycled through a number of varying methods for detecting the cue stick – HoughLines, polygon approximation, background subtraction, and many others. Unfortunately, none of these approaches covered all edge cases by 100% and was not reliable enough. I also incorporated a form of “memory” for the cue stick detection system, which essentially takes the union of detections across multiple frames in order to make up for its inconsistencies. Despite this, many times we still failed to detect the cue stick. My progress this week is slightly behind since the cue stick detection is not working properly. There is an easier way to address this issue – attaching AprilTags on the cue stick. However, this is a last resort and as a team we collectively wanted to avoid doing this in order to have a better user experience. If we use AprilTags to detect the cue stick, then the user must hold the cue stick in a specific position for it to be detected. I will try to put in at least another half-week into finding a way to fix the cue stick detection without AprilTags.
In the coming week, I aim to fix the cue stick detection model and continue with end-to-end integration of the system.