The most significant risks that could jeopardize the success of the project is really only inconsistent CV detection (usually varying lighting conditions). Since we don’t use machine learning, a lot of our computer vision detection uses more “deterministic” computer vision algorithms like HoughLines, contour detection, polygon approximation, etc. In one of the previous days we were testing, there was a lot of sunlight in HHA104 which confused the cue stick detection model as well as the cue ball detection model. This poses a serious potential issue with our system. To manage the risk for the cue stick detection, we were thinking of just resorting to attaching two AprilTags onto the cue stick. We already have AprilTag detection infrastructure integrated into our system, so this would not be a hard change at all. Our only hesitation is that using AprilTags for the cue stick would result in a poorer user experience, since the user would have to hold the cue stick such that the AprilTags are always facing up. Worst comes to worst though, we would go with this option. As for the cue ball detection issue, this really only is an issue with the yellow ball and cue ball. We have two contingency plan options: color the yellow ball a darker shade of yellow, or make the “cue ball” whichever ball the cue stick is pointing at. The second option does not align as well with how pool is actually played, but since this is more of a training system, enforcing the actual rules of pool isn’t as necessary for our use case.
There were some minor changes made to the existing design of the system. For one, the Flask backend had to be integrated with the primary backend; this just made it a lot easier to both do the image processing/physics computation as well as stream it to our frontend. Additionally, this is really helpful in debugging the project. The change does not incur much cost really, only a small part of the backend has to be rewritten. Another change is having a “Calibration” class. This helps us calibrate our backend upon the system starting. This involves things like detecting the walls and pockets, cropping each frame to avoid noise. We also implemented a sort of “memory” for the system. There is a Calibration Python class we created that stores data from previous frames. If we aren’t able to detect certain objects or have some inconsistencies frame-to-frame, we use this class to either take the union of all detected objects in the past n frames (not detecting enough objects per frame), or we find the “median” objects (inconsistent objects frame-to-frame).