Weekly Status Reports

This week, our group’s primary focuses were on the interim demo and continuing to develop/optimize our project as we get closer to the end of the semester.

Overall, the interim demo went well, we were able to successfully take a picture of the current state of the table, detect the pocket locations, detect and classify ball locations, run our best shot algorithm, and then display the best shot onto the board for the user to see. Below is a photo of what that display looks like, and here is an example of what that looks like in action.

A couple of things that we need to work on after the interim demo are

1. During the demo, we would manually send data between subsystems, ie Luke would take the picture of the board and send it to Kevin, who would run the CV algorithm on it and then send that data to Sammy to run best_shot and project the shot. We are currently working on a main file to run all of this on one computer for the final demo
2. Ball categorization is good, but still needs some work. Kevin is currently working on refining the color thresholding for ball categorization, as similarly colored balls are still being categorized as the same (2-ball and 4-ball).
3. Simulated annealing is still random, so sometimes the best shot algorithm returns a local minimum instead of the global minimum. Sammy is working on optimizing the shot simulation so that the best_shot’s simulated annealing algorithm can run with more iterations.
4. The camera is set to auto-focus, so we would sometimes need to refocus the camera before starting the whole process. Luke is looking into testing different configurations to achieve a more stable image feed that is always in focus.
5. Projector/Camera placement is good, but can be better. Luke is looking into the optimal placements for the projector and camera so that they most closely align with the real-world pool table.

Alongside these development/optimizations, we are working on a couple of post-MVP additions to our project. Luke is looking into creating a file that monitors ball movement through the camera so that it automatically starts the whole process when the balls stop moving, instead of requiring the user to click a button. Additionally, the team is discussing whether or not to use a Raspberry Pi to host a web server with a database of a users previous games so that they can see how much they’ve improved.

Overall, there is no change to our schedule — we are essentially at MVP. Everyone is on track for their tasks, and we believe we will have a good working final project by the time final demos happen.

This week I mainly worked on refining the color thresholding for ball categorization and started implementing a new solution regarding edge/rail detection that could serve as an alternative for pocket detection. Currently, for ball categorization solid differently colored balls are accurately being detected, but similar colored balls(blue 2-ball, purple 4-ball) struggle differentiating. To debug this problem I created a gradient function that shows the different HSV thresholding gradients to give us a visual reference. I also started implementing an edge detector as an alternative to our hough circles pocket detector, but it still needs a lot more refinement. I am on track to have a finalized pipeline soon and only have categorization and pocket detection left to refine.

This week, I worked on optimizing the physics simulation to improve a simulated shot’s runtime. During the interim demos last week, we noticed that the best shot algorithm would sometimes give a local minimum instead of a global minimum. I touched on this a bit last status report, but this is happening because simulated annealing is inherently random, and the fewer iterations it goes through results in more random outputs. To fix this, I tried implementing the physics simulation in C++, then using pybind11 to call the C++ simulation in the best shot algorithm file (which is in Python). Unfortunately, due to the overhead created by having to convert data from Python types to C++ types, the latency didn’t improve at all — the runtime was much slower because of the computational overhead.

I then tested three different variations to see which had the best runtime: 1) purely Python, 2) Python best shot & C++ shot simulation, and 3) purely C++. I figured that since C++ is an inherently quicker language than Python, option 3 would have the best runtime. However, through testing, I still found that option 1 (purely Python implementation) had the best runtime. I believe this is because Numpy vectorization in Python is incredibly optimized, and the C++ version of Numpy (Xtensor) is not. Below are the average runtimes of the shot algorithm for the three implementations on 10,000 random board positions (note that my computer was not plugged in, so runtime is slightly slower than the 25ms we were getting earlier with option 1).

Purely PythonPython & C++Purely C++

This next week I will look into different ways to optimize the runtime of the purely Python implementation. I am still on schedule and everything I am working on as of now is pretty much post-MVP.

This week, I focused on exploring improvements to the camera system to support motion detection capabilities. A major area of experimentation involved disabling the camera’s auto-blur and auto-focus features. These automatic settings were interfering with consistent image quality, especially during fast motion, so I’ve been testing different configurations to achieve a more stable image feed that’s better suited for detecting movement.

Alongside this, I’ve started preliminary work on motion detection by analyzing frame-to-frame changes between game states. My goal is to develop a reliable method for identifying when and where motion occurs on the table, so that the algorithm can run the physics simulations without user input.

I have also been discussing with my team on weather to pursue the use of a raspberry pi to host our own web server to keep track of previously played games.

I have been experimenting and have made minor progress on these tasks, however, I am on track with my progress.

 

This week, our team made significant progress in refining the camera and projector system, improving the physics simulation, and integrating the core components of our project in preparation for the interim demo.

Luke focused on optimizing the camera’s positioning and settings to improve image capture accuracy. He worked on integrating the camera with the CV algorithm to ensure reliable ball detection and reduce distortion. Additionally, he experimented with different mounting positions for both the camera and projector, addressing stability and visibility concerns. He also adjusted lighting conditions around the table to minimize shadows that interfered with ball detection.

Samuel completed the find_best_shot algorithm, transitioning from a brute-force approach to a more efficient simulated annealing method. This allows the system to find optimal shot angles, power, and spin within a set time constraint. However, due to the randomness of simulated annealing, the algorithm sometimes returns a local minimum rather than the best possible shot. To address this, he has begun rewriting the physics simulation in C++ to improve performance, reducing simulation time from 25ms to under 10ms. While this won’t be ready for the interim demo, he aims to have it completed for the final demo.

Kevin worked on ensuring accurate integration between ball/pocket coordinates and the physics simulation. He also focused on improving error handling for the CV system, particularly in mis-detected balls and pocket positions. Alongside Luke, he helped finalize the table setup, securing the projector mount and ensuring the camera is positioned correctly for accurate detection.

Our team’s next steps will be to finalize the system integration for the interim demo. Luke will continue to resolve the remaining camera system bugs and refining the UI to improve usability and game state visualization. Sammy will continue developing the C++ physics simulation. Kevin will continue to enhance the CV algorithm with ball detection and pocket detection.

This week I was working closely with the team to ensure we had our table prepped for demo which included: testing lighting, ensuring a secure projector mount, and camera angle/distance. Aside from helping the team me and Samuel have been working closely on ensuring an accurate integration between ball/pocket coordinates and physics simulation. Lastly, on the technical side, this week I spent a lot of time on error handling for the CV processing, this included things such as mis-detected balls and pocket position.

This week, I made significant progress on multiple aspects of the project. I continued testing the camera system extensively to ensure accurate and reliable image capture. In particular, I worked on refining the camera’s positioning and settings to improve its ability to capture the game state with minimal distortion. I also focused on integrating the camera system with the CV algorithm, making sure that the image processing pipeline correctly detects and interprets the positions of the balls on the table. To enhance the user experience, I worked on the UI, making it more intuitive for players to capture the game state with minimal effort.

Beyond software improvements, I spent time optimizing the physical setup of the camera and projector. I experimented with different mounting positions and configurations to ensure stability while maximizing visibility. Additionally, I worked on adjusting the lighting conditions around the table to reduce shadows, which were interfering with the CV algorithm’s ability to detect the balls accurately. This involved testing various light placements and intensities to find an optimal setup that enhances detection reliability.

I have been encountering a few bugs with the camera system, but expect to have them resolved before the interim demo. Overall, I am on track with my progress.

^ Image of camera debugging

^ image of table from system

This week, I finished the find_best_shot algorithm and created a variation of the display mapping algorithm that creates the image of the shot we will project on the board.

For the find_best_shot algorithm, I switched the implementation from a brute force approach to a much quicker simulated annealing approach. This approach calculates the initial angle between the cue ball and the target ball (the lowest numbered ball on the board). This angle is either the direct angle between the cue ball and target ball, or the angle that bounces the cue ball off a wall into the target ball. The algorithm then adds a random amount of angle, power, and spin to the previous shot and then simulates it. If this selected shot improves the solution, it is always accepted. Otherwise, the algorithm makes the move anyway with some probability less than one, which exponentially decreases based on the “badness” of the move.

To fit within the use-case requirements, I constrain the algorithm to return the best angle, power, and spin within either 1000 iterations or 4 seconds. This is then passed into the display mapping algorithm to show what the outcome of the shot will be. These images are attached below.

However, as you can see, since the simulated annealing algorithm is essentially random, the 4-second constraint can sometimes lead to different shots depending on what “direction” it takes in hill-climbing. This means that it will sometimes return a local minimum instead of the global minimum shot. For example, in the attached images, the best shot algorithm sometimes returns the shot that just pots the 1-ball (second image), a local minimum, instead of the shot that pots the 9-ball (first image), which is a global minimum since it wins you the game.

To fix this, I have begun to create a C++ implementation of the physics simulation to speed up the simulation time from an average of 25ms to hopefully under 10ms. I will not be finished with this implementation in time for the interim demos, but I hope to be finished with this by the time final demos come around.

I am currently on schedule and will be meeting with Kevin and Luke to integrate our three DeepCue subsections in time for the interim demo.

This week our team continued to make steady and significant progress especially now that we have the physical table and foundation for our project. While there are a couple roadblocks in our physical setup, the software progress is on track, and we are still confident in meeting our desired MVP goals.

Luke focused on setting up the pool table mount for the camera and projector system. He encountered challenges with securely attaching the equipment while upholding safety standards. Initial mounting attempts either made the system unstable or darkened the playing area, impacting out CV system. Luke plans to refine the mount further to balance stability, visibility, and functionality, with the goal of having the camera and projector system fully operational by the end of next week.

Samuel made progress on the find_best_shot algorithm for the physics simulation. His current approach successfully identifies 9-ball shots that meet key criteria, such as hitting the lowest-numbered ball, avoiding scratches, potting balls, and positioning the cue ball strategically. However, the algorithm still relies on a brute-force approach. Samuel is exploring the use of Simulated Annealing to optimize the algorithm and reduce computation time to under four seconds, as required by our use-case. He remains on schedule and aims to complete the algorithm and display mapping in time for MVP testing.

Kevin supported Luke with the physical setup, providing input on camera/projector placement and lighting to minimize image quality issues. He also began implementing code to project predicted holes, pockets, and ball positions onto the physical table. Kevin is on track and will pivot his efforts toward smooth integration, including refining position coordinates and addressing potential image occlusions.

The team is confident in overcoming current challenges and delivering a functional and polished MVP in time for carnival.

This week I worked on helping Luke setup the table and give ideas on how to setup the camera/projector system and lighting in order to minimize image quality issues. Aside from helping with the physical components I started working on implementing code to project my predicted holes, pockets, and ball placements on the physical table. I am on track and will try to pivot my work towards smooth integration, including position coordinates and mitigating image occlusions. Below I have attached an image of the coordinate system we are using and how it looks on the physical table.