Tjun Jet’s Status Report for April 20, 2024
In the past two weeks, I worked on creating an application for the users to select their ideal spin and velocity of the ball, and also worked together with Andrew to improve the cue stick detection and physics for spin collision.
In order to make our project more interactive, we decided to add in a web application for users to select where they intend to hit the ball (to provide for spin), and their intended strength of the ball. After the user has indicated their preferences, the user will execute their shot. Upon executing the shot our system will provide recommendations on whether the user should hit it harder, softer, or whether their shot was good. Based on the final location of the ball, we will also provide recommendations on the spin that the user provided.
This week, I contributed to the front end application to allow the users to select the spin that they want. The front end application will also display a video of the current predictions. Here is a picture of what the application looks like:
I also spent a good portion of time understanding the physics equations that were behind cue ball spin. As we weren’t able to call a physics engine directly in our implementation, we used the physics equations that were associated directly with the engine. The engine we used was PoolTool where we referenced these equations. Understanding these equations helped us to implement the ball spin.
Here are what some of the equations looked like:
Two things that I had to really learn and pick up in order to accomplish the task of working in a team project such as in capstone, was Github and Reading Documentation. Firstly, Github was an extremely important tool that helped us for version control. Initially, we realized that it was difficult for us to work on the same codebase but merging everything together at the same time. We definitely ran into a lot of issues like Git version conflicts, us not having well coordinated commits. As we went along, we got better and understood how github worked and we eventually improved overtime. This was important in the efficiency of accomplishing these tasks.
The second thing that I found really useful is to read documentation quickly and sieve out important information. This was very important when we were exposed to new material like using cv2 functions and other library functions. Along with github, these are things that are not explicitly taught in a class, but I feel are very important and necessary knowledge to have as ECE engineers. Apart from the technical knowledge gained from capstone, these were definitely two skills I valued most from this experience.
Given that we have fully implemented our physics model before spin physics, we are currently on track for whatever we want to accomplish for this week. We have analyzed the different equations for spin physics and are almost complete with implementing the physics, which we will test out tomorrow. We are a little bit behind on the trajectory verification, and that is the last bit of testing and verification that we will have to do in our meeting tomorrow.
Andrew’s Status Report for April 20, 2024
This week consisted of a lot of optimization, improvements, and cleaning up on my end. This primarily had to do with: 1) cue stick detection, 2) incorporating spin into our physics engine and displaying it. For cue stick detection, we realized that the stick trajectory was not very stable, and I tried many, many different ways of improving the cue stick detection. Ultimately, what worked the best was a combination of color masking, Gaussian Blur, contour detection, and enclosing rectangle + math. The cue stick detection is now significantly more stable and accurate compared to before; this was huge for our system, as the cue stick detection is a crucial part. If the stick detection is off, then the usefulness of the system decreases significantly. The second part Tjun Jet and I worked on was incorporating spin into our physics engine. Specifically, we took a deep dive on the physics of pool ball spin. We incorporated this into our physics engine when we case on both ball-wall and ball-ball collisions. Further, we also take in information about the user’s strike (location of strike + speed) and feed it into our physics engine. Our physics engine uses this input to modify the predicted trajectory in real-time. By having this web application interface link directly to the physics engine, the user is able to see in real-time how spin will affect the ball’s trajectory.
Our progress is on-schedule. In the coming week, we will be looking to finish our Final Presentation, present it, and make some last-minute touch-ups to our project. On a very practical level, I got a very hands-on introduction to computer vision applied to a specific problem. More on the theoretical side, I also had to refresh myself with physics and take a deep dive into the physics of pool. I knew almost nothing about computer vision coming into this project, and I didn’t have enough time to fully understand the theory behind everything by reading textbooks or taking a course. Instead, I found research papers, projects, videos, etc that had some parts overlapping with ours (or what we wanted to do), and I consumed that content. This is the learning strategy I took on to acquire this new knowledge, and I realized how important it is to be able to limit the scope of what you are learning in order to accomplish the tasks at hand. If I resorted to a traditional textbook or course, it would not be possible to finish our system in time. Much of the learning I did was on the fly, hands on.
Team Status Report for April 20, 2024
This week, our team is on schedule. We have not made any modifications to our system’s existing design; however, we did make some improvements to our testing plans. The main focus of this week was improving cue stick detection, implementing the spin feature and conducting tests and validation of our system.
This week, we made more improvements on the cue stick detection model, which we determined was a limiting factor in our system that caused some inaccuracies in our trajectory outputs. We were able to make it much more accurate with our new implementation than the previous method we applied. This week we also added to our web application to create a user interface where the user can select a location on the cue ball that they would like to hit. We then send this location to our physics model, which will calculate the trajectory of the cue ball based on the amount of spin it will have upon impact. We are basing our spin model off of an online physics reference and are currently working on implementing these equations. This is an effort that we will intend to complete in the following week. We will also conduct testing on this feature at the end of the coming week.
In terms of testing and validation, we utilized the following procedures to conduct our tests. We are still in the process of finishing up our testing to gather more data, which will be presented in the final presentation on April 22nd.
Trajectory Accuracy
- Aim cue ball to hit the target ball at the wall.
- Use CV to detect when the target ball has hit the wall.
- Take note of the target ball’s coordinate when it hits the wall.
- Compare the difference between this coordinate and the predicted coordinate of the wall collision.
Ball Detection Accuracy
- Position balls on various locations on the table, with special focus on target balls located on edges of walls, near pockets, and balls placed right next to each other.
- Apply the ball detection model and measure the distance between the actual ball’s center and the center of the perceived ball.
Latency
Time our code from when we first receive a new frame from the camera to the time it finishes calculating a predicted trajectory. We are timing our code for different possible cases of trajectories (ball to ball, ball to wall, ball to pocket) since the different cases may have different computation times.
The most significant risk we face now is the alignment of the projector projection to the table. We noticed that there may be a horizontal distortion from the projector that may not project the detected balls in the correct location on the table. While this would not affect the calculations of our accuracy in our backend system, it may not give the most accurate depiction to the user.
In the following week, we will be coming up with better ways to calibrate the projector to yield more consistent projection alignments. Furthermore, we will continue with our testing and validation efforts and potentially improve our system if there are any specifications that can be improved. As mentioned earlier, we will also continue to implement and improve the spin feature and conduct tests on this feature.
Debrina’s Status Report for April 20, 2024
This week I am on schedule. On the project management side, this week, we’ve spent a decent amount of time planning the final presentation that is scheduled to take place on April 22nd. Furthermore, we’ve worked on making our system more robust and prepared a demo video to present in the final presentation.
In terms of progress related to our system, I continued running tests on our system to continue identifying and resolving issues that would cause our system to crash on some edge cases. On the structural side, this week I also installed lighting to our environment and created a permanent mount for our projector and camera, but left it possible to adjust their vertical position so that we can still adjust the camera and projector as we debug. A big issue this week was calibration of the projector. Our original plan was to manually align the projector to the table by modifying the projector’s zoom ratio and position after the table image is made full screen on our laptop screen. However, there are some limitations to this since the projector’s settings are not as flexible as we had hoped. Hence, another solution we tried out this week to speed up the calibration process is to stream the video from our camera on flask, where we would have more freedom in adjusting the zoom ratio and position of the video feed and hence be able to better align it with the table. This is still an ongoing effort that I will continue to work on in the coming week.
Another big focus for this week was on making further improvements to the ball detections. In order to have more stable ball detections, I implemented a state detection algorithm that would detect whether the positions of the balls on the table have shifted, in which case the ball detections would be re-run and a new set of ball locations would be passed into the physics model for computations. Currently, the state changes are based on the position of the cue ball. Hence, if a different ball moves, but it isn’t the cue ball, the ball detections would not be updated. This is a difficult issue to solve, however, as it would be computationally intensive to match the different balls across different ball detections. I will be working on an improvement for this limitation in the coming week to make the algorithm more robust.
Besides the projector calibration and improving the ball state detection, in the coming week I also plan to continue conducting tests and fix any bugs that may still remain in the backend.
As you’ve designed, implemented and debugged your project, what new tools or new knowledge did you find it necessary to learn to be able to accomplish these tasks? What learning strategies did you use to acquire this new knowledge?
On the technical side, I learned a lot about computer vision, Python development, and object oriented programming in Python. I tried out a lot of different tools from the OpenCV library in order to implement the object detections since a lot of the times they yielded inaccurate results. I had to research different methods then implement them and compare their results with each other to determine which ones would be more accurate and under what lighting / image conditions they would be more accurate. Sometimes the meaning of the different parameters used in the different methods were not very clear, and in order to learn more about these I would either experiment with different parameters and inspect the changes they made, or I would consult blogs that discussed a sample implementation of the tool in the author’s own project. In terms of Python development, I learned a lot about API design and object oriented programming. Each detection model that I implemented was a class that contained certain parameters that could be used to keep track of historical data and be used to return detections based on these historical data. I also tried to standardize the APIs to ease our integration efforts. Furthermore, since our system would be a long-running process, I focused on implementing algorithms with lower complexity in order to allow our backend to run faster. All the learning that I had done was mainly done through trial and error, experimenting with the tools and seeing the behavior of the implementation, and reading documentation and implementation examples to fix bugs or modify the necessary parameters used by the different tools.
Andrew’s Status Report for April 06, 2024
This week I modified our cue stick subsystem to use AprilTags. This turned out to affect the accuracy both positively and negatively. It provided a positive effect in that the cue stick detection itself was more accurate and much more consistent. However, the negative part was that the large AprilTags made our other computer vision subsystems behave unexpectedly. The most detrimental was that our cue ball detection subsystem started to mistake the AprilTags themselves occasionally as cue balls. Additionally, in order for the cue stick to be detected, this now required both AprilTags to be within frame. However, for some edge-case shots near the pool table walls, this would not work. As such, we decided to revert back to the previous polygon approximation method for now, and I am working on relying more on color detection for the cue stick detection. The idea is to use some sort of very bright, unusual color as an indicator for the cue stick subsystem to pick up on (something like bright pink).
Since our schedule is currently focused on integration, I am not too much behind schedule. I caught a pretty bad case of food poisoning Tuesday night and could not do much work until the weekend. However, I’m using the time now to catch up as well as work on newer tasks. In the coming week, I will be looking into integrating ball spin from our web application input into our physics engine + backend.
For verification, the most important component/subsystem I need to verify and improve is the cue stick detection system. The tests I’m planning to run for this subsystem is fairly straightforward: they consist of various frames/video taking various shots as well as lining up a variety of different shots. Verifying the subsystem is not difficult – I look frame-by-frame, output the detected cue stick, and verify it visually. To verify these results, I also added unit tests to the subsystem verifying smaller functional components required to make the entire subsystem operate successfully. This was also repeated for parts of the calibration subsystem as well.
Team Status Report for April 6, 2024
This week, our team is on schedule. We had our initial project demo and received some helpful feedback regarding some improvements we could make in our system. Furthermore, our team also managed to get the projection running and output the trajectories on the table. The main focus of this week, however, was the testing and continued efforts for the full integration of our system. We continued to run tests and simulate different cases to identify any that may yield undesired behavior in our system. This week we also planned out a more formal approach to testing and verification of our system that is more rigorous and aligns well with our initial design plans. These plans will be outlined below.
The most significant risks we face now would be if our trajectory predictions are not as accurate as we hope for them to be. If this were to happen, it is likely that the inaccuracies are a result of our ‘perfect’ assumptions when implementing our physics model. Hence, we would need to modify our physics model to account for some imperfections such as imperfect collisions or losses due to friction.
This week, we have not made any modifications to our system’s existing design.
In the following week, we plan to conduct more rigorous testing of our system. Furthermore, we plan to do some woodworking to create a permanent mount for our camera and projector after we do some manual calibration to determine an ideal spot for their position.
The following is our outlined plan for testing:
Latency
For testing the latency, we plan on timing the code execution for each frame programmatically. We will time the code by logging the begin and end times of each iteration of our backend loop using Python’s time module. As soon as a frame comes in from the camera via OpenCV, we will log the time returned by time.clock into a log. When the frame finishes processing, we will again take the time when the predictions are generated and append this to the log. We intend to keep this running while we test out different possible cases for collisions (ball to ball, ball to wall, and ball to pocket). Next, we will find the difference (in milliseconds) between the start and end times of each iteration. Our use case requirement regarding latency was that end-to-end processing would be within 100ms. By ensuring that each individual frame takes less than 100ms to be processed through the entire software pipeline, we ensure that the user perceives changes to the projected predictions within 100ms which has the appearance of the predictions being instantaneous.
Trajectory Prediction Accuracy
For testing the trajectory prediction accuracy, we plan to manually verify this. The testing plan is as follows: we will take 10 shots straight-on, and we will use the mounted camera to record the shots. In order to ensure that the shots are straight, we will be using a pool cue mount to hold the stick steady. Then, we will manually look through the footage and trace out the actual trajectory of each ball struck and compare it to the predicted trajectory. The angle between these two trajectories will be the deviation. Angle deviation from all 10 shots will be averaged, and this will be our final value to compare against the use-case specification of <= 2 degrees error.
Object Detection Accuracy
For object detection accuracy, the test we have planned will measure the offset between the projected object and the real object on the pool table. Since the entire image will be scaled the same, we will use projections of the pool balls in order to test the object detection accuracy. The test will be conducted as follows: we will set up the pool table with multiple ball configurations, then take in a frame and run our computer vision system on it. Then, we will project the modified image with objects detected over the actual pool table. We will measure the distance between the real pool balls and projected pool balls (relative to each ball’s center) by taking a picture and measuring the difference manually. This will ensure that our object detection model is accurate and correctly captures the state of the pool table.
Debrina’s Status Report for April 6, 2024
This week I am on schedule. This week I resolved a few bugs in the physics model related to an edge case where a ball collision occurs near a wall. This caused the predicted ball location to be out of bounds (beyond the pool table walls). This was resolved using some extra casing on bounds checks. Furthermore I have been working on the ball detections and coming up with potential methods to distinguish between solids and stripes. For solid and stripe detection, I plan to use the colors of the balls in order to classify them. I took some inspiration from the group from a few semesters ago that did a similar project. These tasks are still works in progress which I intend to continue in the coming week.
This week our team also did some testing of our system. Although the system I worked on most was the object detection model, I helped Tjun Jet do some testing on the physics model. The testing we have done this week was rather informal as our goal was to identify edge cases in our physics model. To do this, we ran our system and placed the cue stick in multiple angles and orientations in order to ensure that each orientation and direction would allow for correct predicted trajectories. We wanted to identify any cases that would cause our backend to crash or produce incorrect outcomes. In coming weeks we will come up with more potential edge cases and do some more rigorous testing of the integrated system.
The testing I have done for my individual subsystem is evaluating the accuracy of the ball detections. So far, I have done informal evaluations on the accuracy of the detections in order to determine the ideal lighting conditions required for the model to achieve its best performance in terms of ball detection accuracy. Furthermore, I have evaluated the different cases in which some balls may fail to be detected. To detect these issues, I tried different positions of the balls and observed whether the detection model would perceive them properly (when the detection model runs, it prints out circles around the ball that it detects using cv2’s imshow method). Some cases I tested, for example, were placing several balls side by side; placing balls near the pocket; and placing balls against the pool table’s walls. These tests were done mainly to ensure the proper basic functionality of the detection model.
In the following week, I will conduct tests that are more focused on the accuracy of the ball detections under different possible layouts. Furthermore, I will be testing the accuracy of the pocket locations and the wall detection. For the ball detection tests, the cases I will focus on are:
- When balls are right next to each other (this may make the model have trouble detecting them as circles)
- When balls are right against the walls
- When balls are next to the pocket
- When balls are inside the pocket (ensure they are not detected)
- When the cue ball is missing (ensure that no other ball is perceived to be a cue ball)
In order to test the accuracy of the ball detections in these different conditions, I will be projecting the detections onto the table and taking measurements of the difference between the ball’s centers and the centers of the detections. In our design proposal’s use case requirements, we specified a maximum error of 0.2in.
The procedure for testing the accuracy of the pocket detection will be similar to that of the balls.
In order to test the accuracy of the wall detections, I will also project the detections on the table and measure the distance between the endpoints of each wall to the edge of the actual wall. For this, we are also aiming for a maximum error of 0.2in.
These tests will allow me to evaluate the level of accuracy of our detection models, which will have an impact on the accuracy of our final trajectory predictions.
Tjun Jet’s Status Report for April 6, 2024
This week, I continued improving the accuracy and fine tuning the physics model. I also helped out with some of the testing and verification methods for projecting the ball onto the table to verify its accuracy.
When projecting the output line onto the screen, I realized that there was a flaw in my wall detection model – I was not accounting for the ball radius when accounting for the ball bouncing back against the wall. The reflection against the wall assumed that the ball center was performing a reflection against the wall, where in fact, it should have been the surface of the ball. To rectify this, I shifted all the walls towards the center of the table by the cue ball’s radius and re-performed more accurate calculations from there. This returned the correct location that the cue ball was bouncing off the wall.
This week, I also began making preparations for the verification phase. I wrote code to project the ball’s predicted location onto the screen. Our eventual goal later on is to verify the predicted location with the actual location that the cue ball ends up hitting another ball. We will measure this difference and this should give us an idea of how accurate our prediction model is. This can be seen in the photo below:
In total, the physics subsystem will need to verify two things. Firstly, we will have to make sure the distance between the predicted ball center and the actual ball center of the cue ball hitting the first ball is off by less than 0.2inches. The next thing that we will have to measure is that the angular difference between the predicted ball trajectory and the actual trajectory is less than 2 degrees. In order to perform these two verifications, we intend to execute 10 shots and take a recording of these shots. We will then verify from the recordings of the actual location of the cue ball hitting one of the other balls. We will manually measure the difference from the video and scale it according to the table’s size to get the actual distance error. To verify the angular difference, we will use a ball tracking software that shows the difference in the angle of their trajectories. We will calculate the average error across the ten shots.
This week, as our trajectory output has not been fully stable yet, I only visually inspected that the ball is moving along the correct trajectory by taking a couple of shots. We also took some videos to visually verify that the ball should at least be hitting close to the location that we predicted it to be. Once we improve the stability of our ball and trajectory outputs, we intend to conduct a second round of verification testing that is more accurate as described in the paragraph above.
Having completed the first round of visual testing this week, I am currently on track for whatever I wanted to accomplish. I am still not fully satisfied by the instability of the ball prediction output trajectory line, but our team is working very hard to find ways to make it as stable as possible. During our team meeting tomorrow, we will focus our energy on verifying the accuracy of our predictions, as well as ensuring that our predictions are at the accurate location. This will require the precise alignment between our projector and the camera.
Team Status Report for March 30, 2024
The most significant risks that could jeopardize the success of the project is really only inconsistent CV detection (usually varying lighting conditions). Since we don’t use machine learning, a lot of our computer vision detection uses more “deterministic” computer vision algorithms like HoughLines, contour detection, polygon approximation, etc. In one of the previous days we were testing, there was a lot of sunlight in HHA104 which confused the cue stick detection model as well as the cue ball detection model. This poses a serious potential issue with our system. To manage the risk for the cue stick detection, we were thinking of just resorting to attaching two AprilTags onto the cue stick. We already have AprilTag detection infrastructure integrated into our system, so this would not be a hard change at all. Our only hesitation is that using AprilTags for the cue stick would result in a poorer user experience, since the user would have to hold the cue stick such that the AprilTags are always facing up. Worst comes to worst though, we would go with this option. As for the cue ball detection issue, this really only is an issue with the yellow ball and cue ball. We have two contingency plan options: color the yellow ball a darker shade of yellow, or make the “cue ball” whichever ball the cue stick is pointing at. The second option does not align as well with how pool is actually played, but since this is more of a training system, enforcing the actual rules of pool isn’t as necessary for our use case.
There were some minor changes made to the existing design of the system. For one, the Flask backend had to be integrated with the primary backend; this just made it a lot easier to both do the image processing/physics computation as well as stream it to our frontend. Additionally, this is really helpful in debugging the project. The change does not incur much cost really, only a small part of the backend has to be rewritten. Another change is having a “Calibration” class. This helps us calibrate our backend upon the system starting. This involves things like detecting the walls and pockets, cropping each frame to avoid noise. We also implemented a sort of “memory” for the system. There is a Calibration Python class we created that stores data from previous frames. If we aren’t able to detect certain objects or have some inconsistencies frame-to-frame, we use this class to either take the union of all detected objects in the past n frames (not detecting enough objects per frame), or we find the “median” objects (inconsistent objects frame-to-frame).