Posted on

Team Status Report for April 6, 2024

This week, our team is on schedule. We had our initial project demo and received some helpful feedback regarding some improvements we could make in our system. Furthermore, our team also managed to get the projection running and output the trajectories on the table. The main focus of this week, however, was the testing and continued efforts for the full integration of our system. We continued to run tests and simulate different cases to identify any that may yield undesired behavior in our system. This week we also planned out a more formal approach to testing and verification of our system that is more rigorous and aligns well with our initial design plans. These plans will be outlined below.

The most significant risks we face now would be if our trajectory predictions are not as accurate as we hope for them to be. If this were to happen, it is likely that the inaccuracies are a result of our ‘perfect’ assumptions when implementing our physics model. Hence, we would need to modify our physics model to account for some imperfections such as imperfect collisions or losses due to friction. 

This week, we have not made any modifications to our system’s existing design.

In the following week, we plan to conduct more rigorous testing of our system. Furthermore, we plan to do some woodworking to create a permanent mount for our camera and projector after we do some manual calibration to determine an ideal spot for their position. 

 

The following is our outlined plan for testing: 

Latency

For testing the latency, we plan on timing the code execution for each frame programmatically. We will time the code by logging the begin and end times of each iteration of our backend loop using Python’s time module. As soon as a frame comes in from the camera via OpenCV, we will log the time returned by time.clock into a log. When the frame finishes processing, we will again take the time when the predictions are generated and append this to the log. We intend to keep this running while we test out different possible cases for collisions (ball to ball, ball to wall, and ball to pocket). Next, we will find the difference (in milliseconds) between the start and end times of each iteration. Our use case requirement regarding latency was that end-to-end processing would be within 100ms. By ensuring that each individual frame takes less than 100ms to be processed through the entire software pipeline, we ensure that the user perceives changes to the projected predictions within 100ms which has the appearance of the predictions being instantaneous. 

Trajectory Prediction Accuracy

For testing the trajectory prediction accuracy, we plan to manually verify this. The testing plan is as follows: we will take 10 shots straight-on, and we will use the mounted camera to record the shots. In order to ensure that the shots are straight, we will be using a pool cue mount to hold the stick steady. Then, we will manually look through the footage and trace out the actual trajectory of each ball struck and compare it to the predicted trajectory. The angle between these two trajectories will be the deviation. Angle deviation from all 10 shots will be averaged, and this will be our final value to compare against the use-case specification of <= 2 degrees error.

Object Detection Accuracy

For object detection accuracy, the test we have planned will measure the offset between the projected object and the real object on the pool table. Since the entire image will be scaled the same, we will use projections of the pool balls in order to test the object detection accuracy. The test will be conducted as follows: we will set up the pool table with multiple ball configurations, then take in a frame and run our computer vision system on it. Then, we will project the modified image with objects detected over the actual pool table. We will measure the distance between the real pool balls and projected pool balls (relative to each ball’s center) by taking a picture and measuring the difference manually. This will ensure that our object detection model is accurate and correctly captures the state of the pool table.

Posted on

Debrina’s Status Report for April 6, 2024

This week I am on schedule. This week I resolved a few bugs in the physics model related to an edge case where a ball collision occurs near a wall. This caused the predicted ball location to be out of bounds (beyond the pool table walls). This was resolved using some extra casing on bounds checks. Furthermore I have been working on the ball detections and coming up with potential methods to distinguish between solids and stripes. For solid and stripe detection, I plan to use the colors of the balls in order to classify them. I took some inspiration from the group from a few semesters ago that did a similar project. These tasks are still works in progress which I intend to continue in the coming week. 

This week our team also did some testing of our system. Although the system I worked on most was the object detection model, I helped Tjun Jet do some testing on the physics model. The testing we have done this week was rather informal as our goal was to identify edge cases in our physics model. To do this, we ran our system and placed the cue stick in multiple angles and orientations in order to ensure that each orientation and direction would allow for correct predicted trajectories. We wanted to identify any cases that would cause our backend to crash or produce incorrect outcomes. In coming weeks we will come up with more potential edge cases and do some more rigorous testing of the integrated system. 

The testing I have done for my individual subsystem is evaluating the accuracy of the ball detections. So far, I have done informal evaluations on the accuracy of the detections in order to determine the ideal lighting conditions required for the model to achieve its best performance in terms of ball detection accuracy. Furthermore, I have evaluated the different cases in which some balls may fail to be detected. To detect these issues, I tried different positions of the balls and observed whether the detection model would perceive them properly (when the detection model runs, it prints out circles around the ball that it detects using cv2’s imshow method). Some cases I tested, for example, were placing several balls side by side; placing balls near the pocket; and placing balls against the pool table’s walls. These tests were done mainly to ensure the proper basic functionality of the detection model.

In the following week, I will conduct tests that are more focused on the accuracy of the ball detections under different possible layouts. Furthermore, I will be testing the accuracy of the pocket locations and the wall detection. For the ball detection tests, the cases I will focus on are:

  1. When balls are right next to each other (this may make the model have trouble detecting them as circles)
  2. When balls are right against the walls
  3. When balls are next to the pocket 
  4. When balls are inside the pocket (ensure they are not detected)
  5. When the cue ball is missing (ensure that no other ball is perceived to be a cue ball) 

In order to test the accuracy of the ball detections in these different conditions, I will be projecting the detections onto the table and taking measurements of the difference between the ball’s centers and the centers of the detections. In our design proposal’s use case requirements, we specified a maximum error of 0.2in. 

The procedure for testing the accuracy of the pocket detection will be similar to that of the balls. 

In order to test the accuracy of the wall detections, I will also project the detections on the table and measure the distance between the endpoints of each wall to the edge of the actual wall. For this, we are also aiming for a maximum error of 0.2in. 

These tests will allow me to evaluate the level of accuracy of our detection models, which will have an impact on the accuracy of our final trajectory predictions.

Posted on

Tjun Jet’s Status Report for April 6, 2024

This week, I continued improving the accuracy and fine tuning the physics model. I also helped out with some of the testing and verification methods for projecting the ball onto the table to verify its accuracy. 

When projecting the output line onto the screen, I realized that there was a flaw in my wall detection model – I was not accounting for the ball radius when accounting for the ball bouncing back against the wall. The reflection against the wall assumed that the ball center was performing a reflection against the wall, where in fact, it should have been the surface of the ball. To rectify this, I shifted all the walls towards the center of the table by the cue ball’s radius and re-performed more accurate calculations from there. This returned the correct location that the cue ball was bouncing off the wall. 

This week, I also began making preparations for the verification phase. I wrote code to project the ball’s predicted location onto the screen. Our eventual goal later on is to verify the predicted location with the actual location that the cue ball ends up hitting another ball. We will measure this difference and this should give us an idea of how accurate our prediction model is. This can be seen in the photo below: 

In total, the physics subsystem will need to verify two things. Firstly, we will have to make sure the distance between the predicted ball center and the actual ball center of the cue ball hitting the first ball is off by less than 0.2inches. The next thing that we will have to measure is that the angular difference between the predicted ball trajectory and the actual trajectory is less than 2 degrees. In order to perform these two verifications, we intend to execute 10 shots and take a recording of these shots. We will then verify from the recordings of the actual location of the cue ball hitting one of the other balls. We will manually measure the difference from the video and scale it according to the table’s size to get the actual distance error. To verify the angular difference, we will use a ball tracking software that shows the difference in the angle of their trajectories. We will calculate the average error across the ten shots. 

This week, as our trajectory output has not been fully stable yet, I only visually inspected that the ball is moving along the correct trajectory by taking a couple of shots. We also took some videos to visually verify that the ball should at least be hitting close to the location that we predicted it to be. Once we improve the stability of our ball and trajectory outputs, we intend to conduct a second round of verification testing that is more accurate as described in the paragraph above. 

Having completed the first round of visual testing this week, I am currently on track for whatever I wanted to accomplish. I am still not fully satisfied by the instability of the ball prediction output trajectory line, but our team is working very hard to find ways to make it as stable as possible. During our team meeting tomorrow, we will focus our energy on verifying the accuracy of our predictions, as well as ensuring that our predictions are at the accurate location. This will require the precise alignment between our projector and the camera. 

Posted on

Team Status Report for March 30, 2024

The most significant risks that could jeopardize the success of the project is really only inconsistent CV detection (usually varying lighting conditions). Since we don’t use machine learning, a lot of our computer vision detection uses more “deterministic” computer vision algorithms like HoughLines, contour detection, polygon approximation, etc. In one of the previous days we were testing, there was a lot of sunlight in HHA104 which confused the cue stick detection model as well as the cue ball detection model. This poses a serious potential issue with our system. To manage the risk for the cue stick detection, we were thinking of just resorting to attaching two AprilTags onto the cue stick. We already have AprilTag detection infrastructure integrated into our system, so this would not be a hard change at all. Our only hesitation is that using AprilTags for the cue stick would result in a poorer user experience, since the user would have to hold the cue stick such that the AprilTags are always facing up. Worst comes to worst though, we would go with this option. As for the cue ball detection issue, this really only is an issue with the yellow ball and cue ball. We have two contingency plan options: color the yellow ball a darker shade of yellow, or make the “cue ball” whichever ball the cue stick is pointing at. The second option does not align as well with how pool is actually played, but since this is more of a training system, enforcing the actual rules of pool isn’t as necessary for our use case.

There were some minor changes made to the existing design of the system. For one, the Flask backend had to be integrated with the primary backend; this just made it a lot easier to both do the image processing/physics computation as well as stream it to our frontend. Additionally, this is really helpful in debugging the project. The change does not incur much cost really, only a small part of the backend has to be rewritten. Another change is having a “Calibration” class. This helps us calibrate our backend upon the system starting. This involves things like detecting the walls and pockets, cropping each frame to avoid noise. We also implemented a sort of “memory” for the system. There is a Calibration Python class we created that stores data from previous frames. If we aren’t able to detect certain objects or have some inconsistencies frame-to-frame, we use this class to either take the union of all detected objects in the past n frames (not detecting enough objects per frame), or we find the “median” objects (inconsistent objects frame-to-frame).

Posted on

Andrew’s Status Report for March 30, 2024

This week was again integration and refining the cue stick detection module. For the integration, I merged our primary backend (CV detection, physics engine, etc) with the Flask server that streams the video frames to our frontend. Getting this to work was a little bit tricky, since the frames have to be streamed in real-time to the frontend. For the initial demo, I changed the cue stick detection module to use polygon approximation to detect the end of the cue stick and identify two points which represent the stick. It is not 100% accurate, but is serviceable for the demo. There were alternate ways to implement this; however, most of them were not consistent or do not detect the cue stick often enough for it to be real-time. Also, since we last discussed being able to do ball spin, I implemented a basic selection system on the frontend which lets the user select the kind of shot they want to take with various spin types.

This week I am on schedule. We are wrapping up the MVP of our project and will be ready to show at the interim demo. In the coming week, I hope to demo the project on Monday and continue to improve the cue stick detection. I’m also aiming to improve the frontend UI and incorporate spin into the backend.

Posted on

Debrina’s Status Report for March 30, 2024

This week our team continued our efforts to integrate the full system in order to have a smooth-running demo by Monday, April 1st. In terms of our schedule, I am on track. 

I continued running tests of our backend on the live video and solving bugs that emerged due to edge cases, and I mainly worked on improving the pocket detection algorithm in the physics model. First, I modified it to account for the direction of the trajectory. I also modified the trajectory output when a ball is directed towards a pocket. Previously, the trajectory drawn would be between the ball’s center and the pocket’s center. However, this leads to an inaccurate trajectory; instead it is more natural to extrapolate the trajectory from the ball’s center to the point on the pocket’s radius which it intersects. Lastly, I make sure to check how close to the center of the pocket the intersection between a trajectory and the pocket is. This is because an intersection with the pocket doesn’t necessarily mean that the ball will be potted. An example to consider would be if the trajectory is right on the edge of the pocket’s radius. Even though there is an intersection, the ball will not be potted as it will not fall into the pocket. Hence, in this case, the trajectory would instead collide with a wall. 

Aside from the physics model, I worked on the calibration of our system on startup. I added a method to calibrate the wall detection and pocket positioning using the first 50 frames taken on startup of our system. This is because the wall and pocket locations shouldn’t change much, so we shouldn’t need to re-detect these components on each frame taken by the video. I also added a small modification of our wall detection model to ensure that the left and right walls are to the left and right of the midpoint of the frame (similarly to ensure that our top and bottom walls are to the top and bottom of the frame). This helped inaccurate detection of our walls that caused them to be on the wrong side of the table on some frames. 

Lastly, I have been working on improving the ball detection to make it more accurate and reduce false positive detections. I am using a color mask to distinguish the balls from the table, which will allow the balls to contrast more with the table and eliminate false positives caused by subtle contour detections on the table. This is still a work in progress that I am to complete tomorrow (Sunday, March 31st).

In the coming week, I will continue running integration tests to solve any bugs that may arise as we try out more edge cases. I hope to help out with implementing the spin trajectory — a new feature of our system that we have decided to add and began to work on last week. This week I also ordered LED lights, so I hope to install them in the coming week and make sure that the lighting conditions improve our detection models.

Posted on

Tjun Jet’s Status Report for March 30, 2024

This week, we managed to successfully output the line of projection on the software in real time. This took up the majority of the week, but we finally managed to get the output line shown on the canvas. 

Many errors arose when I was trying to project the output line on the screen. Many edge cases were not considered when I was testing the output line projection on multiple different images, and this led to a lot of issues that came about in real time. One such example is accounting for the directionality of the line projection. For example, if we had a specific line that was going through two points, we couldn’t tell exactly which wall it was going to bounce off. The only way to tell which wall it was going to bounce off was to know which direction the cue stick was pointing to. Thus, we needed to find a method to find the point on the cue stick that was closest to the cue ball. By setting the dx and dy direction with the line, we were now able to figure out which way the projection should go, giving us the right projection. We can see an example image below:

The next step after this is to show this projection on the projector, which we will do so in our group meeting tomorrow. In order to project the output line, we will plot the output line on an empty black canvas. After some testing, we have realized that this was the best way for us to achieve a good line projection. Here is the picture of how we intend the projected line to look like (in the picture below, we haven’t projected the actual output line).

I am currently on track for whatever I wanted to accomplish. Over the week, I was definitely a little behind as I did not manage to get the output line in time. However, we managed to solve quite a bit of edge cases in the cue stick’s direction. Currently, our team’s biggest goal is to make sure that all our subsystems work flawlessly together. We will also need to divide and conquer to try solving the computer vision issues, and making sure that we accurately detect walls, balls, cue balls, and especially cue sticks for every frame. 

Posted on

Team Status Report for March 23, 2024

This week our team continued to work towards the full integration of our system. We continued conducting tests that would allow us to let our different modules interact with each other. We identified several edge cases that we did not account for in both our object detection models and physics models. This week, we worked towards accounting for these edge cases. Furthermore, we had great advancements in improving the accuracy of our cue stick and wall detection models. 

In the computer vision side of our project, our team made significant progress on cue stick detection and wall detection. Andrew spent a lot of time creating a much more accurate cue stick detection model after testing out different detection methods. Although the model was quite accurate in detecting the cue stick’s position and direction, there is an issue where the cue stick is not detected in every frame captured by our camera. This is an issue that Andrew will continue to address in the coming week as it is crucial that the cue stick data is always available for our physics model’s calculations. Tjun Jet was able to detect the walls on the table with much better accuracy using an image masking technique. This advancement is beneficial for the better accuracy of our trajectory predictions. 

The most significant risk for this project right now is we did not successfully get our output line out, which we intended to do this week. Initially, the output line seemed to be working as we tested it on a singular image. However, during integration with the video, we realized that there were a lot of edge cases that we did not consider. One factor we didn’t account for was the importance of the direction of the trajectory lines. In our backend, we represent lines using two points on the line; however, some of the functions need to be able to distinguish between the starting and ending point of the trajectory. Hence, this is an additional detail that we will standardize across our systems. Another edge case was ensuring that the pool stick’s cue tip had to lie within a certain distance with the cue ball in order for us to project the line. Further, the accuracy of the cue stick trajectory is heavily affected by the kind of grip one uses to hold the stick. There were also other edge cases we had to account for when calculating the ball’s reflection on the walls. Thus, a lot of our code had to be re-written to account for such edge cases, putting us slightly behind on schedule regarding the physics calculations. 

One change that was made for this week is that Debrina and Tjun Jet swapped roles. This change sparked as Debrina realized that she was not making much progress in improving the accuracy of the Computer Vision techniques, and Tjun Jet felt that it would be better to have a second pair of eyes to implement the physics model, as it was non-trivial. Thus, Tjun Jet searched for more accurate methods for wall detections and Debrina helped to find out what’s wrong with the physics output line. Another reason for this swap is because integration efforts have begun, and thus, roles are now slightly more overlapped. We also began working together and having more meetups. Thus, our group mates have ended up working together to solve each other’s problems more compared to before.

Updated timeline accounting for new cue ball spin detection feature and updated testing schedules.
Posted on

Andrew’s Status Report for March 23, 2024

This week I spent a massive amount of time debugging and testing, as well as integrating the entire system together. The debugging was specifically for the cue stick detection model. After rigorous testing I realized that there were a number of edge cases that the cue stick detection completely failed at. Most of these specifically depends on how the user grips the cue stick. Varying grips cover different parts of the cue stick that the cue stick detection model relies on to detect the stick, which makes it hard to have consistent results. Further, skin tone also affects the rate of detection, with skin tones that are closer to the color of the cue stick confusing the model more.

I cycled through a number of varying methods for detecting the cue stick – HoughLines, polygon approximation, background subtraction, and many others. Unfortunately, none of these approaches covered all edge cases by 100% and was not reliable enough. I also incorporated a form of “memory” for the cue stick detection system, which essentially takes the union of detections across multiple frames in order to make up for its inconsistencies. Despite this, many times we still failed to detect the cue stick. My progress this week is slightly behind since the cue stick detection is not working properly. There is an easier way to address this issue – attaching AprilTags on the cue stick. However, this is a last resort and as a team we collectively wanted to avoid doing this in order to have a better user experience. If we use AprilTags to detect the cue stick, then the user must hold the cue stick in a specific position for it to be detected. I will try to put in at least another half-week into finding a way to fix the cue stick detection without AprilTags.

In the coming week, I aim to fix the cue stick detection model and continue with end-to-end integration of the system.

Posted on

Tjun Jet’s Status Report for March 23, 2024

This week, our team continued integration efforts across the different subsystems. Our integration process was quite smooth at the beginning with the computer vision working for the various subsystems, but we ran into several edge cases for the physics implementation, which needed a re-implementation on some of our functions, which I will go through in further detail in this report. Furthermore, the wall detection did not seem to be very accurate, so I helped implement a new method to improve the accuracy of our wall detection. Finally, as we realized that we don’t have guaranteed detections of balls, walls, and pockets every frame, we decided to do a 30-frame average to fix the points, which I will also elaborate on later. 

Firstly, we realized that our wall reflection algorithm was not very accurate as it only accounted for walls that are completely horizontal or vertical. However, from our detection model, we realized that this would not always be the case, especially if our table is slanted relative to the camera. Hence, Debrina and I reimplemented the function that accounted for the reflected points on the wall. This used geometry to account for slanted walls as well. This has not been successfully completed yet, which we will hopefully complete during our meeting tomorrow.

Next, we realized that our wall detections were not completely accurate relative to the wall of the pool table. Hence, I explored a different method of wall detection that improved our detection method. This method first masked our pool table with a green color, which is the color of our pool table. Next, it did Canny Edge Detection to output the edges. This algorithm then filtered out the most horizontal and vertical lines from the walls. This returned the four walls in a more accurate manner, which we can see in the photo below. 

However, one problem we faced was that the detections were not present in every frame. This can be seen in the photo below. As the walls, pockets, and balls (at a stationary state) will not be changed throughout the course of the game, we decided to include a “calibrate” button to perform calibration. This means that we will take a 30-frame moving average and get the median detection of the walls. Since the walls are detected in most of the frames, this will allow us to find the wall, pockets, and ball detections whenever the user is not moving, instead of detecting every frame.

I am currently slightly behind schedule for whatever I wanted to accomplish. I was expecting the physics calculations to work and project the output line flawlessly, as I had got it to work on one image. However, I realized I did not account for a lot of the edge cases in terms of the cue stick’s orientation and the direction that it is pointed in. However, our team will be working together to try solving these issues together, especially within the next few days. Our team has been working really well together and consulting each other through ideating and solving each other’s problems, if any. Thus, we’re hoping to continue doing this and hopefully get the output line projected by wednesday this week, so that we can move forward in implementing our spin physics.