Posted on

Debrina’s Status Report for February 24, 2024

This week was a rather busy week, and I didn’t complete as much as I had hoped to. However, I will spend a lot of time on Sunday catching up with our timeline. 

In the coming week I will catch up on detection of solid vs striped balls and improve some of our object detection models to filter out the player when they are making a shot. I also plan to test our existing object detection models on a live video feed. Furthermore, I noticed that the wall detection model I had created the prior week was not very effective in detecting the walls on the images we took of our actual setup. This is most likely due to the lack of contrast between the walls of the table and the table’s playing area. To address this, I plan to install markers on our wall to distinguish the colors of the edges with the playing area, then I will retest the wall detection model on this new setup. 

Here is an image of the pocket detection:

Posted on

Andrew’s Status Report for February 24, 2024

This week the parts we ordered this week arrived. Since I’m in charge of the cue stick system, I integrated the IMU, Arduino Nano, and ESP32 WiFi module together which will all be attached to the cue stick directly. We will be using the IMU to gather data on both acceleration and orientation (pitch, roll, heave). For our first version, we are simply using the accelerometer to let the player quantify how fast/hard they are hitting and give them recommendations from that. Additionally, the gyroscopic data is currently only being used for telling the player whether they are holding the cue stick in the right orientation to hit the ball’s center. I wrote a first version of the IMU module which takes both this data and the gyroscopic data and streams it via the ESP32 module to the webapp that’ll be hosted on our laptop. If we have more time towards the end, we will look into integrating the gyroscopic data with the physics engine in order to account for more complex shots involving ball spin. Tjun Jet and I also looked into computing ball collisions in the physics engine and have a good idea of how the physics work for any general collision involving two balls now. I’d have liked to have more accomplished by this week, but I fell ill/still am a bit sick (as of writing this) and wasn’t able to accomplish much more. I’m hoping to be able to do more in the following week when I’m hopefully recovered.

Posted on

Tjun Jet’s Status Report for February 24, 2024

This week, I implemented the physics of the ball collisions and helped to build the pool table structure. Most of the week was spent reading and understanding the different physics libraries, and even playing pool games to see if the different calculations were right. I will go through some of the math involved in my physics implementation that we are using. 

Using the principles of conservation of momentum and conservation of kinetic energy, we realized that the trajectory of the balls upon collision will always move away at 90 degrees from each other. This assumes that the balls have equal mass and that no energy is lost to sound and heat, meaning it is an elastic collision. It turns out that most of the time, the effects of sound and heat are negligible, as we took ten videos that verified the balls moved away at 90 degrees. With that in mind, we used that to code the physics trajectory model.

 

Our group also met yesterday and spent quite a bit of time assembling the pool table. Initially, we realized that our shelves were slightly smaller than our pool table, but we managed to eventually find a way to unscrew some parts of the pool table and managed to assemble the pool table onto the shelf. We managed to also place both a phone camera and the projector on top, and took some videos to test our detection models. The arrival of these parts have become a significant step to helping us achieve our goal. 

I am slightly behind schedule for whatever I wanted to accomplish. I managed to implement the mathematics of the ball collisions, but I have not managed to show the outputs onto opencv. This is mainly because the ball and cue detection models that we have now are not that accurate, thus, it is difficult for me to simulate the physics trajectory with the ball. Hence, I spent a lot of time trying to correct the accuracy of these models and have not managed to render the trajectory images. However, I am planning to have a simulation image so that I can at least show the trajectory prediction is correct by tomorrow.

Here is an image of our completed frame.

 

 

Posted on

Team Status Report for February 17, 2024

This week we stayed on schedule for our software development goals. Since we were finalizing our parts orders this week, we delayed some tasks that were dependent on hardware (such as building the mount for our camera and conducting tests of our computer vision models with our camera). We also worked on developing a better-defined API to ensure that our different models could integrate seamlessly with each other. So far, we have not made any changes to the design of our system. 

This week’s focus was on implementing the software for our project. This consists of developing our computer vision models to detect different objects in our system (the pool table walls, pool cue, and the different cue balls); creating a physics model to simulate a ball’s trajectory based on its interaction with the pool cue, walls, or other balls; and detecting AprilTags for localization and calibration. 

Andrew focused on AprilTag detection to produce a coordinate frame to which our environment will be calibrated. Debrina used hough line transforms to detect the pool walls. Tjun Jet used open-cv’s findContours method to detect pool balls and the pool cue, and focused on researching different physics libraries and successfully modeled the trajectory of a ball when bounced on a wall. In the coming week we will work towards better integrating these models and improving their accuracy. Furthermore, we will begin creating a design to translate our output to the projector and build the frame for mounting our camera and projector. 

Currently, the most significant risks that could jeopardize the success of our project would be if the models we have developed are not as effective if implemented using a live video feed. When conducting the tests of our models on jpeg or png images, we are to yield accurate results. Testing our models’ performance and accuracy on a live video feed is one of our biggest next steps for the coming week. Regardless, we believe that if we can set up an environment that will provide consistent lighting and positioning, this would enable our models to be accurate after adjusting the relevant parameters.

Below, we outline our product’s implications on public welfare, social factors, and economic factors.

Part A (Written by: Debrina Angelica, dangelic)

CueTips addresses some crucial aspects of public health, safety, and welfare by promoting mental well-being and physical activity. Learning an 8-ball pool game can be challenging, especially for those looking to integrate into social circles quickly. By providing a tool that predicts and visualizes ball trajectories, our product alleviates the stress associated with the learning process, fostering a positive and supportive environment for individuals eager to join friend groups. Moreover, in our hectic and demanding lives, playing pool is intended to be a relaxing activity. Our solution enhances this aspect by reducing the learning curve, making the game more accessible and enjoyable. Furthermore, the interactive nature of the game encourages physical activity, promoting movement and coordination. This not only contributes to overall physical health, but also serves as a form of recreational exercise. Additionally, for older individuals, engaging in activities that stimulate cognitive functions is crucial in reducing the risk of illnesses like dementia. The strategic thinking and focus required in playing pool can contribute to maintaining mental acuity, providing a holistic approach to well-being across different age groups and skill levels. In essence, our product aligns with considerations of public health, safety, and welfare by fostering a positive learning experience, promoting relaxation, enhancing physical activity, and supporting cognitive health.

Part B (Written by: Andrew Gao, agao2)

In terms of social factors, our product solution relates in primarily three ways: responsiveness, accessibility, community building. Pool is a game that requires immediate feedback and a high level of responsiveness and skill. When a player aims at a ball, they instantaneously predict the trajectory of the ball on a conscious/subconscious level. Because of this requirement for instantaneous feedback, we need our solution to create predictions as fast as the human eye can process changes (< 150-200 ms). In order to keep the game engaging and responsive, our system must run at this speed or faster. In terms of accessibility, we want our product solution to reach a wide range of users. This motivates our choice in outputting visual trajectory predictions. Both beginner and expert pool players can understand, use, and train using our system. Our product solution achieves community by how people will use it. Pool is inherently a community-building game between multiple players. Our system helps people get better at pool and thus contributes to an activity which serves as a massive social interest in many parts of the world. 

Part C (Written by: Tjun Jet Ong, tjunjeto)

CueTips presents a unique and innovative solution that addresses a diverse range of users, catering to individuals spanning from beginners seeking to learn pool to seasoned professionals aiming to enhance their skills. Given that our target audience spans a broad range of different users, encompassing different age groups and skill levels, our product is inclusive and accessible for everyone, thereby creating room for substantial economic potential. To distribute our product, we could strategically partner with entertainment venues, gaming centers, and sports bars to target the demographic who are not only passionate about the game of pool, but also seeking interactive and technologically advanced gaming experiences to improve their game play. Simultaneously, we could also plan to offer these CV pool tables for direct-to-consumer sales, capitalizing on the growing market of individuals looking for home entertainment solutions. Through a combination of upfront hardware sales, subscription models, and potential collaborations with game developers for exclusive content, there is a lot of potential for our product to contribute significantly to the economy while providing an engaging and versatile pool gaming experience for users at all skill levels and ages. 

 

Here are some images related to our progress for this week:

Posted on

Debrina’s Status Report for February 17, 2024

This week I worked on developing a computer vision model to detect and distinguish between different objects on our pool table. Initially, I did a bit of exploration on different computer vision models online for detecting the pool balls. I looked through different implementations people have done online and tried running these locally to determine which method would have the best accuracy and low latency. 

My main focus this week was on detecting the four edges (walls) of the pool table. This data will be used in our physics calculations to predict the trajectory of a ball when it bounces on the wall. I decided to use Hough Line transforms to derive this data as it seemed to be the most effective in detecting straight lines. 

I got a decent amount of work done this week; however, there were some goals I didn’t meet. As I have become more familiar with open-cv this week, I am confident that I can finish these up in the coming week. Next week I plan to develop more models to be able to distinguish between striped and solid balls. Furthermore, I plan to make the wall detection model more robust as it currently is prone to some errors. I also plan to do some research and implement physics calculations to simulate interactions between different objects in our system (i.e. ball-to-ball or cue-to-ball interactions). I will also test out our current computer vision models on live video feed since I am currently running tests on static images. 

Posted on

Andrew’s Status Report for February 17, 2024

This week I finished up the AprilTag module. I modified an existing AprilTag library, extended it, and provided an API that detects AprilTags within an image and creates a coordinate plane with it. This allows the computer vision The AprilTag module is written in C/C++ and Python. I also wrote some unit tests, documented the usage of this AprilTag API, and handed it off to the rest of my team.

After this, I looked into the hardware components we need to assemble this system together. For detecting pool stick motion and player behavior, I decided on an Arduino Nano + 9DOF IMU + ESP32 module. This encapsulates the motion processing system. The Nano gets data from the IMU, then streams it to our laptop via the ESP32 module over WiFi. I also wrote some of the motion detection software for Arduino, though I don’t have a chance to test it on hardware until the parts arrive.

I also started writing code for our projection system (projecting our predictions onto the actual pool table). Currently, the predictions are outputted as lines drawn onto an image/frame of the pool table. I wrote out v1 of the projection system which currently just displays each frame live as the projection system outputs them. I’m going to improve on this by just displaying the trajectories and being able to automatically calibrate the projector by scaling the image size up and down depending on the AprilTag coordinates.

Posted on

Tjun Jet’s Status Report for February 17, 2024

This week I focused on implementing the ball and cue stick detections using contour line detection using OpenCV. I also managed to create some functions to extrapolate line detections from the cue stick and output the reflections with the walls of the pool table. 

Using cv2.findContours() and cv2.minEnclosingCircles(), I managed to get a good detection of the cue balls and the cue stick. I then wrote a function to find the center and radius of each of the contours, and used that to distinguish between the cue stick and the balls. The accuracy of my contour detection was not the best, but it was enough for me to do a preliminary testing of my physics model. Some images can be found below. 

Debrina managed to successfully output the wall detections of the cue table using cv2.HoughLines. Using her output of the wall detections, I tried finding the intersection point between the cue stick’s trajectory through an extrapolation line function that Iwrote and output the predicted reflected trajectory against the pool table walls. The following image shows how our reflections work.

In terms of ballcollisions, I am currently testing between two models: A git package called pool tool: https://github.com/ekiefl/pooltool, and also referencing a project that was done from a team from Cornell University called “Pool Projection”. I have yet to successfully output a good predicted trajectory from the ball collision model from pool trajectory, but once I am able to do so, most of the work will be to fine tune the accuracy of the model and I can now present the proper outputs for Andrew to output onto the projector. 

I am currently on schedule and accomplished whatever I wanted to accomplish this week. The most crucial part next week is to get the ball collision model working. Furthermore, now I am only testing my model on a single image. Once I get everything working, I aim to perform the work on real-time video and tweak my code and parameters such that the physics simulations can be done on the video. I also want to start testing the IMU and see if we can get some substantial compass data to improve the accuracy of our cue stick detection. 

Posted on

Tjun Jet’s Status Report for February 10, 2024

This week I mainly focused on setting up a Flask Server to stream our edge detection algorithm and also research on physics simulation methods to predict the trajectory of cue ball collisions.

The intent of using a Flask Server is because we want to be able to wirelessly communicate between our cameras and projectors. In order to do this, we will have to send a byte stream of data into our computer and output the image. The reason for this is to show the actual live stream of the camera feed into our web application, and also, it will help us with debugging when we are able to view the images on our computer. I successfully managed to set up the server to stream wirelessly between different computers, and the video feed looks very smooth when streamed wirelessly, with almost no lags.

In terms of physics simulations, I reference an online Git package called PoolTool: https://github.com/ekiefl/pooltool. There are some good resources on calculations of ball trajectories which I hope to take reference from and replicate for our use case. Once we are able to do so, it will be a good segway to integrate between the object detection algorithms and the physics trajectories.

I am currently on schedule and accomplished whatever I wanted to accomplish this week. Next week, I aim to actually implement the physics trajectory calculations on code. I am intending to pass in hard coded coordinates of circles to represent pool balls, as well as a hardcoded line that intersects with the cue ball. If we manage to get the object detection done, I want to use the object detection predictions as input to the algorithm, and see if we can get a real-time prediction of the physics trajectory. If that is not possible, I hope to be able to plot drawings of those images and see if I can output the trajectory.

Posted on

Andrew’s Status Report for February 10, 2024

This week, I was primarily focusing on figuring out the AprilTags for our pool table. We planned to use AprilTags on the four corners of the pool table for camera normalization, and we also want to use these AprilTags to construct a 2D coordinate system. We don’t have the AprilTags yet, but much of the software can be written first without having physical AprilTags.

I looked around different Python libraries and repositories and found a few resources for detecting AprilTags. After testing out a few of them and tinkering with the code, I managed to get AprilTag detection working for my laptop’s live video/camera:


Screenshot of  my laptop’s camera feed

It runs primarily on OpenCV but makes use of an open-source AprilTag detection library written in C by researchers from the University of Michigan. Their library also returns a list of metrics and coordinates, which I then process into a 2D coordinate plane with one of the AprilTags being the origin (0, 0). When we eventually port this onto a pool table, we should be able to easily create the coordinate plane.

I also did some research on transmitting data from the pool table camera to our video processing server. We decided on having a camera attached to a microcontroller with an ESP32 card. After some testing, we determined that transmitting frames over Wi-Fi via HTTP was sufficient.

Posted on

Team Status Report for February 10, 2024

This week, we met several times and managed to stay on top of schedule. We successfully implemented Canny Edge Detection to detect edges and timed the model. For each complete edge detection frame, we managed to achieve a latency of 12ms, which currently fits within our goal of ensuring a 100ms latency for the entire pipeline. We also successfully managed to carry out April Tag detection. Finally, we managed to stream our edge detection video frames across the WiFi network using a Flask Server, and we were pleased by how smooth the video feed is. 

The most significant risk of our project this week is hoping that we can order our materials in time. We believe that the most time consuming part of our project is to build a frame that can house both the projector and the camera. This is going to require a lot of effort to make sure that we have a stable camera for more accurate object detection. Given that we have not ordered our parts, it is currently the most significant risk to our project. 

Compared to what we shared in our proposal presentation, we made a change in our work allocations. Debrina will now be exploring object detection methods using computer vision, while Andrew will focus on researching wireless communications and hardware devices. Tjun Jet will continue to look into the physics simulations and trajectory calculations, and will also help to integrate AprilTag detection and EdgeDetection into one single backend application. 

No changes were made to the schedule this week. 

Here are some pictures that are there this week: