Team B9: BreakTime

Harry

I finished up on my modules. I worked with Sam and Christina to tune values and work on integration, but the week was also extremely busy for me with a series of cascading deadlines. I’m looking into other modules to increase stability and reduce variance for the line side of things, but at the same time our priority is metrics gathering. I’m looking into accurate and reproducible methods of testing our CV separately, our GUI and software separately, and our system end to end.

Christina

I finished up on edge detection and implemented speed detection. I’m focusing on integration with Sam and Harry for the rest of the system. I’m working with Sam to tune our GUI size and orientation to match the pool table. It’s taking a significant amount of time to adjust the constants to match parts of the table with our camera feed. I also tuned our cue stick detection algorithm, we attached red tape to the end of the cue stick to have it better recognized, and cropped the frame to reduce conflicting lines. We’re able to have a much more stable image than our earlier demo.

Sam

I finished up the additional bounces, and added the speed to our GUI. I’m focusing more on integration with the rest of our system, and adjusting ball offsets to match our Pygame output to the table. It’s taking a lot of time adding in these offsets, we reason that this is because the projector and camera are curved and detect parts of our table different. In addition, we switched to a more minimalistic GUI to prevent further projector feedback. I’m looking forward to finishing tuning to proceed to metrics gathering.

Team

Our week was divided into pre-demo and post-demo. For pre-demo, we focused on integrating our changes into our complete system. We completed our cue-stick speed detection and GUI representation, but chose not to display it due to the need to tune it further. For our demo day, we found that the brightness from the projector was heavily impacting our edge detection threshold. We discussed with Professor Tamal and Shraddha, and decided to implement transparent images from our projector. Post-demo, our focus now is stabilizing our setup for minimize setup time. We were able to acquire a table and align our table to the corner of our frame. We are getting more accurate results, and are working hard to measure our metrics for our presentation and poster!

Harry

I worked on helping make our CV pipeline more robust. I built an aggregation-tool that helped reduce jitter through our HSV and Edge Detection algorithm outputs. The tool takes statistical averages of the data to exclude outliers, and give the user a smoother output. Perhaps in the future we could use simple machine learning techniques to smooth this out, but currently our statistics techniques are paying off. In addition, I added a calibration mechanism for our HSV input. This should save significant time setting up for various demo environments.

Christina

I worked on using edge detection as an alternative method to detect balls. While before, we found edge detection to be inaccurate, we looking at dramatically changing the threshold. We’re also looking at combining the HSV detection as a method of getting data from two sensor inputs. In addition, I looked into using Logitech Gaming Software as a method of tuning for brightness, color and saturation. Surprisingly enough, the auto-focusing feature of our camera is what was causing our significant jitter – the camera was zooming in and out. With this software, I was able to create a more stable image and tune it manually.

Sam

I worked on extending our ball prediction technique. Using the physics engine I built before, I added functionality for further bank shots (1, 2 and 3 bounces), and predicting other ball bounces. In addition, I’m predicting other ball bounces. I’m also starting on our ball speed detection showing up in the GUI. To start off, I’m showing a predicted end point of the ball to help the user see the result of their shots.

Team

As a team, we finished up implementing our features in anticipation for the next milestone demo. We spent a majority of time improving on our trouble areas, which were specifically jitter on the ball detection. We focused on building redundancy into all our components. This put our original predicted schedule behind by a week as we focused on accuracy, but we are confident we can finalize ball speed detection as our final feature. We’re looking at a strong demo that shows off our accuracy and core features!

Christina

During the week, I spent a lot of time preparing the Computer Vision algorithms for the demo. I got around to testing with the demo camera and demo setup, and I noticed a lot of discrepancies compared to the pictures I originally tested on. We also noticed the lighting conditions really affected the CV, and this meant that conditions in the lab varied at night and during the day. With the help of my colleagues, since nighttime testing did not work well, they accompanied me early in the morning on demo day to help with some final tuning. With the feedback from the demo, we plan on developing a calibration system so that we do not have to always tune for a specific lighting condition and pool table location. Thanks team for all the help and late nights/early mornings!

Harry

This week, I assisted my teammates in their calibration efforts. I worked on helping groom our schedule to keep our tasks up-to-date. In addition, I re-evaluated and prepared the new tasks that we can do in the time between now and the demo. This was necessary since we noticed that testing and tuning was taking significantly longer than expected. I drafted pros and cons of each of the tasks, and hosted a quick session to re-evaluate our goals. In addition, I’m expanding on the CV component by adding simple edge detecting to increase the predictability of the ball detecting. I’m hoping to integrate this edge detection with our HSV and contour detection in order to achieve a high reliability of object detection.

Sam

This week, I mostly spent my time in the lab, providing positive encouragement and reinforcement to the CV developer(s) of our team. They had a relatively long week, sacrificing their health to work on this capstone project! I wish I could have done more to directly help on CV, but all I could do was be supportive. 🥪🍊 I mapped out my future implementation plans — in terms of MVP, I believe I am caught up, but further discussion needs to be done for our team’s plan moving forward.

 

TEAM

We spent a lot of time integrating for our demo. A few areas that needed work before we could demo were synchronizing the normalization between our CV component and our software component. Mainly, adjusting the values in the software backend to match the real life pool table dimensions and matching CV coordinates to software coordinates. In addition, we discussed our plans for the next few weeks. In a nutshell, we decided to implement features that helped streamline our process such as color synchronization and projector-camera registration based on our feedback.

Harry

This week I focused on strengthening the frame and upgrading some of the hardware for the eventual demo. I cut our 2nd order of PVC and designed a stronger frame that should be much more stable. This second frame design reinforces our primary frame design at the base and wraps around the pool table. I’m preparing to cement our initial frame. In addition, I added mounting screws for a stable projector mount. I also put in a order for higher resolution camera and a camera mount (turns out 1080p is not the same through different cameras, for our test photos we were getting 12 MP, our original camera was reporting 7 MP, so we upgraded to a camera with 15 MP – Megapixel is relevant to image quality but not video quality).

Christina

This week, I tested my CV code with different camera types. Previously, I had been using my phone’s camera, which has a very high resolution and quality. With the previous camera, the CV algorithms were having a hard time with the low resolution, since the edges are not as clear, which made detecting the edges of the pool table difficult. An example result is below.

Harry has the same higher resolution camera that we are planning on purchasing, so I got to test and tune with that. This was much more better for detecting the pool table edges, and the results are shown below. Next week, I plan on tuning the ball detection so that we can have the camera CV part working in real time.

Sam

Last week, we started integration with Christina’s CV and Sam’s software. It was mostly a hacky mess, so this week, I took the time to refactor our code and make a new ‘main’ module to act as the main entrypoint. Currently blocked from making progress by a team member. 😒 It’s partly my fault, too… We win together, lose together, but it’s always better when we’re together. 🤗

TEAM

We picked up more parts from Home Depot! We had an enjoyable car ride there — I believe bonding moments are important for team chemistry and morale. Overall, we’re confident we can deliver a working MVP by the demo day. ✌️ We’re meeting tomorrow (Sunday) to fix some bugs we found after integration. We are optimistic about our project. Waka waka. 😉

Christina

Over break and the course of the week, I finished up the core code for the CV component. It is still buggy and needs some additional work for integration, but we can now detect the pool table edges and all the balls and cue sticks to an accurate degree. The perceived vs. actual location difference of the balls was within .3 inches, which is very close to our requirement of within .5 cm. This week, I worked on integrating the CV component with our Software Backend for a full project. A few issues that arose included outputting the coordinates in a scaled manner that the rest of the backend could use. In addition, I focused on testing the accuracy of the individual CV component. The CV component with our sample test cases proved fine, but when integrated with our test hardware proved less accurate. I narrowed this down to the megapixel resolution of our camera. I am exploring options of a lower camera height or a higher resolution camera as a means of gaining more accuracy for the rest of our software!

Harry

I worked on integrating the frame, projector, and camera with the rest of our software. I re-built our frame into a “n” shaped piece, and ordered more parts. We focused on minimizing wobble in our switch to this configuration. In addition, we explored using velcro strips versus zip ties as means of attaching our electronics. In addition, I am exploring different mounting options to get the most physical stability and consistency in the ability to reproduce our test cases. I’m looking at buying more parts to finalize a frame design before we can cement our pieces together to further increase stability.

Sam

I worked on integrating the Software Backend with our CV component. This involved translating the classes that Christina wrote (e.g. coordinates for the pool balls and points on the cue stick) into classes that my code could use. A few issues that popped up were the dependencies and including our modules. I focused on adapting my physics and pool modules into a global main runner. In addition, I adopted my interface to match with the CV outputs. I also looked at extracting and refining the ghost ball tracking as our first feature. I am looking at trimming my other components into further features for our demo. In the end, we got a somewhat working product at the end of our Friday meeting. 🙌

TEAM

This week, we focused on integrating the parts of our project together. We mounted the projector and the camera onto our PVC frame and ran our end to end simulations. We spent a decent amount of time working together in lab. In conclusion, we had to resolve several Python module dependencies and other integration-related software bugs. We are a bit behind schedule due to these issues with software integration. However, we aim to work hard and sprint next week to deliver for our demo! Choo-choo, no stopping us now!!! 🚂💨 So say Geronimo!