Team B9: BreakTime

This is the final blog post of our project, containing the link to our Final Report.

Harry

I worked with Christina to tune values and work on integration with our HSV filtering and Edge detection algorithms. I worked with the team to collect our testing results for the final presentation, and I am now working to increase stability and reduce variance for our balls and cue stick for the final demo. Christina and I are working on really fine-tuning the cue stick detection.

Christina

I helped collect the CV testing data for our final presentation, and I spent time understanding the full project and rehearsing my Wednesday presentation. Prior to the final demo, I am mostly working on improving the consistency of our CV’s readings for a smooth user experience. Harry and I are working on getting the cue stick angle and position to be as accurate as possible.

Sam

To help collect test data, I built a physical testing setup to measure our software backend metrics. This helps us compare our software prediction with the true pool table physics. Additionally, Tamal suggested that we incorporate cue stick speed, so I created a cue stick speed module. I tuned the module and included PyGame functionality so that there is useful projector output of the cue stick speed.

Team

We worked a lot on metrics and gathering testing results for our final presentation. We ran tests on our CV component, measuring the difference between the physical and CV-computed locations of the pool balls. For the software backend, we created a testing setup for different angle shots and recreated that setup in the software. From this, we compared the actual ball angle to the software-predicted ball angle. We also created our end-to-end test suite and compared the location differences. After all of these tests, we all worked to create our final presentation and supported Christina’s talk.

Now, we are working on getting our project ready for the final demo. We want our system to be very consistent, function correctly in the gym demo space, and have an intuitive and usable interface for our users. Wish us luck in these final days!

Harry

I finished up on my modules. I worked with Sam and Christina to tune values and work on integration, but the week was also extremely busy for me with a series of cascading deadlines. I’m looking into other modules to increase stability and reduce variance for the line side of things, but at the same time our priority is metrics gathering. I’m looking into accurate and reproducible methods of testing our CV separately, our GUI and software separately, and our system end to end.

Christina

I finished up on edge detection and implemented speed detection. I’m focusing on integration with Sam and Harry for the rest of the system. I’m working with Sam to tune our GUI size and orientation to match the pool table. It’s taking a significant amount of time to adjust the constants to match parts of the table with our camera feed. I also tuned our cue stick detection algorithm, we attached red tape to the end of the cue stick to have it better recognized, and cropped the frame to reduce conflicting lines. We’re able to have a much more stable image than our earlier demo.

Sam

I finished up the additional bounces, and added the speed to our GUI. I’m focusing more on integration with the rest of our system, and adjusting ball offsets to match our Pygame output to the table. It’s taking a lot of time adding in these offsets, we reason that this is because the projector and camera are curved and detect parts of our table different. In addition, we switched to a more minimalistic GUI to prevent further projector feedback. I’m looking forward to finishing tuning to proceed to metrics gathering.

Team

Our week was divided into pre-demo and post-demo. For pre-demo, we focused on integrating our changes into our complete system. We completed our cue-stick speed detection and GUI representation, but chose not to display it due to the need to tune it further. For our demo day, we found that the brightness from the projector was heavily impacting our edge detection threshold. We discussed with Professor Tamal and Shraddha, and decided to implement transparent images from our projector. Post-demo, our focus now is stabilizing our setup for minimize setup time. We were able to acquire a table and align our table to the corner of our frame. We are getting more accurate results, and are working hard to measure our metrics for our presentation and poster!

Harry

I worked on helping make our CV pipeline more robust. I built an aggregation-tool that helped reduce jitter through our HSV and Edge Detection algorithm outputs. The tool takes statistical averages of the data to exclude outliers, and give the user a smoother output. Perhaps in the future we could use simple machine learning techniques to smooth this out, but currently our statistics techniques are paying off. In addition, I added a calibration mechanism for our HSV input. This should save significant time setting up for various demo environments.

Christina

I worked on using edge detection as an alternative method to detect balls. While before, we found edge detection to be inaccurate, we looking at dramatically changing the threshold. We’re also looking at combining the HSV detection as a method of getting data from two sensor inputs. In addition, I looked into using Logitech Gaming Software as a method of tuning for brightness, color and saturation. Surprisingly enough, the auto-focusing feature of our camera is what was causing our significant jitter – the camera was zooming in and out. With this software, I was able to create a more stable image and tune it manually.

Sam

I worked on extending our ball prediction technique. Using the physics engine I built before, I added functionality for further bank shots (1, 2 and 3 bounces), and predicting other ball bounces. In addition, I’m predicting other ball bounces. I’m also starting on our ball speed detection showing up in the GUI. To start off, I’m showing a predicted end point of the ball to help the user see the result of their shots.

Team

As a team, we finished up implementing our features in anticipation for the next milestone demo. We spent a majority of time improving on our trouble areas, which were specifically jitter on the ball detection. We focused on building redundancy into all our components. This put our original predicted schedule behind by a week as we focused on accuracy, but we are confident we can finalize ball speed detection as our final feature. We’re looking at a strong demo that shows off our accuracy and core features!

TEAM

This short-week (because of Carnival), we all came together to make an alternative image perception algorithm worked. We employed and tested an edge detection algorithm with circle contours to detect the different pool balls. Additionally, we restructured how the camera would be attached to the frame so that the camera position is independent of the projector. We are experimenting with different combinations of pool ball detection to find the method with the most accuracy. Only 3 weeks left — let’s make it count! 😇

Christina

During the week, I spent a lot of time preparing the Computer Vision algorithms for the demo. I got around to testing with the demo camera and demo setup, and I noticed a lot of discrepancies compared to the pictures I originally tested on. We also noticed the lighting conditions really affected the CV, and this meant that conditions in the lab varied at night and during the day. With the help of my colleagues, since nighttime testing did not work well, they accompanied me early in the morning on demo day to help with some final tuning. With the feedback from the demo, we plan on developing a calibration system so that we do not have to always tune for a specific lighting condition and pool table location. Thanks team for all the help and late nights/early mornings!

Harry

This week, I assisted my teammates in their calibration efforts. I worked on helping groom our schedule to keep our tasks up-to-date. In addition, I re-evaluated and prepared the new tasks that we can do in the time between now and the demo. This was necessary since we noticed that testing and tuning was taking significantly longer than expected. I drafted pros and cons of each of the tasks, and hosted a quick session to re-evaluate our goals. In addition, I’m expanding on the CV component by adding simple edge detecting to increase the predictability of the ball detecting. I’m hoping to integrate this edge detection with our HSV and contour detection in order to achieve a high reliability of object detection.

Sam

This week, I mostly spent my time in the lab, providing positive encouragement and reinforcement to the CV developer(s) of our team. They had a relatively long week, sacrificing their health to work on this capstone project! I wish I could have done more to directly help on CV, but all I could do was be supportive. 🥪🍊 I mapped out my future implementation plans — in terms of MVP, I believe I am caught up, but further discussion needs to be done for our team’s plan moving forward.

 

TEAM

We spent a lot of time integrating for our demo. A few areas that needed work before we could demo were synchronizing the normalization between our CV component and our software component. Mainly, adjusting the values in the software backend to match the real life pool table dimensions and matching CV coordinates to software coordinates. In addition, we discussed our plans for the next few weeks. In a nutshell, we decided to implement features that helped streamline our process such as color synchronization and projector-camera registration based on our feedback.

Harry

This week I focused on strengthening the frame and upgrading some of the hardware for the eventual demo. I cut our 2nd order of PVC and designed a stronger frame that should be much more stable. This second frame design reinforces our primary frame design at the base and wraps around the pool table. I’m preparing to cement our initial frame. In addition, I added mounting screws for a stable projector mount. I also put in a order for higher resolution camera and a camera mount (turns out 1080p is not the same through different cameras, for our test photos we were getting 12 MP, our original camera was reporting 7 MP, so we upgraded to a camera with 15 MP – Megapixel is relevant to image quality but not video quality).

Christina

This week, I tested my CV code with different camera types. Previously, I had been using my phone’s camera, which has a very high resolution and quality. With the previous camera, the CV algorithms were having a hard time with the low resolution, since the edges are not as clear, which made detecting the edges of the pool table difficult. An example result is below.

Harry has the same higher resolution camera that we are planning on purchasing, so I got to test and tune with that. This was much more better for detecting the pool table edges, and the results are shown below. Next week, I plan on tuning the ball detection so that we can have the camera CV part working in real time.

Sam

Last week, we started integration with Christina’s CV and Sam’s software. It was mostly a hacky mess, so this week, I took the time to refactor our code and make a new ‘main’ module to act as the main entrypoint. Currently blocked from making progress by a team member. 😒 It’s partly my fault, too… We win together, lose together, but it’s always better when we’re together. 🤗

TEAM

We picked up more parts from Home Depot! We had an enjoyable car ride there — I believe bonding moments are important for team chemistry and morale. Overall, we’re confident we can deliver a working MVP by the demo day. ✌️ We’re meeting tomorrow (Sunday) to fix some bugs we found after integration. We are optimistic about our project. Waka waka. 😉

Christina

Over break and the course of the week, I finished up the core code for the CV component. It is still buggy and needs some additional work for integration, but we can now detect the pool table edges and all the balls and cue sticks to an accurate degree. The perceived vs. actual location difference of the balls was within .3 inches, which is very close to our requirement of within .5 cm. This week, I worked on integrating the CV component with our Software Backend for a full project. A few issues that arose included outputting the coordinates in a scaled manner that the rest of the backend could use. In addition, I focused on testing the accuracy of the individual CV component. The CV component with our sample test cases proved fine, but when integrated with our test hardware proved less accurate. I narrowed this down to the megapixel resolution of our camera. I am exploring options of a lower camera height or a higher resolution camera as a means of gaining more accuracy for the rest of our software!

Harry

I worked on integrating the frame, projector, and camera with the rest of our software. I re-built our frame into a “n” shaped piece, and ordered more parts. We focused on minimizing wobble in our switch to this configuration. In addition, we explored using velcro strips versus zip ties as means of attaching our electronics. In addition, I am exploring different mounting options to get the most physical stability and consistency in the ability to reproduce our test cases. I’m looking at buying more parts to finalize a frame design before we can cement our pieces together to further increase stability.

Sam

I worked on integrating the Software Backend with our CV component. This involved translating the classes that Christina wrote (e.g. coordinates for the pool balls and points on the cue stick) into classes that my code could use. A few issues that popped up were the dependencies and including our modules. I focused on adapting my physics and pool modules into a global main runner. In addition, I adopted my interface to match with the CV outputs. I also looked at extracting and refining the ghost ball tracking as our first feature. I am looking at trimming my other components into further features for our demo. In the end, we got a somewhat working product at the end of our Friday meeting. 🙌

TEAM

This week, we focused on integrating the parts of our project together. We mounted the projector and the camera onto our PVC frame and ran our end to end simulations. We spent a decent amount of time working together in lab. In conclusion, we had to resolve several Python module dependencies and other integration-related software bugs. We are a bit behind schedule due to these issues with software integration. However, we aim to work hard and sprint next week to deliver for our demo! Choo-choo, no stopping us now!!! 🚂💨 So say Geronimo!

Christina

This past week, I collected more photograph data for my computer vision portion so that I can improve the algorithms over spring break. We split up our design doc work into parts, and I put my seldom-used writing skills to use for my sections. On Wednesday, I borrowed a car to make a trip to Home Depot with Harry to pick up our parts. 10′ pipes are really tall! We cut the pipes down into more manageable parts and got help sanding from the kind people in the Makerspace. We were able to prototype the frame build, though we will need to eventually adjust the height for the camera and projector. Next, I plan on improving the CV accuracy for our demo table so that we can start integrating everything 👍

Harry

Last week, I spent a significant portion of our time contributing to our design doc. I was personally responsible for the overarching block diagram, my part of the design trade studies, and the project management section. I aided my teammates in editing and contributing for other parts of the doc. For my specific part of the project, we made a trip to Home Depot and collected the PVC pipes and associated parts. I cut and assembled a rough frame in the lab. I held up the camera to the estimated frame height and have plans to make the frame taller. I hope to have the frame ready for a full integration when we get back from break!

Sam

This was a short week for me – I left Pittsburgh on Wednesday to start an early Spring Break. Relating to Capstone, I spent a bit of time cleaning up and refactoring the code I’ve done so far before resuming development. Most of the time this week was spent working on the design documentation with my team. Back home, I’m sharpening my pool skills at a local pool hall — can’t beat the price at $7/2 hours. 😜

TEAM

The design documentation took up a lot of our time, collectively. We had a few (very long) meetings: one to divide up work, another to sync-up, and finally one to merge all our work together and finally submit. Overall, we are satisfied with the design document we submitted despite some ambiguity. We think it will serve as a very solid draft for the final report we submit at the end of the semester. I think we went through a lot this week, but we got through it together. Team above all. 😉

Christina

This past week, I worked on the design presentation and report outline with the rest of the team. Our 40″ pool table arrived on Monday, so we finally had access to the table that we will use on demo day. We were able to take photos and accurately measure a random layout of objects of our demo table. I modified the computer vision object detection for this new pool table and ball setup. The object detection is measuring all of the balls needed for 9-ball pool, and I will work on improving the accuracy of the ball center detections. One major complication was the green pool ball against the green felt of the table, and I hope to continue testing this distinction especially as our lighting conditions change. In the next week, I hope to compare the physical measurements we took with a ruler to the CV’s perceived object locations. I will try to fine-tune the detection parameters to minimize the difference between the actual and perceived pool ball locations.

Harry

Last week, I was the presenter for our design presentation, so I spent a lot of time preparing, finalizing, and rehearsing the presentation. I started working on the design of our frame and how we will use the PVC parts to build our sturdy frame. I researched most of the hardware parts we will need for our camera and frame setup, and we have placed the orders for them. When the camera came in, I helped Christina integrate the CV program with the video camera input. Along with the others, I worked on the design slides and documentation for the report. In the next week, the parts will arrive at Home Depot, so I can start prototyping and building the frame setup.

Sam

This past Sunday, I accompanied the team at one of our most productive meetings. It was a very windy day but we found save haven in one of the giant auditoriums at Tepper. Harry proceeded to kill the presentation the next day — sighs, podiums, and all. Later in the week, our 40″ pool table arrived and I played around with it. After shooting a few balls and trying different types of shots, I realized that if we end up using the 40″ pool table for our final demo, the physics is much simpler. For starters, there is no ‘english’ (spin) really possible on the ball, which makes things simpler. Additionally, the ball has to be hit at a minimum speed, or the balls will ‘break’ like golf balls on a green, due to the manufacturing errors in the table (not a flat surface). All-in-all, learned a lot this week and hopefully can move forward. Next week, I will wrap up my code for the MVP, refactor, and possibly work on integration with CV.

TEAM

As a team, we polished up our design presentation. We met together the day beforehand to finalize all the details and clarify any remaining topics. Once we received the design presentation feedback, we had a candid discussion on what went well and the different ways we could improve our system design. We followed the presentation directions well and were very prepared for any potential questions on our design. However, there were places we should have added in more detail and explained our reasoning for our decisions.

After receiving presentation feedback, we met on Thursday to start drafting our design report. We began with a brainstorming session to enumerate the components and workflow of our system.

We then started outlining our report, adhering to the final project report template. We have already collaborated and discussed the major points for our paper, and we are on track to finish our report Monday.