Jimmy’s Status Report 2 (09/28)

This week I accomplished establishing a pipeline to interpreting the camera data and calculating the coordinates of where the gantry system needs to move. After some discussion and research, I also changed the decision to use the OAK-D Pro depth camera rather than the D435. I initially started by helping Josiah with running experiments to test whether a simple kinematic model would suffice to accurately estimate the landing trajectory. However, we figured that too small deviances in the measurement would result in very large estimation errors, so I had to do more research on other more complex models. My work this week for the camera system pipeline mainly revolved around literature review for the Kalman Filter to predict the trajectory of the ball based on it’s past movements. I also looked at hough circle or YOLO models as feature extraction based vs CNN based approaches to implementing ball tracking. 

I mainly accomplished what I set out to do last week, although I would like to get some preliminary implementation done with the CV system, even if it’s on a rudimentary camera with no depth information. I will also need to consult with more experienced sources in this field to finalize and recognize the trade offs between each of my implementations, perhaps faculty in field of robotics or computer vision. Next week I will mainly aim to work on a first implementation of the ball tracking mechanism whilst I wait for the camera to arrive, as I do not need any depth information for that to be functional.

Team Status Report 2 (09/28)

This week we started our trajectory testing, a proof of concept that we can estimate the ball trajectory from just the first few frames of our camera capture. To do this, we measured and drew up a grid system on a whiteboard, threw a ping pong ball in an arc, and used a phone camera to record and manually track the position of the ball through the first few frames. Then we used classic kinematics equations to predict the landing spot. This was a simplified experiment, as not only did we have to estimate position by hand, but it was also converted to a 2D plane. During the actual project, the camera will be doing the location pinpoint and there is also a third dimension, but both those factors could be removed as we were simply testing to see if the kinematic equations would be enough. 

Unfortunately we discovered that the simple kinematics equations ended up consistently overestimating the ball’s landing spot. We found a python model online that took air resistance into account for trajectory calculations, and with that model it gave a better prediction, but still not the best. Under ideal conditions, our estimate was around 6-7cm off the mark. Following the experiments with our physics model we looked into potentially implementing more complex prediction systems into our camera or KRIA system that will give us better results. Additionally, with further testing (automated), we can determine if our predictions are accurate or precise. If it proves to be more precise in that we overshoot frequently, we could explore a basic heuristic (such as a linear shift) that aligns our precision to the center of the target.

In other news, we also worked through creating our design presentation slides. Since we now knew more about each component, we also locked down more specifics about what hardware we would be using. The Luxonis OAK-D PRO camera was picked, the AMD KRIA KR260 was ordered and retrieved from ECE inventory, and we decided to go with the gantry system. Another change with the system was that we moved the CV from being on the KRIA into the camera, as we discovered the OAK-D PRO has those capabilities. This makes it easier to transfer data between camera and KRIA, as now the camera doesn’t have to send whole frames through, and can just send coordinates for the KRIA to do calculations.

Specifically for the KRIA board, I met with Varun the FPGA TA and he was super helpful in letting me know what I would need to work through the KRIA setup, and I sent out order requests to fulfill the missing wire connections I needed to operate the KRIA.

 

 

Week 2 Specific section: Part A written by Gordon, Part B written by Josiah Miggiani, Part C written by Jimmy

Part A: Public health, safety, or welfare.

Our project solution doesn’t really have an aspect for public health, safety, or welfare. The only potential aspect is a stretch, and that is how by catching the thrown ball and thus preventing it from rolling around in the room. This minimizes the chances that anyone slips/trips on a loose ball and minimizes the need to go lift up furniture to get stray balls. Again, this is a stretch. 

 

Our project solution doesn’t really apply for this part, because our project’s use case is simply not targeted towards public health, safety, or welfare. Our project focuses more on the social aspect. We are trying to solve a pretty specific problem so we’ve considered everything and public health, safety, or welfare is not applicable.

 

Part B: Social factors

Splash has limited social factors as motivation for our project. It can be postulated that because cup pong is a social game, splash is a social aid because it helps users improve at cup pong. I would expect Splash to be most relevant to college students or young adults who are likely the largest demographic of cup pong players. 

 

Part C: Economic Factors

With regards to production and distribution of our project, we hope that our product will be designed in a way so that it can be viably be reproduced either for potential sale or by similar enthusiasts. Our system is relatively modular and easy to produce once we finish the prototype, and our ball-catching system should be consistent enough such that our functionality is not only limited to the demo portion – it should ideally work for anyone following our instructions.

Josiah’s Status Report for 9/28

Accomplishments

This week I ran some preliminary real-world tests to determine whether basic projectile motion equations could accurately predict where a ping pong ball would land, given two very close time frames and positions. These tests were ran using our smartphones, slow-mo recording (240fps), a whiteboard gridded out with a marker for translating the ball’s position in the video to position in real life. We restricted the axes to only x and y, tossing the ball parallel to the whiteboard. While my calculations turned out to be off by 10s of centimeters, by adapting to a python algorithm that takes air resistance into account, that error came down to less than 7cm. An important takeaway is that because the timeframe between ball frames is so low, getting accurate ball positions is CRITICAL. A difference of a centimeter can make a massive difference in computed initial velocities.

Progress

Besides the testing, I whipped up a spreadsheet for our bill of materials for the project, and added the materials required for the xy-plotting gantry. The total cost came to around ~$150, and could come down further if there are materials already at CMU we can take advantage of.

Gordon’s Status Report for 9/28

This week I met with Varun the FPGA TA, and he helped me a lot in figuring out exactly what I needed to do for setting up the board and getting started. Specifically he pointed me to his github repository that contained a guide for setup. I briefly read through the guide, and as the KRIA board arrived from the ECE inventory, I shifted my attention to setting up the actual board. I had learned from Varun that there were quite a few wires and ports that I had to setup, so I looked into what came with the KRIA from inventory and determined that I needed to get a display port cable and microSD’s from amazon. The KRIA came with the power supply, ethernet cable, and micro-usb, and I could bring a USB mouse and keyboard from home. I sent those orders off, grabbed a lock from receiving, and chose a box in 1307 to store the KRIA overnight.

 

The rest of the week I was helping the team execute the trajectory testing, and writing the design review presentation slides. I was talking with Jimmy about the camera and KRIA connection, and specifically where the CV would be done. We had decided to use the OAK-D PRO camera, and it turns out the camera is capable of doing the CV, so the KRIA usage was adjusted in our project plan to not have the CV. This is a better decision choice, as now all the camera needs to do is to send the identified coordinates over, and the KRIA can handle the calculations. Since we still don’t know the full capabilities of the OAK-D and if it can actually do the CV, we still have the idea of doing CV on KRIA as backup. 

 

It took the combined effort of the three of us to setup and execute the trajectory testing, as we had to use a ruler to mark up a white board grid, capture video, and parse through it to get coordinate estimations. The results of that are explained in the team status post. 

 

For the design review, I took on the role of redoing our block diagram, adding colors and taking inspiration from the guidance post on canvas. I was pretty proud of how it turned out, and we ironed out a lot of the vague terms as now we had identified what hardware we are using. I also added similar style blocks to other sections of the presentation, and contributed greatly to the content of those slides. 

 

Next week, besides listening to presentations and giving feedback,  I am hoping to get a chance to receive the parts I ordered and test the setup and connection to the Vivado and Vitis software that I need for KRIA use. I need to figure out how the microSD’s work, and get the KRIA up and running.

Jimmy’s Status Report 9/21

Accomplishments 

This week I looked at the design of the camera / CV system. For the camera, I researched and decided between the Intel Realsense models, namely the D515, D435 and the OAK-D S2. Eventually, I decided to pursue the D435 as it had the most suitable specifications for our project needs: cost of component, good range of operation (3 meters), having depth and colour sensing capabilities, having an onboard IMU (inertia measurement unit) for velocity sensoring. Furthermore, I did more research on the various algorithms / CV processes that could be used to track the position of the ball. On top of this, I also helped Josiah look at some of the feasibility of the gantry system (Cartesian robot) versus using an omnidirectional robot. Our main concern with the omnidirectional robot was precision as it would be difficult to calibrate the wheels and motor. Furthermore, pinpointing a central reference point for the robot to return to after every catch would also cause precision issues. However, this robot would be easy to assemble as many kits already exist for assembly. On the other hand, the Cartesian robot / gantry system would be very difficult to assemble and would require CAD and mechanical experience, which we all don’t have. However, the robotics portion of this would be much simpler as the gantry system naturally operates on a cartesian grid.

 

Deliverables

Next week I will finalize the system of algorithms of which satisfies both accuracy and latency challenges which our project face. Firstly I will need to come up with a prototype of the algorithmic approach to calculating the Cartesian coordinates of where the ball will land, given all the sensor data that I have. Furthermore, I will have to take this implementation and experiment with the real time latency. Based on the feasibility of the latency times, I would need to either redesign the algorithm, or port the implementation over to the Kria board using if fpga component.

Team Status Report for 9/21

Week 1, Sep 21

 

We are still debating between using the cartesian gantry/robot or an omnidirectional robot for our receiving system. This is a big decision for the project as a big portion of the design rests on which type of robot we choose, so we took some more time to look into the pros and cons of both but this time on a more technical level. We thought through exactly what we would need for the gantry and the omnidirectional robot, looking up numerous guides online for how other people used both systems. Despite our research, we still could not make the big decision of committing to either one. We’re trying to make a decision soon, and build a small prototype of whichever model we choose and verify the feasibility. 

 

For the cartesian gantry, there are a number of existing guides available online, some more detailed than others, that detail hobbyist implementations of cartesian robots. However, when reducing the scope by affordability and complexity, a couple guides remain, which are both XY plotting/drawing robots. Assuming the bill of materials are correct for these guides and assembly doesn’t turn out to be horribly difficult, we would need to modify the existing arduino code for our purposes, which is heavily reliant on GRBL. Alternatively, if we can manually generate a g-code file that instructs the machine to move to position (X,Y), we can use the code as is.

 

For the omniwheel robot, there are two especially intriguing guides that show the construction of a 4 and 3 wheeled drawing robot respectively, both by the same creator. A cursory look at the attached arduino code suggests it would require less re-tooling than the XY drawing robot’s, as there are already functions for GOTO XY position written. This would involve interfacing the Kria so that it can communicate the landing position of the robot to the arduino, and then it can run this GOTO. To return the robot, run GOTO 0,0. This also begs the question of whether the robot can be controlled directly by the kria, or if we have to add an arduino to control the robotics.

 

In any case, I think it should be feasible to pursue one method, determine if it’s promising, and if not, return to the other robot. Both are within the budget.

 

Another avenue that we are considering to take for the robotics side of things is to obtain our microcontroller first (probably arduino) and mess around with that to ensure that the latency of integration will not be a huge issue. Getting our hands on the microcontroller portion of the receiving system early could be good, as if any unforeseen problems pop up we can solve sooner. It’s also important to check the latency requirements so we can make adjustments to other parts of our system if needed, and it’s easier to make adjustments earlier in the design process. 

 

Regarding the camera / CV system, we are deciding between the exact camera to use. The most significant risk in this part is that the camera system cannot accurately calculate the landing coordinate of the ball, therefore this week we are spending time to design and choose a system that can achieve this using the most appropriate array of sensors. A close second risk is regarding latency of getting the calculations out in real time. Therefore, more research was also done this week regarding the exact sensor requirements, the sensor output information we can acquire (e.g depth, velocity measurements) and combine that with the appropriate system of algorithms and CV models so we can actually calculate the ball’s landing coordinate position. Further considerations are made to reduce the total amount of latency of the algorithm portion so that we can optimize for anything computationally expensive to be offloaded to be implemented on the Kria board. 

 

We haven’t made any changes to the design of the system. However we did change our schedule, as our original gantt schedule did not include fall break and thanksgiving break.

Gordon’s Status Report for 9/21

Week 1, Sep 21

This week we mainly focused on finishing and presenting our proposal. We put a lot of effort into making the proposal presentation slides, and we decided that I would be the one to present it on Monday. Through the weekend I practiced presenting the slides numerous times, and became extra familiar with the slides and our project as a whole. Besides presenting, in class on Monday and Wednesday we listened and gave feedback to all the proposals, and it was very useful to see how other people are approaching their project, and gave us ideas on how we could improve ours.

After the proposal presentations, we gathered to brainstorm a big decision that we would need to make soon, and that is whether to use the gantry or the omnidirectional robot as our receiving system. We looked into how each system has been used in other projects, and tried to determine which one we could use. This was mainly Josiah’s delegation but we all chipped in since it was a big decision affecting everyone.

We also split off into our own areas and started the design and development process for each component. I booked a time with Varun, the FPGA specialist TA, to talk through how the KRIA integration would work. Prior to this meeting with Varun, I am planning to look into more details about exactly how the KRIA would need to be used, so I can arrive at the meeting with quantitative questions. Wrote up points that I need to look into about KRIA integration, and will be looking into it before I meet with Varun.

We are in the design phase of the schedule, and this is week 1 so the ball is just getting started to roll. We are on pace, doing a lot of research into feasibility and how to get started. For next week, I am hoping to figure out how to secure the KRIA KR260 board from Varun, as well as developing a plan to start working.

Josiah’s Status Report for 9/21

Accomplishments

This week, we completed our proposal presentation. For the purposes of the presentation, I came up with a number of strict and granular tests that range from unit tests of individual components to cumulative tests that are concerned with the entire system. I also beautified our website to give it a bit of cup pong flair. The emojis also help us stand out B)

 

Progress

I made great progress on the topics of viable robotics solutions for moving our cup to the landing location of the ball. There are a number of online guides that detail hobbyist-tier XY plotter drawing machines, or omniwheel drawing robots. I believe these existing guides can be repurposed for the simpler task of moving a robot to a specific location. I dug into the arduino code for the omniwheel robot, and there is existing infrastructure that would easily support our purposes. Next week, I hope to decide on one method to start with, and then order the materials necessary for construction of the robot. I believe progress is on track.

Introduction and Project Summary

Have you ever tried for a cup pong shot, only for the ball to bounce off the rim and skitter into the darkest, most inaccessible recesses of your living room? No longer! Splash is a computer vision-assisted robot that will aid individuals practicing their cup pong shots. Splash will track thrown ping pong balls and move the target cup to the ball’s projected landing location. We hope that our robot will be able to correct for inaccurate shots within a ~10cm radius of the cup as well as doing so within a typical ball flight time of ~1.0 seconds. Our mechanism will be comprised of a cup-moving robot (omnidirectional wheels OR cartesian gantry), a Kria board that runs the CV algorithms, and a depth camera for determining the ball’s position in the real world.