Team A2: 🌊Splash💦

For the camera / estimation and tracking pipeline on Jimmy’s end, the decision was made to stop using the OpenCV kalman filter library, and ended up making a good in house implementation which yielded good results. This will be good to set us up as now the 2D visual pipeline is done, which can be a good contingency plan incase our 3D pipeline does not give good results. Some of the risks of this subsystem now lie entirely on the accuracy of the 3D model that is going to be implemented. As for validation, there will be two components to the test for validation for object detection and trajectory estimation. The first is to making sure that the estimates are accurately predicted and tracked. The second is to ensure that the calculation time can be done in a timely manner. The overall goal of the camera and detection/prediction pipeline is to generate a coordinate to send to the gantry so that it can catch the ball in time. Making sure of these conditions are met in the verification of this subsystem will ensure that our overall project goal can be achieved. 

For the Raspberry Pi hardware integration side, a lot of work was put into getting past the setup stage and having functional work be done on the Pi. Jimmy has been working on developing a good Kalman prediction code from a recorded video that he is able to pass into the code. Once I finalized the setup and finished installing the few dependencies, I was able to run the Camera on the Pi. The depth AI package that the camera came with also had many example files that could be run to showcase some of the camera’s capabilities. On Monday I focused on learning how the file system structure is set up and looked through many examples of the camera and what different pieces of code could make the camera do. The next step was to adapt Jimmy’s detection function to take in camera input for video instead of a recorded video. That was successfully done, but the resulting video was only 15 FPS, and it struggled to detect the ball. We had a few ideas of how to improve that, and the details of how we went about implementing that are elaborated more specifically in Gordon’s status report, but essentially we were able to identify that making the camera window smaller greatly increased the FPS, as shown in this table below.

Camera with 2D Ball Detection FPS Experiments

Window Size (All 1080 resolution, cant do worse)	Resulting FPS	Comments
camRgb.setVideoSize(1920, 1080)	13-15	Default screen size, FPS is way too slow for reliable ball detection. This is without any optimizations of code, which we are looking into
1200, 800	33-35	Ball detection is pretty smooth, need more integration to see if its enough
800, 800	46-50	At this point window size is probably as small as it should go, starts to inhibit ability to capture ball trajectory. Ball detection is super smooth however, a balance between screen size and FPS should be studied
700, 800	~55	^
700, 700	60	Achieved 60 FPS, but again window size is likely too small to be of use

The comments provide a baseline overview of what I concluded from this simple verification test. From what I’ve gathered, I believe that this framerate and window manipulation will be a good tactic that we can use to increase the precision and effectiveness of our tracking and trajectory prediction. There are even more tactics that we haven’t tried yet, and plan to test out in these last few weeks if any design requirements have still not been met. There is still a good amount of work to be done to add the third dimension into it, and to convert it to instructions for the XY robot. These last bits of integration we will work on as much as we can as a team, as we still need to figure out how to structure the whole system to give real world coordinates. In general I think our team has done a lot of verification within each of our own subsystems, and it is almost time to start the validation process of integrating all of them together. The RPI and Jimmy’s camera trajectory work are already relatively connected and working, but like aforementioned, connecting to the XY robot will require work from everyone. 

XY Robot:

As described in the design report, there is only a single, though important, test to be run. The big picture is that the robot is agile in motion and precise in positioning, such that it can support the rest of the system. Thus, we have the following validation test:

1. Supply a series of valid, arbitrary real-world coordinates to the Arduino running grbl firmware through G-Code serial communications. For each coordinate, validate that the robot:
   1. Moves the cup within <0.5cm of the position
   2. Moves the cup to the position within <1.0s

The following conditions should also be met:

• We define a valid and arbitrary coordinate to be any position within a 10cm XY radius around the origin of the cup.
  • Thus, the cup will only have to move AT MOST 10cm at a time.
• After each movement, allow the robot to reset to (0, 0).
• Ideally, configure the firmware to make the robot as fast as possible.
  • We want to give as much leeway to the other systems (computer vision/trajectory prediction) as possible.

For the demo, I’ve made fairly good progress towards most of these goals. The firmware is set up to a possible maximum configuration of XY acceleration/feed rate, though I think this can be further configured. The demo entails moving the cup 10cm in the 8 cardinal directions + diagonals, and back to (0,0). This leads me to believe that the robot should be able to handle any instructions that call for less than 10cm of translation too. While unrigorous, I’ve also timed these translations, and they seem to fall within 0.3-0.4s, which is fairly decent.

 

Future testing would be to set up a python script that communicates G-Codes directly to the arduino through the serial port. I believe this should be possible. This can automate random valid coordinates as well. I also need to purchase a different set of limit switches. I had a misunderstanding over how they function, and so in order to enable homing, must set up four at the ends of each rail.

Monday’s ethics lecture gave us a lot of new perspectives and it was interesting to sit in and listen. After that, we continued work in our own areas. As Gordon continued to set up the KRIA and Jimmy continued to work through the camera, we were thinking through the feedback we received last week about the overlapped functionality between camera and KRIA. Jimmy has done great with setting up the CV model with our OAK-D camera, and even got the chance to run a Kalman filter on it. Josiah has been doing great with the robot, getting the chassis built. Gordon was working next to Jimmy during this, and it sparked a conversation about what would be left for the KRIA to do if the camera is capable of running the Kalman filter as well. 

 

Originally, we had planned the KRIA to potentially house the CV, run the Kalman filter + trajectory calculation models, and utilize the on board FPGA for hardware acceleration, while the camera would be sending in video frames or ball position. We hadn’t really dived into the exact divide of labor between camera and KRIA, because we weren’t sure of the exact capabilities of both systems. We knew generally what they could do, but didn’t know how well each component would work in a technical sense, like with what accuracy can the model of the camera detect the ball, latency of sending data between systems, or how complex the Kalman filter code is and the feasibility of writing HLS to utilize the hardware acceleration. Thus we had originally proceeded knowing that there were potentially going to be questions about the camera and KRIA interface, and potentially an overlap in functionality. 

 

We didn’t expect the camera to be so powerful, capable of not only doing depth sensing but also running the CV and the Kalman filters. At this point with the camera doing so much, all the KRIA can offer is the on board FPGA to accelerate any calculations before sending off the landing coordinates to the robot. Even that would require writing our own Kalman filter model in C in order to use the Vitis HLS for hardware acceleration, which is a very challenging task (difficulty of doing this confirmed by Varun, FPGA TA guru). Yes it could also be nice to utilize the SoC of the KRIA to run CV models instead of a laptop, but something like a Raspberry Pi could get the same job done with way less technical difficulties and setup overhead. The difficulties of the KRIA and what exactly is “difficult to set up” are elaborated upon more in Gordon’s solo status report.

 

Because of all this, more discussions about the role of the KRIA were held, and Gordon met with Varun the FPGA guru to get his input as well. Overall, it was decided that next steps could follow this plan:

 

1. Research into if a raspberry pi will be able to handle everything necessary, keeping the KRIA as a backup plan. Consult table below for pros and cons of KRIA vs raspberry pi.
2. Benchmark how long the CV+trajectory calculation would take running on a Raspberry Pi, hopefully interface with robot and determine if we can get away with just using the Pi
3. In parallel with #2, if Kalman filter is just in python right now, try changing it to be in C
   1. Two reasons why. First, C programs run faster than python programs, so with latency being even more crucial now this could be worthwhile.
   2. Second reason, in case we need to fall back to KRIA, to use HLS we need a C file for what we want to be converted to RTL for hardware acceleration
      1. Although it’s possible that people have wrote similar Kalman filters for HLS already, research online on availability and feasibility of other people’s code

 

	Pros	Cons
KRIA	Potentially the fastest overall computation, although unsure exactly by how much More powerful SoC, potentially able to run programs faster than raspberry pi Already have the board	Difficult to set up and get running, would require a lot of research into how it interfaces Would take a lot more work to have Kalman working on HLS Potential speedup maybe not worth the massive knowledge and setup overhead
Raspberry Pi	Significantly easier to use and setup Effectively the same function as KRIA without FPGA	Potentially more latency No hardware acceleration

 

Admittedly, timeline wise this is later on in the semester than we hoped to be having these design changes, but given our workloads and other events happening during the past month and the fact that we already have a lot done, we are confident that we will be able to pull through with these changes. In fact, changing to the Raspberry Pi eliminates the complicated KRIA work and makes it more feasible to complete this aspect of the project. We will look into where to get the Raspberry Pi ASAP, and begin work to catch up. 

Regarding the robot, assembly is well underway, and is looking to be completed this next week. Check out the photos for progress!

For the camera, a lot of work was done in integrating the detection and tracking pipelines, and now we are able to fully detect and track the coordinates of a moving ball, with good accuracy thanks to the addition of background subtraction filter. A big risk the remains is the kalman filter, with details being described in Jimmy’s status report.