Category: Ishaan’s Status Report

These past 2 weeks, we have finally completed Ballbot and have tested different parts of our project. I helped redesign the casing for the hardware , added a GUI for the touch screen, and also tuned the computer vision to work with outdoor tennis courts. Instead of just having the hardware on the roof of the robot, I helped design a second layer to go on top of the hardware that would also encase the touchscreen. We decided to add a touch screen to the top of the Ballbot so that it is easy for a user to turn the Ballbot on and off.

For the UI, I made a simple GUI in Python with 3 buttons to start and stop collecting balls, as well as turn Ballbot off. I integrated the GUI with our existing software so that controlling Ballbot is as easy as possible. To do this, I used Tkinter for the UI elements and the Threading library to have a thread waiting for user input while the rest of the code is running. On the software side, I also tuned the computer vision thresholds to better work with out door lighting. This allowed us to test the Ballbot outside and see it work on the tennis courts.

This week we made progress towards finishing the physical aspects of the robot and placing the finishing touches in the software. We finished constructing the basket to carry the tennis balls. We ensured that the basket to fill 20 to 30 balls so that we could meet the requirements we laid out. The basket was made with waterproof plastic to make sure the basket was lightweight and waterproof. We also used the same swivel wheels as used on the arms of the robot to lift the basket off the ground. We still have to finalize how to attach the basket so that it is secure.

On the software side, we added functionality so that when no balls are detected, the robot spins in a 360-degree circle to see if there are any balls outside of its view. If so, it stops spinning and goes after those balls. If it still does not find any tennis balls, the robot stops spinning and waits until a ball is seen.

This week, we were not able to test on the tennis courts as panned because of rain. However, the weather app says that there will not be rain on Monday, so we hope to test on the tennis courts then. In the meantime, we have tested indoors and everything seems to be working as planned.

This week, I worked on integrating the software and mechanical parts of the project and testing them together. The software required the Nvidia Jetson Nano, Intel Camera, and buck converter to give the Jetson Nano 5V of power which were not part of the mechanical tennis ball launching system. We made an acrylic roof for the Ballbot on which we placed all the hardware required to run the software as well as the battery and motor controller which were part of the mechanical system. In order to connect the battery to both the motor controller and the buck converter for the Jetson Nano, we soldered wires onto a protoboard that we also placed with all the other hardware. To secure the hardware to the acrylic roof, we used M3 dual lock adhesive which acts similar to Velcro.

Once the components were all on Ballbot, we first adjusted the speeds of the motors so that we were using minimal power while still ensuring balls would not get stuck partway up the ramp. Then, we finally started running our computer vision code and tested everything together.

Our integration worked successfully and the Ballbot was able to go after tennis balls and launch them towards the back of the robot. Next week, we plan on adding the basket to the back of the robot so that the balls are collected in the basket instead of just being launched back.

Also, we painted the exterior of the wood so that the arms Ballbot would be weatherproof. We still need to enclose the hardware to ensure the hardware can survive in the outdoors. Also, next week I plan on adding code to the software so that if the Balbot does not see a tennis ball, it turns in a circle once and ensures there are not any tennis balls outside of its field of view. Currently the Ballbot just stops when it does not see a ball which is not the best for picking up all the balls. We also hope to test the Ballbot on a tennis court if there is good weather next week.

This past week, I made progress on making the Jetson Nano and all the other connected hardware portable with the iRobot Create 2. To do this, I first figured out how to power the Jetson Nano and the Intel depth camera with the rechargeable battery we had. After testing that all our software worked as normal on the Nano when it was powered by the battery, I moved to testing the software with the Create 2 robot free to move after the tennis balls. This was not possible earlier since the camera and Jetson Nano were being powered from the wall outlet. We temporarily taped the camera to the front of the robot and also placed the Jetson Nano on top. When Rashmi held a tennis ball in front of the robot, the robot accurately followed the tennis ball, which shows that our software is working.

On the hardware side, I helped Ryan attach the arms and motors to the Create 2 robot so that the tennis ball collection system was mostly complete. We screwed the acrylic ramp to the arms and tested the hardware system. We manually controlled the robot and were able to launch the tennis balls that were funneled into the robot.

Next week, I plan on working on integrating the software and hardware into 1 system and also building the platform on which we will place the Jetson Nano, Intel camera, and battery. The 2 systems seem to be working well independently, but merging them together will be challenging.

This past week, I continued to work on the software to get it to build on the Nvidia Jetson  Nano and also helped on the building side of things.  For the software, I was finally able to get the iRobot Create 2, the Intel Realsense Camera, and the OpenCV computer vision code all running together on the Nvidia Jeston Nano. This involved finding a workaround to building the Intel Realsense Python library from source as well as fixing some other issues related to transferring the code from a Windows implementation that was running on my laptop to a Linux now for the Jetson Nano.  Now, when running the code, the Create 2 robot turns towards the closest tennis ball in view of the camera until the tennis ball is centered. To better test this, we need to power the Jetson Nano from a battery instead of the wall port. This will allow the entire system to be portable instead of tethered to the wall.

On the hardware side, I helped Ryan construct some of the wood parts of the Tennis ball launching system. More specifically, we worked on building the sides of the ramp that the tennis ball will go over, as well as the pieces that the motors are connected to. To connect the pieces of wood, we decided to use wood glue instead of screws so that we were free to cut however we needed.

Next week, I plan on trying to power the Jetson Nano with a battery and seeing if there are any issues with that. Also, I will try to get most of the code written to control the motors. I will help Ryan continue working on the building side of things so that hopefully the motors are ready to be tested along with the rest of the hardware.

This week I spent some time with Rashmi working to integrate my OpenCV computer vision algorithm with her robot motion code. As suspected, integrating the c++ with python was harder than expected. We tried using pybind but for some reason that didn’t quite work the way we wanted it to. So, to save some time, we just converted the c++ code into python line by line. At first this felt like 2 steps back but one step forward but we eventually were able to get the python to behave exactly how the c++ did. The speed was not much slower either. We performed some tests to make sure that the code did not contain any bottlenecks.

After rewriting the code, we were able to integrate the output from the camera to the iRobot controls. The robot can now move towards a tennis ball if the tennis ball is in the field of view. Rashmi is going to work on improving this algorithm so that it can use the depth output from the camera and properly generate a route.

Additionally, I also helped Ryan assemble the robot on Friday. Although we got the pieces cut, the assembly was more complicated than expected. A lot of the cuts were at 45 degree angles and screwing them together was not easy. Although we did our best, we think that we should redesign the parts so that it will fit together better.

This week, I plan to spend some time building the python library on the Jetson Nano. This is currently not working because they don’t support pip and other necessary tools that I need to get this to work. Additionally, I might also need to go help Ryan cutting and building the robot since (due to the fact that we had a setback this weekend and the makerspace does not open until Monday) we might be slightly behind.

This week I worked a lot with the computer vision algorithm trying to make it more robust. I took the camera outside and tested the algorithm in daylight. At first, even if the camera angle changed a little, the ball stopped being detected. Furthermore, I was outside for a few hours and I noticed that as the daylight itself changed, I found that I had to retune the algorithm to work. However, after rewriting some of the code and tuning the parameters of the HSV color space, I believe that I have finally found a happy medium threshold where the tennis ball is detected all the time in daylight.

I also spent some time this week helping my team write the design review document. We were hoping to have feedback from the professors regarding our design process by now but we have not heard anything.

Next week, I plan on working with Rashmi in order to integrate her working with regards to the motion planning of the iRobot and my computer vision algorithm. I’m not sure how this process is going to go as neither one of us has really worked in the other’s domain. I anticipate that this process of integrating it and then finetuning the motions and fixing the issues that arise will take us this whole week and possibly some of next week.

This week, I continued to work on the computer vision part of our project. After being able to interface with the Intel RealSense Depth Camera, I was able to convert the input into the HSV color space and select the range of colors for each channel that would allow us to find tennis balls. The algorithm now can locate multiple tennis balls in view of the camera and draw a target on each one to show what it is detecting. Currently, I am using my laptop to run the algorithm but will work on building the code on the Jetson Nano in the upcoming week.

Also, I helped Ryan with the hardware portion of the project and we created an initial prototype to test the motors and see if we could control them. We created a set up to find the best distance apart to keep the motors and were able to power the motors with the power supply from the Makerspace. We also programmed the Arduino to control the speed of both motors, but the motors were being very finicky at anything but their top speed.

We are not currently behind schedule and in the next week, I hope to further improve the algorithm to work better in different lighting conditions. Also, I want to work on moving the code over to the Jetson Nano where it will have to eventually run. In addition, I want to look into integrating the iRobot Create 2 API calls Rashmi is working on with the OpenCV code.

This week, much of my time was spent finalizing parts lists and submitting orders. Along with my teammates, I found sites where we could get the parts we needed such as the motors, motor controller, and mounts for the motor. A few days after submitting the order form for the parts I was in charge of finding, we found out that the iRobot Create 2 we needed was out of stock on the main website as well as on Adafruit. I spent some time looking for a different site and we found a small business that was selling the robot at a discounted price.

While waiting for parts to arrive, I worked on the Computer Vision algorithm for detecting tennis balls. It took a while to learn the OpenCV API for C++ and see what functions were available to me. Along with the efforts of my teammates, we were able to create a basic program that could mask out objects within a specified color range. We also were able to interface with the Intel RealSense camera and read data from it.

We are not currently behind schedule and in the next week hope to obtain all our needed parts. Also, I plan on continuing to develop the computer vision algorithm to be more focused on detecting tennis balls. It is likely that I will need to deal with the challenges of detecting the tennis ball in different lighting conditions without too much noise. To deal with this, I want to implement noise filtering in the computer vision algorithm by next week.