Carnegie Mellon ECE Capstone, Spring 2025 — Matt Rhee, Marcus Tita, Raunak Sood
The last two weeks Ive been testing and rebuilding the robot arm to make it more reliable. I have updated system to have both arm and electronic controllers on the same 3d printed structure, and revised pieces to make wires internal. I tested some pwm ramping functions, and am still revising control software.
I had difficulty threading the wires through the structural components, and adjusting calibration for each new iteration.
Next week we are going to work on dialing in arm movement.
What did you personally accomplish this week on the project?
This week, I worked on integration testing with the QR code scanner system and the robotic arm. We realized that the robotic arm was sometimes too slow and sometimes too fast at picking up the packages, so we had to make sure that it was reliable. After tweaking the code, I tested the robotic arm’s pick up reliability and 9/10 times it was able to pick up the moving box. In addition to this, I worked extensively on the final report. We received a lot of constructive feedback for our midway reports, so I wanted to start early on the final report to work on the areas that need improvement. I wrote about half of the report so far including the introduction, use-case requirements, design and implementation sections.
Is your progress on schedule or behind?
Our progress is on schedule!
What deliverables do you hope to complete in the next week?
Next week, we plan on doing more testing with the entire system. Specifically, we 3D printed a few boxes of different heights and we want to verify that the system picks up all three boxes reliably. I will also continue working on the final report, taking the feedback from the midway report into account.
Testing and Verification Supplement
Our testing mostly includes integration tests with the entire system. This involves running an end-to-end test by placing a box on the treadmill and seeing if the robotic arm is able to pick up the box correctly. We prefer this over software-only tests because they tell us more information about what we need to improve. For example, the arm dropped one of the boxes in one of these tests, and we realized that the end effector had significant jitter, which was causing this. As a result, we ordered a new servo for the end effector to mitigate the jitter that was causing the box to be dropped. Our measured results include the number of times the box is able to be picked up reliably. For example, we test the end-to-end process 10 times and see how many times the robotic arm does its job. Initially, it was only getting it right 1 or 2 times out of 10, but now it gets it right pretty much every time. These are the most important metrics that we look at.
This week we were able to initially integrate all of our components (Arduino arm/kinematics code, Python vision/kinematics/control code). We were able to integrate the CV with the robot arm to track qr codes and determine box position/speed/height, send the robot arm sequential commands, and pick up the moving packages. We were able to pick up multiple packages in a row.
Our current biggest risk is that inaccuracies/imperfections in each submodule will add up to make our end-to-end system insufficiently accurate and/or reliable. Currently our vision module is not perfectly accurate, and our arm and kinematics are also slightly imperfect / idiosyncratic. We are ramping up our unit testing to ensure that these issues are minimized. We are also working on optimizing the PC->Arduino command sequencing to minimizing these risks.
Next steps are adding better move selection and qr tracking. We also plan to more definitively test and map out the arm’s exact angle idiosyncracies. Similarly, we plan to do more robust testing of the vision module’s real-world mappings. Best case, by the end of the week, we hope to have the entire box-grabbing system working in a reliable and decently accurate manner.
This week I wrote more code to control the robot arm, and designed a clamp to connect the camera to our treadmill in a robust non moving manner. I isolated the vacuum from the sensors to help with keeping steady voltage feedback. We were able to get the robot to pick up multiple packages in a row. Next steps are adding better move selection and qr tracking.
What did you personally accomplish this week on the project?
This week, I tested the entire software system with the robotic arm. This involved running the main function, which relied on the QR scanning and kinematics modules to send commands to the Arduino controlling the arm. I was able to successfully verify the general logic of the main function and send accurate, fast commands to the robotic arm’s Arduino. One of our main concerns was that the delays of scanning the QR code, getting the box speed, sending the commands to the arm, and the arm actually moving would be too slow to efficiently grab the boxes on our relatively short treadmill. We were able to assuage those fears, as the system ran very quickly with time to spare. I greatly extended the main file this week, as it needed to be updated with the new additions I made to the QR scanning file and supplemented with the greater understanding I gained of the Arduino control inputs. I also spent a lot of time this week tweaking and improving the main and QR scanner files, as there were numerous issues with the accuracy and precision of predicting the box’s movements and converting the camera readings into real world coordinates relative to the arm (pixels -> centimeters).
Is your progress on schedule?
We are running on schedule.
What deliverables do you hope to complete in the next week?
I hope to continue testing the software system that runs on PC (vision, kinematics, and command modules) with the arm. The arm currently has some idiosyncrasies with the angles that haven’t been completely tested and mapped out. I plan to do some more robust testing of the vision module’s real-world mapping as well by placing boxes of different heights at specific locations on the treadmill running at specific speeds. Hopefully I’ll be able to verify that the real-world position/speed mappings are accurate now that I’ve revamped that code. I also hope to at least gain a better understanding of the exact arm angles relative to the command inputs, as I haven’t had that much time with the arm. Best case, by the end of the week, I will have the entire box-grabbing system working in a reliable and decently accurate manner.
What did you personally accomplish this week on the project?
This week, I worked on integrating the QR code detection algorithm with the robotic arm. Previously, we had to send commands manually to tell the robotic arm when to pick up a box. But, this week, we got hooked up the QR code system with the arm, so now it picks up the box when the QR code is detected. I worked on debugging some of the code that was causing the arm to be too delayed in picking up the box. This involved changing the delays in the QR code detector code to be less. I also worked with the rest of the team to test our system with real boxes moving on the treadmill. We were able to get the arm to reliably pick up moving boxes based on the QR code detection. Aside from that, I’m also starting to draft aspects of the final report. Based on feedback from our midway report, we need to add more details to our final report and tell a story about how our product is better than existing products. I’m starting to list some bullet points of what we might want to talk about in the final report.
Is your progress on schedule or behind?
Our progress is definitely on schedule.
What deliverables do you hope to complete in the next week?
Next week, I plan on working on getting the robotic arm to place boxes in different containers based on the QR code. Currently, the robotic arm places all objects in the same location. So, I need to add a bit of logic to tell the arm to place the robotic arm in different locations. Once that’s done, all of our main goals for this project should be covered. I also want to get a good chunk of the final report drafted since we have a good sense of what we need to include at this point.
This week we put it all together and got our robot to pick up packages off a moving conveyer belt. This was accomplished by wiring up the robot fully, troubleshooting a lot of angle sensor / actuator issues, and programming the Arduino to run multiple to-position moves. This serves as a big milestone for us as we have achieved a working robot that can complete tasks. We also tested the CV code for getting the QR value, determining the box height, and determining the box speed/position on our complete system with boxes on the conveyor belt. We also tested our timing/speed/latency, and we definitely should have enough time to successfully scan/track/calculate box variables and move the arm into position to pickup boxes before they fall off the belt.
Our biggest concern right now is that the vision module and Arduino arm control will not interface sufficiently well. This integration could become very time consuming and put us behind schedule, as we would potentially have to make many changes to each individual module in addition to working on the interfacing itself. We plan to finish and test the PC->Arduino interfacing code as soon as possible. We are also making sure to modularize the respective PC and Arduino code to make changes easier to implement down the line.
Our next steps are fine tuning our control algos over the coming weeks to pick up multiple packages quickly. We need to dial in our time tolerances and create bins for the robot to put boxes into. We also plan to finish linking our Python and Arduino code, which will enable us to test the QR scanning software and the robotic arm together. Hopefully we can also start testing the entire system next week, since the scanning system can currently acquire all the necessary metrics, and the robotic arm can currently pick up, move, and drop boxes.
This week I finished up wiring the robot and wrote code to run and send angle feedback position moves. It took alot of trouble shooting to get all three angle sensors accurately measuring 0-180 of the three main actuators but luckily it wasn’t to hard to get the vacuum to turn on and off. I spent alot of time this week wiring, troubleshooting, and programming the robot arm. Luckily all the work paid off and we had a successful package pick up and move. I was able to get the robot to run to 7 diffrent positions in a row using a command sender app and a Arduino control sketch. I got the robot to pick up a moving package on its 15th try.
Our next steps are combining our QR and robot algos, then we should be able to sort package efficiently
What did you personally accomplish this week on the project?
This week, I tested the QR scanning code for getting the QR value, determining the box height, and determining the box speed/position on our complete system. Creating physical boxes and attaching various QR codes to them, I then ran them on the conveyor belt on different speed settings with our Brio3000 webcam mounted above. I was able to successfully verify the correctness and accuracy of the code to determine the box value, height, speed, and position.
Is your progress on schedule?
We are running on schedule.
What deliverables do you hope to complete in the next week?
I hope to finish linking our Python and Arduino code, which will enable us to test the QR scanning software and the robotic arm together. Hopefully we can also start testing the entire system next week, since the scanning system can currently acquire all the necessary metrics, and the robotic arm can currently pick up, move, and drop boxes.
What did you personally accomplish this week on the project?
This week, we made huge progress on combining all of the working parts together. Namely, the robotic arm is now able to pick up objects off of the conveyor belt, which is the MVP for our project (see slack for video). My contribution to this mainly involved writing Arduino code that controls the arm. I wrote a significant portion of the code that controls the third motor (horizontal axis) which allows the robotic arm to move side-to-side (so that it can drop off the box after picking it up). This was the final piece of the code that enabled our arm to pick up and drop off boxes in their bins. I also continued working on some of the extra features I wanted to add from last week (the ML object detection code) in parallel.
Is your progress on schedule or behind?
Our progress is on schedule!
What deliverables do you hope to complete in the next week?
While our robotic arm picks up and drops of boxes correctly, it is quite slow. It takes about 15-20 seconds for the entire process of picking up and dropping the box. We want to increase the speed of this overall process and bring it down to ~5 seconds for the entire process. This will involve increasing the RPMs of the motor and writing code to handle this. We will also have to test that the arm doesn’t make more mistakes because of the increased speed. Additionally, we want to begin hooking up the camera system to the robotic arm so that we can pick up objects with the QR code detection.