April 19 – April 25
This week, I presented during the final presentation and spent time preparing to clearly communicate our system design and progress. In addition, I conducted several system-level tests to better understand how the perception, pickup, and navigation components were performing together.
On the web application side, I added a feature that allows the system to send an email notification to the user once the rover successfully picks up the object. This improves the user interaction aspect of the system by providing a clear signal that the robot is ready to deliver the item. I also continued refining the structure of the web interface to ensure that it can be smoothly integrated with the rover-side logic as the full pipeline becomes more stable.
After the final presentation, we divided responsibilities for the remaining work leading up to the final demo. I will be focusing on completing the final video and continuing system-level testing, particularly for the web application and its interaction with the rover. Overall, I believe our progress is on track, and for next week I plan to continue working on the final video while supporting additional testing and integration efforts as needed.
April 5 – April 18
For the past 2 weeks, I mainly worked on the navigation and user interface components of our project. At the beginning of the week, I developed a creeping line search algorithm as a potential backup approach for object searching. However, after discussing with our professor, we determined that this method would not be robust enough for our use case, so we transitioned to using a Nav2-based frontier search approach.
I then worked with Jasmine on developing the navigation script that combines Nav2 with frontier search. The robot performs exploration using frontier search, and once an object is detected, it switches to our previously tuned approach sequence to accurately align with and pick up the object. We were able to test this pipeline in a small-scale environment, and it performed reasonably well, although further testing in a larger space is still needed.
During development, I also encountered a hardware issue where a motor driver was damaged due to a wiring mistake. After identifying the cause, I corrected the wiring and ensured a more stable setup moving forward.
In addition to navigation, I worked on developing the web application interface for the system. The current implementation includes an initial item selection page, a confirmation/start page, and a control page that allows the user to either stop the robot immediately in case of an emergency or indicate that the item has been successfully received. While the rover-side actions are currently implemented as placeholders, the structure of the web application is designed so that it can be easily integrated with the full system once the backend pipeline is complete.
Overall, I believe our progress is on schedule. For next week, I plan to continue refining the navigation system, perform more extensive testing in larger environments, and begin integrating the web application with the robot’s full pipeline.
Mar 29 – April 4
This week, I attended the interim demos and observed the full system integration of our robot. During the demo, I was able to see how the perception pipeline, including object detection and orientation estimation, connected with the drivetrain and arm control to successfully pick up the box. This helped me better understand how the individual components we developed come together in a complete system.
In addition to the demo, I began researching the navigation component of our project. I reviewed how navigation will be implemented in our system, including how perception outputs can be used to guide movement and how the robot should behave when searching for and approaching targets. This helped me prepare for the next phase of development, where navigation will be integrated with the rest of the system.
Overall, I believe our progress is on track. For next week, I plan to focus on implementing the navigation component and working toward integrating it with our existing perception and control pipeline.
Mar 22 – Mar 28
This week, I continued working on the computer vision component of our project, focusing on improving the robustness of our object detection and orientation estimation. After evaluating the previous model trained on approximately 650 images, I observed that it struggled to generalize well, particularly for tilted cases and objects at farther distances. The model appeared to overfit to the initial dataset and was not reliably capturing orientation variations.
To address this, I collected an additional 500 images, with a stronger focus on edge cases. In particular, I targeted scenarios where the box is tilted to the left and right across all three base configurations (small, medium, and large faces on the ground), as well as cases where the box is positioned farther from the camera. This helped diversify the dataset and improve the model’s ability to generalize across different orientations and distances.
Using this expanded dataset, I retrained the YOLOv8 model and was able to significantly improve performance. The updated model can now classify both the base orientation and tilt state of the box using a structured labeling scheme: S, M, and L to represent which face is on the ground, combined with 0, 90, L, and R to represent the tilt relative to a reference orientation. In testing, the model was able to reliably identify these states even at greater distances than before.
Overall, I believe our progress is on track. For next week, I plan to focus on start implementing and testing the navigation component of our system.
Mar15-Mar21
This week, I worked on both the ethics lecture and the computer vision component of our project. On the CV side, I focused on improving our object detection pipeline and exploring how we can estimate the orientation of the box for grasping.
I began by collecting and labeling a custom dataset of approximately 650 images of our target box. This was done to improve detection performance compared to using only publicly available datasets, which did not always generalize well to our specific object and environment. I also experimented with YOLOv8 and explored how detection outputs could be extended beyond simple bounding boxes to include information about the box’s orientation.
In particular, I investigated whether it is possible to determine the tilt and orientation of the box using a combination of model predictions and geometric reasoning. Rather than relying solely on the model, I explored an approach where we first classify which face of the box is on the ground, and then estimate the tilt angle relative to a reference orientation using image-based features. This helped me better understand the limitations of detection models and the role of additional processing in robotics perception.
Through this process, I also gained a clearer understanding of how the camera, YOLO model, and downstream logic connect, especially in terms of how detection results can eventually be used for positioning and grasping. I found that while detection is working reasonably well, estimating orientation and handling different viewing angles remains a challenge and will require further refinement.
Overall, I believe our progress is on track. For next week, I plan to continue improving the dataset and model performance, and work toward integrating orientation estimation more reliably into our perception pipeline so it can be used by the robot for more precise manipulation.
Mar 8- Mar 14
This week, I mainly focused on the computer vision side of our project. I prepared a dataset of box images from Open Images and used it to train a YOLOv8 model for our object detection pipeline. I then tested the trained model on the Jetson using the RealSense camera to verify that the detection results could be combined with depth information to estimate the distance to detected objects. Through this process, I gained a better understanding of how the Jetson, camera, and YOLO model work together and how detection outputs can later be used for robot navigation and object pickup. During testing, I observed that the model detects humans much more reliably than boxes, and that box detection was more consistent when the object was fully visible and had a clear rectangular shape.
In addition to the perception work, I also completed my part of the ethics assignment for the team report. Overall, I believe our progress is still on schedule, although the delayed arrival of the motor drivers means that some integration tasks have been slightly postponed. For next week, I plan to begin connecting the perception pipeline more directly to the rest of the robot system.
Feb 22 – Feb 28
This week, I mainly focused on working on our design report before spring break. I wrote several sections of the report including the Introduction, Use-Case Requirements, Test, Verification and Validation, and Related Works sections. Writing these sections helped me think more carefully about how the system should behave and how we will measure whether our design meets the requirements we defined earlier in the semester. In particular, working on the testing and verification section required us to clearly define measurable criteria for our robot’s performance, which helped strengthen the overall design documentation.
In addition to working on the design report, I also met with my team to work on the robot hardware. We disassembled the Braccio arm to debug the servo issues and discovered that the servo horns were worn down rather than the servos themselves being broken. I helped replace the damaged servo horns and reassemble the arm so that we could continue testing the robot arm.
Overall, I believe our progress is on schedule going into spring break. After the break, I plan to focus on implementing drivetrain control and beginning work on the computer vision portion of the project, including integrating the camera and preparing for the object detection pipeline.
Feb 15 – Feb 21
This week, I attended our design presentations and listened to feedback given to other teams, which helped reinforce areas we should continue refining in our own project. Outside of presentation time, I worked with Matthew and Jasmine to test the servos of the robot arm. During our initial attempts to control the servos through the Arduino, we ran into issues with the setup. I helped debug the problem and we realized that we needed to use the Braccio shield in order to properly interface with the arm. After correcting this, we discovered that three of the six servos were not functioning properly.
Although this was an unexpected setback, we were able to obtain replacement servos from Roboclub and plan to replace them next week. Overall, I believe we are still on schedule with only minor adjustments needed. For next week, I plan to focus on replacing the faulty servos and conducting torque and load testing to determine whether the Braccio arm is strong enough to reliably grip and lift a representative grocery box. This will help us decide whether we continue with this setup or transition to our backup arm before spring break.
Feb 8 – Feb 14
This week, I attended our Monday meeting with our faculty and TA where we finalized the updated project direction. During the discussion, I helped clarify why changing the object use case was a suitable decision and made sure I fully understood the new workflow where the robot searches for a human and delivers the object directly. This meeting helped solidify the scope of our system and the requirements moving forward.
I also spent time researching additional robotic arm and drivetrain options that we could purchase alongside the components we borrowed from inventory. Our professor recommended evaluating multiple hardware options in case one setup does not perform reliably during the project. During this process, I found the robotic arm and drivetrain from ‘Waveshare’ that the team reviewed and agreed would be strong candidates for our system. In addition, I helped work on the design presentation slides with the team.
Our progress is currently on schedule. For next week, I would like to focus on helping assemble the robot hardware and begin researching how we will train and deploy the machine learning image model on the Jetson. I also plan to look into how we will use the Arduino for motor control on the base so we can better prepare for system integration.
Feb 1 – Feb 7
This week, I mainly helped prepare for the proposal presentation by contributing to the slides and reviewing our project goals and use case. I also attended several other teams’ presentations, which helped me better understand the expected scope of the project and gave me ideas for improving our system design. During our proposal presentation, we received feedback from our professor regarding our original ramp-based design. The professor raised concerns that requiring users to retrieve items from a ramp location would reduce accessibility and convenience. Based on this feedback, our team decided to change our design to use an arm mechanism that can lift objects directly to a reachable height for the user. This change improves the practicality and usability of our robot.
Following this design change, I spent time researching potential robotic arm and gripper components that could support the updated system. I focused on identifying off-the-shelf options that could integrate with our drivetrain and computing platform while staying within budget and design constraints.
Currently, our progress is on schedule. Although the design change required additional planning, we made the adjustment early enough that it should not significantly affect our timeline. Next week, we plan to begin working on the design presentation and continue selecting parts so that we can finalize our hardware decisions and start submitting part orders.