April 5 – April 18
The past week were final presentations. We also worked on getting all the components working together. Matthew finished the WebApp, Joon finished the website, and I worked on integrating them all and doing final testing.
After testing on the ground rather than on the tables, I realized a flaw with our Nav2 setup. Nav2 was creating ghost walls, which showed up on the /map topic and on RVIZ. This was a big problem because it would trap itself in the ghost walls, and not be able to get out of them. Therefore, I made an algorithm to use the Laser Scans (see image below) to move forward until a distance in front of the laser scans, then turn. It actually worked very well. Therefore, we are going to use that navigation algorithm as our final algorithm.
I also worked on incorporating all the components together, and created the mission controller script which is an FSM to do all the tasks in order. We also got testing numbers for our poster and report.
Next week, the poster is due, video is due, we have demos, and the report is also due. We are on task to get all of these in and working. We need to do more testing and a little more refining because right now, the battery level affects some components. However, we should be on track.
April 5 – April 18
The past 2 weeks we worked on tuning the robot to pickup accurately and NAVIGATION. We broke some more motor drivers and had to wait a little to get new ones, but we fixed them all. Navigation was the main goal of the past 2 weeks.
For getting the robot to pickup accurately, Matthew and I tested and refined a script that detects object, then gets distance. Then it rotates based on the distance from the center of the camera, then moves forward half the distance. Then it gets new rotation value and distance then rotates and moves forward. This was better than the old script we had because it could more accurately get to the object and pick it up.
The main goal was navigation. I got Nav2 to work which took a very long time. I tested it using Rviz2, getting feature by feature working together. Eventually I got the /map working and full slam using slam_toolbox. Then I also got Nav2 working and tested some simple commands on the CLI. Next, Joon and I worked on finalizing a script to do frontier search using Nav2. The script works by doing frontier search, then when 10 object detections come in, we stop and wait, get new rotation + distance values, then executes the script we tuned the week prior. It works decently, but we haven’t been able to test in a bigger room yet.
I also realized that the YOLOv8 inferences were using the CPU, which is why they were taking 1.6s each. I instead use the GPU via CUDA which was able to make it do inferences in 25ms.
I also reorganized the fs and wrote the ROS2 launch script. With this script I was also able to make a bash script for a systemd service that executes on powering the rover. That way, we can simply plug in the Jetson to the battery and our navigation script can execute. Testing can become much easier with this feature, and it is one that will be used in our final demos.
Next week are final presentations. We will continue working on the robot, tuning navigation scripts, and incorporating webapp and the shoe/ankle YOLO model. Our final demo is in 2 weeks, so we need to make sure everything is working by then.
Mar 29- April 4
This week were interim demos. We demoed twice and got some good feedback. Unfortunately right before we demoed, one of our motor drivers broke, a plastic piece on our arm broke, and our arm fell off our drivetrain. Luckily we were able to patch everything together to get a working demo. We just showed our robot driving forward, identifying the orientation of our box, then picking it up. There were a couple hacks we had to do to get this to work, but it was able to do it, at the very least.
Everything we need for batteries came as well as our backup motor drivers. After the demos, I fixed the motor driver, and setup the batteries so that it can run without being plugged into the wall. It is also able to run at a slower pace and turn now. We also worked together to tune the robot to the new pace since it is more consistent now.
The next steps are to get navigation working, and make the Jetson work without being plugged into a monitor. I will also train a YOLO model on shoes with ankles so that the robot can find a human. Next week our focus will be on getting navigation working. That is our priority, then if we have time we can add in the YOLO model and the website. We are on schedule, but will need to get a lot done.
Mar 22- Mar 28
This week we had a meeting with our professor and TA and updated on our project and got some feedback. We have our interim demos next week, so our focus this week was getting a working project for the interim demo. We decided our goal is to put a box in any orientation in front of the robot, and make the robot recognize it and its orientation, then drive up to it, then pick it up.
I worked with Matthew to clean up the wires on the drivetrain and also 3D print something that can hold the Jetson, motor drivers, arduino, and batteries. On the hardware side, we had to rewire everything a couple of times and everything isn’t as secure as we’d like it yet. Some wires still come loose and its a lot of debugging that. I researched and ordered everything we need for lipo batteries and connecting everything together, but the parts haven’t arrived yet. It will remain messy for the interim demo, but when batteries come we plan to clean everything up.
On the software side, I updated all the scripts to interact with each other. The jetson sends “orientation, distance” via serial. Then our Arduino reads that and drives distance then uses the orientation to adjust the arm to pickup the object. I am also working on writing a launch file program that can be used with the Jetson so that we don’t have to plug it into a monitor to run commands. I will have to use systemd, I believe but will work on that tomorrow.
We are on schedule. This week the main focus was getting everything ready for the interim demo, but then next week we can focus on getting navigation working and everything else as well.
Mar 15- Mar 21
This week we had our Ethics lecture. We also worked on our independent parts. We didn’t have a meeting with our professor and TA this week, but we got some feedback.
I focused on getting perception and navigation of the robot working this week. I got the RealSense D455 camera connected to the Jetson and got it publishing to camera topics. Then I wrote a script that subscribes to those camera topics and does YOLO inference to determine if it detects a box or not. The YOLO model is one that Matthew trained, and is not our final model. It is not great yet, so I set our confidence threshold to 30% for testing, but we will get that up to 95% after Joon finishes training the model. With the 30% confidence threshold, it works decently well. I also did some research and am using the depth part of the camera to calculate how far in meters, x y z the object is from the middle of the camera’s view. The custom ROS topic sends position, orientation, and object type. The position is calculated, the orientation is currently calculated as well, but Joon is trying to train the YOLO model to detect orientation, and the object type is determined by the YOLO model.
Next, we are using Nav2 for navigation. Nav2 uses SLAM internally which should help making visualization easier. Nav2 also uses a global and local planner to determine navigation. I have started on the script that will subscribe to the camera topics and will use the perception topic to determine if an object is found. It publishes to motor control topic that should tell the motors what to do. Right now, it is missing the arm control parts, and it hasn’t been tested yet. We just received our motor drivers after some delays with Amazon and reorders on Friday, so we should be testing next week.
We are on schedule. The goal is to get our motor drivers working next week, and test out our navigation. After that, we need to get our arm working with the rest of the system.
Mar 8- Mar 14
This week we had our Ethics assignment and worked on our project. We made got feedback on our implementation from our design report and worked on finalizing our design.
We had a meeting with our professor and TA this week and talked about how our navigation is going to work. We decided on using Nav2 which uses SLAM internally to create an algorithm that can navigate rooms. We had to decide if we wanted to narrow our scope to a certain type of room, with and without counters and obstacles and decided on creating an algorithm that can navigate this on its own. It is detailed in our design report, but we may tweak the waypoints and how those work.
This week, I worked on setting up headless connection to the Jetson so we can connect access it from our laptops. I also finished installing ROS2 and other dependencies for OpenCV and Nav2. I started working on combining OpenCV with ROS2, but will continue that next week.
Our progress is on schedule. We will have to get a lot working before the interim demo and the final demo, so we will have to work on it a lot. Next week, the goal is to get perception and navigation working.
Feb 22- Mar 7
This week and through spring break, we worked on the design report, and made some progress on the robot. We finalized our design and wrote it all up in our design report. I helped Matthew to disassemble the robot and get the arm working. I also helped him out with creating the API for basic arm movements. We also submitted requests for all the parts we will need.
The main work I completed was setting up the Jetson platform. I flashed the Jetson Ubuntu OS image onto the SD card and successfully booted the Jetson Orin Nano. After verifying the system was running correctly, I began installing ROS2 (Robot Operating System) on the Jetson to support the robot’s software framework. I also considered different system architectures for the project, including whether to containerize the robotics stack (e.g., ROS2, perception, and navigation components) using Docker/Podman to improve reproducibility and dependency management. I decided that the best for now was to install everything on bare metal, at least for testing, then if time allows, to containerize everything to make it cleaner, modular, and more reproducible.
Our progress is on schedule. We plan to get a lot of the building parts done after spring break. The next step is to get the motors working and to get navigation done. I plan to get the navigation and perception stack done in the next 2 weeks. This will be a big part of the project, so the sooner we can get it working, the better.
Feb 15 – Feb 21
This week we had our design presentations. I gave the presentation and received some feedback from our advisor. He suggested we think about how we are going to do our notifications, whether we do them through the webapp or if we send emails. We originally were going to send email notifications, but we are thinking about just going through the webapp instead.
We also got together to assemble the robot. We used the base and arm we got from inventory and assembled it. We also started testing the servos for the robotic arm and found that a couple of the servos are broken. I also started figuring out how the Jetson works and our plan is with that.
Our progress is on schedule, we wanted to order all the parts, and get our final design in order before spring break, then actually start hacking after spring break. We have the parts we want to order and plan to submit the requests soon. Next week, we have the design report so we should have our final design figured out. We also want to get started on some technical work if we can, with the code or mechanical parts to make sure our design is what we want it to be.
Feb 8 – Feb 14
This week we started off by finalizing our use case and our goals with our advisor and TA. We want to pick up boxes instead of items and be able to find a human and lift the box up to the human. We also did more research on the parts that we want to incorporate for our project. I submitted an inventory request for the Arduino Braccio robotic arm and Matthew picked that up. We did, however, find better a better base and arm online that we could buy that are more sturdy and well built if we need to. I put all the components together to see how this could start looking and the base may not be strong enough to have the robotic arm and a box that the arm is holding. Therefore, we may have to buy the stronger base just in case. We also started working on the design presentation slides this week and thinking through what kinds of software we will need as well.
Our progress is on schedule. Next week are design presentations and I’m going to be presenting. I would also like to do more research on how ROS will integrate with OpenCV and our YOLOv8 model. We are also thinking about using an arduino instead of the Jetson for motor control and PWM signals so I want to do more research on how those integrate as well.
Feb 1 – Feb 7
Proposal Presentations were this week. I didn’t present, but I helped make the presentation slides. Listening to the presentations gave me an idea of what other groups were doing and gave me some inspiration for our own project. We also changed our use case a little bit based on feedback from our professor. Instead of having a ramp that the robot returns up on, we will incorporate some sort of arm to lift items up to the users. Therefore, the use case is better as the user doesn’t have to walk to where the ramp is to grab the item, it is lifted up for them.
We also chose some basic parts we want to incorporate such as the Jetson Orin Nano and Intel RealSense Depth Camera. We researched and chose these because of how they will integrate with our project. I submitted inventory requests for those parts and picked them up. I also grabbed a 2-motor tank base for our robot. We should be on schedule. Next week, we will begin working on the design presentation and we will finalize all parts that we will need, and start submitted parts requests. I am doing the design presentation so I will also get ready to present that.