April 19-25
The start of this week involved finishing up what I was doing last week. As it turned out, I was labelling the feet datapoints incorrectly (I was labeling both the shoe and the leg). At the time, I thought that this would give the ML architecture better context and an idea to look for a shoe with a human leg. However, it turned out that this bounding box might have been too big, and the model ended up predicting almost everything as a foot. After limiting the bounding boxes to just the shoe (even if a human foot was wearing the shoe), I had much better results.
Here is an image of the labels on my validation set.
After this, to prepare for the presentation, I came up with a testing plan to test our computer vision models (both object detection AND human detection). I decided that it would be best if we had separate trials for each orientation, while letting the ML model run on a webcam (realtime inference). I mainly chose this method of testing because it simulated the application of our ML model (on our robot) very well.
After the final presentation, I realized that we had a lot of work leftover in terms of the final video, final report, final poster and final demo. Because Jasmine still had to finish navigation, I decided to finish up the final poster (still have to include final tests for nav, but will add them once done), and start the final report. We agreed that Joon would finish up the final video. By pursuing this strategy, we offload a lot of work for Jasmine so that she can fully focus on navigation.
April 12-April 18
In addition, last week, adding the CADed walls helped a lot with jolting. However, there was still a very minor jolt, simply because the drive_forward function was going from 0% power immediately to 100% power. Thus, I wrote a draft of code that would increment the power in a smoother way. I plan to test this more once Jasmine finishes with navigation, which will be this weekend.
This week, I focused on training a YOLO model for shoes/feet to detect feet. Initially, I trained with 40 images (which was enough for the blue box), but this turned out to be ineffective. Thus, I am currently trying to integrate a much larger dataset. Because shoes are a lot less distinct that a blue box, this task is slightly harder than last time.
To be honest, I believe that we are off schedule. Earlier in the week, one of our motor drivers got shorted. Unfortunately, the spare seemed to also get shorted while we were replacing the damaged one, so we had to wait 2 days for new parts to arrive. This put a delay in the navigation progress (which Jasmine is now responsible for). Joon will now be taking over the app. We still have a lot of testing to do, which we plan to do over the weekend once navigation is ready.
April 5-April 11
The start of this week focused on incorporating some feedback we received from the interim demo. Jasmine and I were able to meet and improve our pick up algorithm. Instead of a one shot pick up method (detect distance, drive, pick up), we first center our robot (by rotating in place), drive distance – fixed variable X, center again (in case our robot got off center), and then drive the remaining distance. The first reason why this algorithm works a lot better than the old one is it is a lot more robust in terms of the object being off center from the robot. Moreover, the final distance (X) is now a lot more consistent, meaning that we really just have to tune that distance well in order for our project to work.
After finishing this, I started CADing new wall and electrical mounts for our robot.
Previously, we had a piece of cardboard holding our Arduino in place. However, this was quiet flimsy and involved us using electrical tape to tape it in place. As an improvement, I CADed an electrical holder as shown below:
Moreover, another issue was that our robot was quiet back heavy. I added some wall components in the front (where we can also attach heavier objects if needed).
This improves the jolting a lot, and we no longer fall backwards if we go from 0% to 100% power.
I believe that we are on schedule. While Jasmine and Joon work on the navigation script, I will finish up the web app and help with navigation components so we can finish the project.
March 29-April 4
I prioritized the start of the week for the interim demo. Jasmine and I worked on tuning our motor converters to drive the correct distance to pick up the box. In addition, there was a major issue, where our motor driver broke, and we had to use a very rudimentary hack (where we just use an enable pin to run the motor at 100% or 0%). This worked, but it caused our robot to be very jolty and unstable. Moreover, our tuning parameters worked, but they didn’t work as well for longer distances. They were enough for the interim demo, but we decided that we would change the algorithm by moving a few steps forward, recentering the box and using the height of the position of the bounding box to determine if the robot was close enough. This would be a more closed loop system where we would have feedback for our robot controller for every single step.
After working a long time preparing for our interim demo, Jasmine and I were able to integrate Joon’s script and get a completely working robot that would: sense the object, detect distance, drive the detected distance up to the object, send the detected orientation to the arduino, and pick up the box in its orientation. Again, except for longer distances, this actually worked quite well.
During the interim demo, some elements of our robot started to break again. For example, on our second interim demo day, our arm completely broke off of the base and had to be duct taped. After the interim demo, I spent time fixing this issue.
I believe that we are on schedule. Once we implement the more robust pick up solution, we will branch off to working on the website and navigation (which we already have a script for).
March 22-28
This week, I met up with Jasmine over the weekend to prepare for the interim demo. The first thing we accomplished was interfacing the jetson with the arduino to send a command to pick up an object based on the rotation of the object. Essentially, we were able to mount our camera on to the 3D printed mount, send the feed to the Jetson, have the Jetson make a prediction (using the ML model I made last week), send that prediction along with an angle calculation to the arduino, and have the arduino move the arm and position the wrist to the given angle.
These are two frames from the same video. As you can see, our ML model was able to differentiate between these two rotations.
Jasmine and I then installed the motor drivers onto our drivetrain. After wiring the motors, drivers, and Arduino, we were able to get the wheels to start moving. However, we realized that we had too many components, and it would be hard to mount them in the space behind our arm.
Thus, on Monday, Jasmine and I spent some time measuring and CADing another component for our electrical components.
As you can see, we have compartmentalized our space for each component (Jetson, Arduino, motor drivers, breadboard) as well as added holes in the walls to allow for wiring/interfacing.
Later in the day, I went to roboclub to 3D print the part.
After testing the compatibility of this part with our robot, we realized that too much of the shelf would be hanging off of the back of our robot. Moreover, Jasmine and I discovered that we could try mounting the motor drivers inside of the drivetrain (the drivetrain frame is a hollow shell that can be taken apart). Before, we were planning on putting the battery inside of the drivetrain, but we realized the battery we were purchasing probably would not fit.
Thus, I CADed a second iteration of the electrical components shelf, which looked like this:
As you can see, we now have a bottom shelf for the battery.
I also CADed a second part to hold the motor drivers. The main goal of this part is to shelter the motor drivers from damage while inside the hollow shell of the drivetrain. We also are able to better restrict the movement of the motor drivers themselves, preventing internal wires from detaching.
After 3D printing, these were the results:
Even though our batteries did not arrive yet, Jasmine and I rewired the motor drivers and integrated the entire shelf onto our robot. While Jasmine worked more on the wiring side, I spent more time attaching the shelf onto the robot and rerouting all of the wires from the motor drivers onto the newly placed arduino.
Now, we are able to drive our drivetrain along with all of our electrical components (obviously we still have to plug our Arduino and motor driver power into the wall because our batteries haven’t arrived).
Over the weekend, Jasmine and I plan to integrate Joon’s new model with the arm and drivetrain to get a full pick up. I believe that we are on track with our schedule. After this weekend, the main things left to do are the app, searching algorithm, and shoe detection (for humans), which is completable within 2 weeks. We should then be able to finish off with testing.
March 15-21
This week, my main focus was integrating the gripper with the intel real sense camera. In an effort to kill 2(3) birds with one stone, I CADed an arm and camera mount (that would also add the necessary height to our arm. 
The front part is the mount specifically for the camera while the hexagonal extrusion is a mount for our robot arm. I purposefully angled the camera downwards 15 degrees because our camera will be around 4-5 inches high off the ground.
Moreover, I also reimplemented a YOLOv8 model. We realized that it wouldn’t be professional if our cardboard box was an STM32 microcontroller box (yes, from 18349), so we decided to cover the box up in blue tape. While this made our box look more professional, it also helped when it came to detections.
First, I started out by training a simple YOLO model with around 40 images taken of our blue box (from random angles and backgrounds).
Running a simple test script on my computer, I was able to draw a bounding box around our target object. Furthermore, I also used the dimensions and not-so-blue corners of the bounding box to determine the orientation of the box.
As you can see from the screenshot, I am detecting that this is in the narrow orientation. We are also able to detect if the box’s largest dimensions are facing the box (that will report an angle of 90), and if the box is left or right angled.
These predictions will be very useful when it comes to angling the wrist of our gripper.
I was also able to integrate the standoff onto our robot after 3D printing it.
Here is the full robot:
While testing my robot, we ran into a few issues. For one, one of the servos experiencing most of the torque seemed to break down. Moreover, a servo horn also got grinded down (preventing our servos from actually controlling the arm). Each time something internal in our robot breaks, we are forced to disassemble a lot of the project.
Personally, I believe that we are on track. Even though the motor drivers not arriving prevented us from working on the drivetrain this week, I still feel like our progress this week got us caught up for the interim demo. Thinking more about what we actually want to display for this demo, I believe we will try to show the arm, arduino, camera, jetson stack, where we are able to pick up a box under any angle/orientation. Furthermore, on Sunday, I plan to work on the drivetrain a little bit (since the motor drivers just arrived on Friday). If the drivetrain is able to get finished before then, we can also have a short and quick demo of our drivetrain being able to move.
March 8-14
Unfortunately, the motor drivers still have not arrived. Thus, I was not able to get the drivetrain to start moving. That being said, I was able to implement a detail that our design report uncovered.
While writing our design report, we realized that our braccio arm was not long enough to reach our height requirement. Thus, we thought about replacing one of the links in our arm with a longer (custom CADed link). Because I am responsible for firmware/mechanics of our robot, I looked into this issue this week. Even though CADing was a solution that we initially considered, I decided that it would be better to add an additional link into our arm, scrapping part of the second braccio arm we purchased. This way, we could get additional height while still having braccio compatible parts. I then recoded the arm to pick up a box again.
On top of doing this, I also finished up the ethics assignment, which made me think a lot about how our robot could be used and misused. Specifically, I thought about facial recognition and computer vision bias. Because facial recognition models have the potential of being racist (especially when it comes to law enforcement using facial recognition technology to find criminals). I now realize the importance of having a diverse data set, especially for detecting people and objects from all cultures and backgrounds. I know that we our currently fixating on just one (fixed) object, but if we have extra time near the end of our project, having a representative set of objects from all cultures and backgrounds will be key.
I believe that our schedule is slightly behind. This is mainly because our motor drivers have not arrived yet, so I am unable to complete the motor driver API. That being said, I was able to context switch to another task, which was extending the length of our arm, which is something that had to be done anyways. Thus, hopefully, on Monday, our robot drivers will arrive in time so that I can smoothly start working on the drivetrain.
By next week, I hope we can finish the drivetrain and start moving our entire robot around.
Feb 22-28
This week was the week right before spring break. Because we had a design report due, I made sure to allocate a lot of time to describe our methods on paper. More specifically, I took the part of writing our Design Requirements, Design Trade Studies, and some of Use-Case Requirements. One very specific piece of feedback that I tried to implement was the fact that we did not have a lot of explanations behind our numbers (in the proposal and design presentations). Thus, I really tried to hone in on where each number was coming from (and changed it if it didn’t make sense). I think going through this process was very rewarding because I was able to come up with a very solid reason behind every single requirement. In addition, one requirement often depended on another, so it was fun to derive new requirements using older ones. Moreover, for design trade studies, it was fun to list out everything we considered, and I felt like I was proving why our current design was the best out of all other designs. Going through these motions helped solidify my own understanding of our project.
On top of working on the design presentation, I also spent a lot of time this week adjusting the robot arm to work. On Monday, we realized that our robot arm’s servos were actually working. The root issue was actually that the servo horns were worn down. Luckily, we had exactly 3 spare servo horns for the 3 servo horns that were worn out.
After reassembling the robot arm, I then put started programming the servo arm using the Braccio library in the Arduino IDE. Ultimately, I was able to get our servo to pick up a very small box (which is going to be the item that we are going to pick up for the rest of the project).
From my test, I was able to verify that our arm is in fact strong and grippy enough to pick up our target object. Because of this success, instead of purchasing another arm completely, I convinced the rest of the team to buy another spare Braccio arm. This way, we do not have to completely recode our robot arm half way into the project. In addition, if more servo horns get worn out (especially during demo day), we can easily replace our arm.
Adding onto getting the robot arm to work, I also tried to help Jasmine with getting the drivetrain to work. This was a bit more complicated, as we had to use an hbridge that did not have good documentation. Unfortunately, we were not able to get this aspect of our project to work before spring break.
I believe that our schedule is back on track. The API for the servo arm is essentially done, and we plan to be able to drive our robot around within a few days of coming back from spring break. For next week, I hope that our team can:
- Finish programming the drivetrain
- Finalize the API for servo
- Start implementing YOLOv8 model on Jetson
Feb 15-21
This week, we went through design presentations. I did not present this time, but it was nice to be able to see what other groups were thinking. Especially when it came to groups that were planning to build a similar robot, I was very interested to hear what their use cases were.
Because we had design presentations though, we did not have a lot of in class lab time like other weeks. Thus, we had to meet outside of class numerous times to get on schedule.
On Wednesday, Jasmine and I met to attach the braccio arm to the drivetrain. Even though the through holes on the arm did not match the through holes on the drivetrain, we were able to mount with 3 screws. Because two of the screws needed to be mounted on slots (because no other holes aligned), we used washers to ensure the base was clamped down.
On Friday, Joon, Jasmine and I met again to attempt to test the servos of our arm. While our initial attempts to control the servos through an arduino failed, we realized that we needed a braccio shield, which we had left in our box. After utilizing the shield, we learned that 3 of the 6 servos were broken. Thus, Jasmine and I went to Roboclub to get 4 servos (1 extra) so we can replace them next week.
I believe that our schedule is slightly off track. By this week, we were supposed to assemble our robot, which we did. However, because 3 out of the 6 servos on our arm are broken, we are going to have to disassemble the arm and replace the servos next week. This setback will take extra time. That being said, we are only slightly off track. This un-forseen obstacle isn’t a big deal and we can assign one person to work on it while the others take care of next week’s tasks.
By next week, I hope that our team can:
- order parts (so they can arrive once we come back from spring break)
- replace the dead servos on the current arm
- have a program to pick things up using the arm.
Feb 8-14
This week, we first solidified our capstone design on Monday, when we met with our faculty and TA advisors. Instead of just returning back to home base, we will look for a human inside the room and find them. This change will mostly effect the ML computer vision model. I spent the entire meeting pitching my ideas and thoughts to the team and my advisors.
Next, I conducted some research on what gripper we could use. With my team, we thought a lot about if the drivetrain we rented was going to be wide enough to support a heavy arm. Because we couldn’t be completely sure, I also spent some time checking out various websites for purchasable drivetrains. In addition, Jasmine was also able to put in a request for an arduino braccio robot arm for our gripper. On Wednesday, I was able to pick this up. After picking our robot arm up, I tried picking up a few light boxes up (controlling the robot angles with my hands), and saw that it was a pretty solid mechanism.
Other than planning and checking our physical-inventory-borrowed parts, I also worked on the design presentation. Specifically, I created a flowchart that would map out all of the functions and message passing through all of the systems of our project.
I believe that our schedule is mainly on track right now. It is really great that we were able to pick up a gripper and drivetrain from inventory, because that allows us to create a version 1 of our robot.
In terms of deliverables, I believe the next week will be full of presentations, so we will have less time in class to work on our project. That being said, outside of class, I would like to be able to finish assemble the robot with my team so we can start programming it to move.
Feb 1-Feb 7
The main focus of this week was the proposal presentation. I decided to represent my group for this presentation, so I spent a lot of time preparing. I made sure to practice in front of some of my friends so I could have feedback, and one of my friends even recorded me so I could see myself presenting.
Seeing what the other groups were doing on Monday and Wednesday also gave me a good sense of the scope and difficulty of our project in comparison to everybody else. Personally, I felt like we are in a good place, and that we will be able to deliver our requirements in time.
After my presentation, it seemed like our faculty advisor did not approve of our ramp idea to get the object to a correct height. We talked over the various options and we decided to use an arm instead. Instead of navigating to a ramp after finding the object, we will simply raise up our arm so that the object is accessible to our user. Upon making this decision, I talked with Jasmine about what parts we could take from the inventory, and we were able to secure a drivetrain, jetson, and camera.
For next week, I plan to research and decide on some options for the gripper and arm so we can start flushing out our actual robot. I am currently on progress regarding my schedule (mainly because we just started), but it would be best to finish designing our robot before spring break, so we have the entire second half of the semester to debug.