Posted on

Teddy’s Status Report for 4/12

I spent the last two weeks overhauling the entire physical design of the gantry. I replaced the aluminum angles with aluminum extrusions, and used brackets with t-nuts to provide an easily adjustable yet sturdy frame. I made it so that the supporting rod now goes through two wheel attachments which slide across the rails instead of having the rod sit on the edge of the frame. The z-axis rack has been changed from a 3d-printed part to a lazer cut one in order to make sure that it is rigid and straight. I also abandoned the bearing wheels on the z-axis in favor of a 3d-printed guide slot. I replaced some parts of the conveyor belt in order to add tension, and was able to pull the belt taught. I also made a mount for the conveyor belt stepper and added the stepper and the timing belt onto the conveyor, so that the conveyor now moves with the stepper motor. In short, the physical components of the gantry are essentially finished.

Next week, I plan to try to get the depth information from the stereo camera and help work on all of the steps necessary for integrating the work done by my teammates. I am behind schedule but I believe we should get everything done in time.

Verification

For our testing, we will test the gantry’s ability to move with a certain granularity. We’ll move it a certain number of steps and measure its movement in order to determine what the distance corresponding to a step is. We’ll also be doing rigorous testing with multiple common trash/recyclable items such as bottles, cans, cardboard, jars, chip bags, etc. to ensure that the end-effector is able to handle a wide variety of materials and surfaces. We will also make sure to test objects with different weights to determine if it is able to lift objects up to 1 lb. We’ll also be testing the overall accuracy of the depth information (z-coordinate of the object) calculated from the stereo camera.

Posted on

Team Status Report for 4/12

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk to our project is accumulated error. Since our sub components depends on necessary communication between each other (machine learning model results to web app and machine learning model results to Arduino and Arduino commands to gantry movement), if one component does not obey their respective design requirement exactly it could risk the whole project’s validation. In order to prevent, we are taking extreme precautions to ensure that the each sub component is working as intended through rigorous testing. Our aim is to mitigate individual risk to mitigate overall risk.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

There are no current changes made to the existing design of the system.

Provide an updated schedule if changes have occurred.

There are no schedule changes as of now.

Validation.

We plan to do multiple runs, first isolating by each component of the system (e.g. centroid & classification, depth accuracy, end-effector pickup rate) and doing multiple trials with different types of trash items. We will then do the same with the complete, combined system.

Reliable Sorting:

We will test a variety of trash/recyclables with multiple surface types and materials, in order to make sure that the end-effector is able to pick up objects with 95% success rate. We’ll also measure the distance that the gantry moves over a certain amount of steps in order to determine its granularity in the x,y,z movement directions.

Real-Time Monitoring:

We plan to ensure that the bytes from the Jetson Orin Nano reaches the web app in 30 FPS by timing when they leave the Jetson and arrive to the server using either Wireshark or timestamps in the code of each entity communicating over the network.

Real-time Object Detection:
We plan to use a set of real-life trash objects like plastic bottles and cans. We will do multiple sets (10) of static images each containing a different variety of objects to ensure that the machine learning model can work regardless of the images in the camera frame. We will also need to analyze that labels that the model outputs to see if it lines up with reality. We are aiming to match the 0.70 precision. from the Design Report.

Posted on

Ethan’s Status Report for 4/12

This week, I finished setting up the necessary dependencies for the machine learning model (I was able to download correct PyTorch, OpenCV, and NumPy to be able to use the Jetson Orin Nano’s GPU). Moreover, I met up with Alejandro to start the integration process with the machine learning model and the web app. We decided on a scheme to send messages to the web app. Together we were able to get eh bounding boxes to show up on the web app for real-time monitoring.

Currently, I am behind schedule a little as I need to be met up with Alejandro again to setup a communication protocol between the Jetson Orin Nano and the Arduino.

Machine Learning Model Verification:

In order to verify the machine learning model’s performance, I took pictures of real world trash (empty plastic bottles, soda cans, and etc) on the actual conveyor belt. From there I run the model over the images and plotted the bounding boxes to see how tight of a fit it has on the image. From the image below, we can see that the bounding box is able to cover the object and that gives us confidence that we can hit the ±5 center pixel requirement from our design requirements. Moreover from the timing code I wrote, the model runs is able to run 50-70 ms way below our 150 ms design requirement. And finally, on the validation set we were able to get the 0.70 precision that we almost specified in the design requirements.

Posted on

Ethan’s Status Report for 3/29

This week, I worked on setting up the Jetson Orin Nano. I currently was only able to setup PyTorch after a long couple of days because I hit a couple of roadblocks with setting up CUDA and cuDNN, Most of my issues stemmed from incorrect paths that weren’t properly set in the environment variables and incorrect versions downloaded by “pip3 install torch”.  Unfortunately most of progress was wiped as I accidentally did something to the drivers (I have no idea what because I was levels deep in random Nvidia documents). In a better light, I also wrote some code for Monday’s demo.

Currently, I am behind schedule a little. I plan to fix this by continuing to work the Jetson Orin Nano for the rest of this week and early next week.

Posted on

Ethan’s Status Report for 3/22

This week, I started full dataset training for the YOLOv8 OBB model. I implemented the loss function using three tunable hyperparameters that adjust the weight of the bounding box regression loss, the classification loss, and the angle loss. The intent of this was so to show more transparency in the unified loss calculation (the sum of all the previously mentioned loss), and if one loss was too high I could adjust how much the penalty would be. This method would hopefully allow us to better control the model’s convergence and allow for a better check pointing scheme that would save each best model in all four loss types (unified, regression, classification, and angle) to retrain later on. While the model started to train, I started working on creating a Docker image for the Jetson to make porting over the model easier.

Next week, working on some visualization code that would plot the predicted bounding box and class on static images in a clean manner (the core logic of this will be scaled out later for the Jetson)

Currently, I am on schedule.

Posted on

Team Status Report for 3/15

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risks to our project is getting the machine learning model to real-time speed. In particular, the YOLOv8 architecture can achieve real-time speed on the Jetson Orin Nano however it has been noted it is incredibly frustrating to do so as it is careful speed and accuracy tradeoff analysis. To mitigate this risk, the model not being fast enough, we need to approach the speed and accuracy tradeoff analysis very meticulous, in particular we plan to have a detail log that we can reference to determine the sweet point for our use case.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

There are no current changes made to the existing design of the system.

Provide an updated schedule if changes have occurred.

There are no schedule changes as of now.

Posted on

Ethan’s Status Report for 3/15

This week I was able to get the YOLOv8 OBB model work on the simple single object toy images and multi-object toy image as described in the Design Report. To recap simple single object toy images, are image where there is a single object in frame that is relatively close to the camera and the multi-object toy images. Visually the bounding boxes and classification labels on objects in these images are accurate (± 5 pixels within the center of the true bounding box). With this, I think we are ready train the model on the full dataset and see how its purpose is.

Next week, if the results are good I plan on porting the model weights to the Jetson and seeing its performance there.

Currently, I am on schedule.

Posted on

Ethan’s Status Report for 3/1

The majority of effort this week was spent on finding bugs in our YOLO codebase that caused the results from last week to look really poor. I discovered it was the loss function. Previously I was using a more naive approach that combined a weighted mean-square loss for the bounding boxes with a weighted cross-entropy loss for classification. After reading a couple of Medium articles about YOLO, I realized that I was implementing an entirely different loss function. Once I fixed that I also a little bit more training infrastructure that would hopefully making training analysis easier: curve plotting for each loss. By monitoring loss, I can identify parts of the model that would need tuning in future runs.

Next week, I plan on getting detection working on toy images.

Currently I am on schedule.

Posted on

Ethan’s Status Report for 2/22

This week, I was able to finish the training infrastructure to train the YOLOv8 OBB model. Now that I am able to train the model, I need to employ some sort of verification strategy determine that the model was implemented correctly before I do full batch training. I decided on training the model one single basic image (I am defining basic as an image where the object is close up and on top of a distinct background). After training on this image for a significant number of epochs, I found that the detected bounding box was completely off. Currently, I believe that something went wrong with the model’s OBB detection head and spent a majority time this week trying to verify this assumption.

Next week, I plan on getting detection working on this toy image and hopefully training using the entire dataset and analyzing the results from there.

Currently, I am on schedule.

Posted on

Ethan’s Status Report for 2/15

This week, I was able to finish the initial implementation of the YOLOv8 OBB model. Unfortunately, I found that the dataset I found on Roboflow last week is no in the format that YOLO models expect. Inside of having a (x, y, w, h, theta, class label) ground truth for each object in an image, the dataset actually has the following ground truth (bbox coordinate 1, bbox coordinate 2, bbox coordinate 3, bbox coordinate 4, class label). In order to finish this, I need to re-annotate the dataset. I plan on finishing a script by the end of tonight to fix this problem.

Currently, I am a little behind schedule. To remedy this, I plan on continuing to work on implementing the training infrastructure tomorrow and Monday. My goal is to start training the model by Tuesday. This way I will have sufficient time to (i) debug my model implementation and (ii) write data augmentations to artificially increase the amount of data.

Next week, I plan to have the model trained for a reasonable number of epochs to determine what optimizations I need to do on it for the best performance on the training dataset.