Category: Uncategorized

This week I worked on finishing the web app button control of the whole system, the speed control and the statistics counters. They all work now. Additionally, I ran testing on the webapp latency. I also helped with getting the vacuum pump and the z-axis movement to work on the Arduino side of the code. I also changed the range of the gantry to accommodate the physical design changes. I also experimented with a higher voltage supply to try and make the motors move faster.

For the remaining time left, I intend on finalizing path finding movement and getting the Arduino that controls the conveyor belt to work with the jetson via serial connection.

I am currently on schedule.

This week, I spent the majority of time analysis the results of our newly trained YOLOv8-OBB Nano model with the previously trained YOLOv8-OBB Medium model. Compared to the Medium model, the Nano model doesn’t do as bad as I thought on our real-life test images (since Medium has x10 the number of parameters). As a safe guard to this, I also started training a YOLOv8-OBB Small model that will get done by tomorrow morning. I will spent morning comparing the results of these models and based on the results will decide what the new model we will be.

Currently, I am on schedule.

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk to our project is preventing the accumulating of error between the machine learning model and the gantry system. To prevent this we want to make sure that the machine learning model and the gantry is as accurate as possible. This means performing a lot of unit tests for each system.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

There are no current changes made to the existing design of the system.

Provide an updated schedule if changes have occurred.

There are no schedule changes as of now.

Unit Tests

Gantry System:

Weight Requirement:  We tested with a few different types of objects that were all around 1lb to make sure the gantry met the 1lb requirement.

Pickup Consistency: We tested on 30 different recycling objects with different materials and textures for just the pick-up and drop off, with focus on the end-effector.

Speed: We timed the gantry’s movement assuming the maximum amount of motion in order to get its maximum time for the pickup/drop off sequence.

Web Application:

Usable Video Resolution: Counted the number of pixels in a static frame from the video stream using an image snipping tool.

Real-time Monitoring: Measured round-trip timestamp checking of 416×416 images.

Fast Page Load Time: Measured the time it takes for the initial webpage to load in a browser.

Machine Learning Model:

Ability to identify Different Trash (One Environment): Gathered a various number of trash from the 5 classes and placed them on the conveyor belt to see if the model was able to correctly identify and localize them.

Ability to identify Different Trash (Multiple Environment): Gathered a various number of trash from the 5 classes and placed them on the conveyor belt to see if the model was able to correctly identify and localize them. Before the image was send to the model, I tried to adjust the brightness, the color, and how blurry the image was to see how the model would in this scenario (since we don’t really know what the environment of the final demo room will be).

Speed of Inference: Feed a random subset of images from our real-time photos and calculated the average time of inference to see if it met the value in our design requirements.

Overall System:

We tested on 30 different recycling objects the whole detection, communication, and pick-up/drop-off sequence for timing and consistency.

Findings:

– The machine learning model is robust enough to work in any lighting scenarios.

– The web app’s latency is heavily dependent on CMU-SECURE WiFi.

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk to our project is ensuring that the new time-of-flight sensor has the same expected performance of the originally planned depth sensing camera. Before making this change, we did some preliminary testing, so this was a calculated change. From what we have seen it has good performance and we need to ensure that it has the same performance on the actual system.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

We changed the depth sensing camera in favor of using a time-of-flight sensor. Since this sensor is really cheap, it had a negligible impact on our budget.

Provide an updated schedule if changes have occurred.

There are no schedule changes as of now.

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk to our project is accumulated error. Since our sub components depends on necessary communication between each other (machine learning model results to web app and machine learning model results to Arduino and Arduino commands to gantry movement), if one component does not obey their respective design requirement exactly it could risk the whole project’s validation. In order to prevent, we are taking extreme precautions to ensure that the each sub component is working as intended through rigorous testing. Our aim is to mitigate individual risk to mitigate overall risk.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

There are no current changes made to the existing design of the system.

Provide an updated schedule if changes have occurred.

There are no schedule changes as of now.

Validation.

We plan to do multiple runs, first isolating by each component of the system (e.g. centroid & classification, depth accuracy, end-effector pickup rate) and doing multiple trials with different types of trash items. We will then do the same with the complete, combined system.

Reliable Sorting:

We will test a variety of trash/recyclables with multiple surface types and materials, in order to make sure that the end-effector is able to pick up objects with 95% success rate. We’ll also measure the distance that the gantry moves over a certain amount of steps in order to determine its granularity in the x,y,z movement directions.

Real-Time Monitoring:

We plan to ensure that the bytes from the Jetson Orin Nano reaches the web app in 30 FPS by timing when they leave the Jetson and arrive to the server using either Wireshark or timestamps in the code of each entity communicating over the network.

Real-time Object Detection:
We plan to use a set of real-life trash objects like plastic bottles and cans. We will do multiple sets (10) of static images each containing a different variety of objects to ensure that the machine learning model can work regardless of the images in the camera frame. We will also need to analyze that labels that the model outputs to see if it lines up with reality. We are aiming to match the 0.70 precision. from the Design Report.

This week, I finished setting up the necessary dependencies for the machine learning model (I was able to download correct PyTorch, OpenCV, and NumPy to be able to use the Jetson Orin Nano’s GPU). Moreover, I met up with Alejandro to start the integration process with the machine learning model and the web app. We decided on a scheme to send messages to the web app. Together we were able to get eh bounding boxes to show up on the web app for real-time monitoring.

Currently, I am behind schedule a little as I need to be met up with Alejandro again to setup a communication protocol between the Jetson Orin Nano and the Arduino.

Machine Learning Model Verification:

In order to verify the machine learning model’s performance, I took pictures of real world trash (empty plastic bottles, soda cans, and etc) on the actual conveyor belt. From there I run the model over the images and plotted the bounding boxes to see how tight of a fit it has on the image. From the image below, we can see that the bounding box is able to cover the object and that gives us confidence that we can hit the ±5 center pixel requirement from our design requirements. Moreover from the timing code I wrote, the model runs is able to run 50-70 ms way below our 150 ms design requirement. And finally, on the validation set we were able to get the 0.70 precision that we almost specified in the design requirements.

This week was spent doing work for the conveyor belt, updated gantry frame, and the end-effector. I was able to construct a rough conveyor belt when the materials arrived, such that the cloth has enough tension where it rolls when the bars it is wrapped around rotate. I spent some time working out how the new gantry frame will be build using the new aluminum extrusions, with the plan being to cut the rods sometime next week for assembly. I was also able to finish the design of the end-effector by attaching a solenoid to the end-effector such that a hole is plugged and unplugged to release the objects grabbed by the suction end-effector. I plan to attach it to the gantry tomorrow so that it is ready in time for demo.

Next week I hope to rebuild the frame and attach a stepper motor to the conveyor belt so that it is able to turn when commanded by the Arduino. I also hope to start the movement estimation code so that the gantry can move to the object’s location given that it is moving at a constant speed after detection. I am on schedule.