Author: jyw2

I evaluated the YOLO model’s performance with unit tests as described in the final presentation and team status report (using a dataset of test images), and found that the model’s performance was below what we initial set for our use-case requirements. Examining the images that the model misclassified, I found that it would fail to classify certain objects that weren’t well-represented in the training dataset, like milk jugs. I collected a larger dataset of recyclables to train on, and also added some trash images, and trained new version of the model. This one performed much better, with accuracy scores pretty much in line with the use case requirements.

I also made progress with the mechanical build. The shelf that we wanted to use was ordered very late, and might not arrive in time. After discussing with the group, we decided to make the structure ourselves. I figured out the dimensions for the table, CADed a design, and enlisted the help of a friend with woodworking experience to build it. We used scrap wood from Techspark, so the table won’t cost anything in our budget. Some adjustments have to be made, like the camera holder (more details in the team status report), but the build looks good, and I’m glad we were able to put it together on such short notice.

My plan for next week is most finishing up the build. The camera holder will need to be raised, and the door needs to be laser cut to a size that will fit the hole. We will also be wiring everything together.

Schedule looks good, we should be able to get a working project done for the video.

Here is the CAD of the table:

One thing that we are concerned about is that the acrylic that we are using as the swinging door is clear, which we think could interfere with the CV’s ability to categorize the objects. In order to manage this risk, we are planning to either paint the acrylic, or cover it in some sort of paper or tape to make it a solid color.

We have also built the structure of the recycling bin, as well as the post to which we are planning to attach the camera. However, we have noticed that the height at which we are placing the camera may not be far enough from the platform for the entire object to fit in frame. In order to fix this issue, our solution is to rebuild the post so that the camera will be placed higher. We have tested with a few objects to find the optimal height to place the camera.

We decided to switch back to woodworking instead of the pre-built structure because the shelf did not arrive on time. Fortunately, we were able to get some help from someone with woodworking experience and finished building the structure of our product within a couple of hours.

We are also moving the ultrasonic sensor to the door rather than being next to the camera above the table. We found that the camera will need to be higher than the holder currently is (12 inches), but that distance will make the ultrasonic sensor less accurate. We plan to move the sensor in front of the motor, facing sideways across the door. This will allow it to detect items as they are placed.

We had a little bit of delay in the schedule for building our mechanical part, but we are in progress of finishing our build and should be able to start final testing next. This had no severe changes to our schedule.

CV Units Tests:

I tested the YOLO model’s performance by running detection on a test dataset of 133 trash/recycling images. These were images that weren’t used in the training or validation steps, so the model had never seen them before. I evaluated the model’s performance using the percentage of images classified correctly. I initially found that the model was not able to detect certain objects that were not well-represented in the training dataset. For example, the original training dataset did not include many milk jugs, and the model failed to classify milk jugs as plastic in testing. I retrained a new version of the model on a much larger dataset, and accuracy improved dramatically. I also used this testing to tweak the threshold confidence score at which the model would classify an item.

Web App Unit Tests:

I tested different components of the web app to make sure that everything was displayed accurately on the screen through written unit tests, and also by user testing. The tests showed that everything was displayed correctly, but sometimes waiting for the information to load on the screen would take a few seconds. I also tested the http call requests to make sure that they would be able to consistently make accurate requests to the backend and the jetson, and they responded to the calls consistently.

Hardware Unit Tests:

I tested each component of the hardware (ultrasonic sensor, servo motor) separately with different scripts to make sure that they work as expected before combining them and implementing the serial communication with the Jetson. I tested with different distances and angles to see what’s ideal for our use case requirement. I also adjusted the delay time between each operation based on how they performed when all the components were combined.

Overall System Tests:

Mechanical construction of our project has started, so once we are finished and have secured all the components together, we will start the overall system tests. This includes taking different objects and placing them on the table and timing how long it takes for the object to be detected, classified, and sorted. We will also test the accuracy of our overall project by hand classifying the objects that we are testing with, and checking how many of them are sorted into the correct bin. We will also make sure that items that are sorted into a particular bin (ie. recycling/trash) actually make it into the bin without falling out.

The most significant risk to the project right now is the mechanical structure. We ordered a shelf to use as the main structure, but it may be arriving later than expected. While that is arriving, we will build what we can with the parts we have. We can laser cut our tabletop and door, and once the mounting brackets for our motor arrive we can drill holes to mount our components. The shelf should arrive with plenty of time for us to complete final assembly for the video and demo.

We have decided to not implement the rejection system for the recycling bin. Our original plan was to have the bin reject any items that were unable to be recycled or thrown away, such as batteries, paint, or chemicals. Unfortunately, datasets for these kinds of special waste are difficult to find online, and the nature of the materials makes it difficult to make the dataset ourselves with actual photos. We explored making a dataset out of images online, but most of the images we found were stock images, or didn’t have the items in the sort of photo conditions that we will have for EcoSort. We did find a dataset of battery images, and trained a version of our final model to recognize batteries, but we have decided that the main focus of our project is to be able to differentiate between recyclable and non recyclable items, and only recognizing one type of special waste wouldn’t add too much utility to EcoSort.

No scheduled changes have occurred.

Our color palette for our web app:

I decided to order a usb camera to test if the image quality would be better, and if the integration with the Jetson would be less headache-inducing. It looks like the camera (a Logitech webcam) has autofocus functionality, which is very nice, and the pictures aren’t tinted in any way, unlike our previous camera, where the feed would be tinted orange for a few tenths of a second. In addition, the integration with the Jetson (particularly the ability to control the camera with a Python script) was much easier. We will be using this camera moving forward. I was also able to get TensorRT working. Essentially by converting the model to another format (.pt weights file to .trt model), we can run inference faster, and without having the build the model each time. This means that inference can run faster. I tried training a version of our recycling detection model to also detect batteries, and it seemed to perform decently. However, the team decided that since we were only able to recognize batteries (datasets for other types of special waste were hard to find), the rejection system would be too specific to be worth pursuing for the project. From testing, overall model accuracy is at around 75% on our test dataset, below the 90% we set as a goal. This could partially be explained by the dataset using lower quality images, and so maybe the model will perform better when actually used in EcoSort. Either way, the current version of our model is accurate enough to be deployed, so I will devote most of my attention to integration and mechanical construction, in accordance with the schedule. We have the acrylic that we plan to use for our tabletop. Next week I will laser cut the door out, and drill holes so we can attach the motor and mounting brackets. Once the shelf arrives,  I will work on assembling that.

Most of my coursework in machine learning was theoretical, discussing how models worked, so I didn’t have as much experience training and validating models for a specific task, and that was something I gained a lot of experience on in this project. I had to figure out how to gather datasets, choosing the right one for the task, and evaluating the model after training. It was definitely a lot of trial and error, as I ended up trying multiple combinations of different datasets and training parameters. I also had to get familiar with some libraries for the project, like Pytorch and OpenCV. Luckily, there are a lot of resources available online for this kind of “applied” machine learning. I also learned a lot about the Jetson. I didn’t really know too much about the Jetson’s capabilities before capstone, but a semester of working with it has showed me what a powerful platform it is. I consulted a wide variety of resources, from NVIDIA documentation to forum posts from other Jetson users.

The most significant risk in our project right now is that the USB connection to the Jetson and the camera connection are a bit unstable. Occasionally, the USB port to the Arduino would suddenly not be recognized by the Jetson, and we would have to reconnect or restart the Jetson in order for it to work again. The camera was also not recognized at one point, and it started working again after restarting the Jetson a couple of times. We are unsure of what the root cause of this is, and to mitigate this risk we plan on testing our system with a different camera or a different cable, to see what the exact problem is.

No changes to the system’s design are planned. Depending on how much time we have, there are some additional features we could explore adding. One possibility we discussed was ultrasonic sensors to detect the fullness of the bins. Once we are confident in the soundness of our existing design, we will look into these features.

No updates to the schedule have occurred.

Integration testing will consist of connecting multiple systems together and confirming that they interact as we expect.We will place an item in front of the sensor, and verify that an image is captured and processed. After the classification is finished, we confirm that the Jetson sends the correct value to the Arduino, and that the Arduino receives it, and responds correctly. We will also verify that the time from placing an object to sending the classification signal is less than 2 seconds. For all of these tests we will look for 100% accuracy, which means that the systems send and receive all signals we expect, and respond with the expected behavior (ie. camera only captures image if and only if the sensor detects an object).

For end-to-end testing, we will test the entire recycling process by placing an item on the door and observing the system’s response. As the system runs, we will monitor the CV system’s classification, the Arduino’s response), whether or not the item makes it into the correct bin, and if the system resets. Here, we are mainly looking for operation time to be <7 seconds as specified in the user requirement. We will also confirm our accuracy measurements from individual system tests.

(Our current graph with stubbed data)

This week I trained a few more versions of our model on recycling datasets. The nice thing about training computer vision models is that they can be validated quite easily with a test dataset, and the training procedure for our YOLO models will output statistics about the model’s performance on a provided test dataset. I have been using this to evaluate the accuracy of our CV system. Here is a confusion matrix from the best run:

It appears that the model is very good at detecting recyclables, but often confuses trash in the test set for plastic and paper. A solution to this could be to remove trash as a classification output, and instead only classify recyclables, since trash is a very general category, and the sheer variety of items that would be considered trash may be confusing the model. In such a case, we will classify any item that isn’t classified as any recyclable category with high confidence as trash. We will also have to test our model’s performance on trash items, making sure that the model doesn’t recognize them. After I am satisfied with the model’s accuracy on a test dataset, we can move on to capturing images of actual waste with the camera and classifying those. We will test with a variety of objects, with the camera positioned at an appropriate height and angle for where it will sit on the final product. As mentioned in the design report, we want our model’s accuracy to by >90%, so no more than 10% of the items we test should be classified incorrectly (recycling vs non-recycling).

I am also working on figuring out how to deploy YOLO onto the Jetson using the TensorRT engine. If we can convert the model to an engine and load it onto the Jetson, we won’t have to rebuild the engine every time we start up our classification. Once I figure that out, our model will run much faster, and we can do the same procedure if we ever update the model: just convert the new weights into the TRT model engine format, and we will be able to run that. I hope to be able to get that working in the next week, although it’s not a necessity since even without TensorRT it should be more than fast enough.

Schedule is looking on track. Once we get the model deployed, the CV system will be a state where it could theoretically be used for a final demo.

One aspect of our design that could potentially interfere with the functionality of the system is wire management. We have multiple components that need to be connected and placed in particular places, such as the camera needing to be placed higher up to get a good shot of the item, while the motor needs to be attached to the swinging door. There are also many components connected to the Arduino, which are the ultrasonic sensor, servo motor, and the weight sensor. It might be difficult to place all of these components in place while they are still attached to the Arduino. We will use a breadboard to organize the components and connections wherever possible, and we can also mount the Jetson and Arduino in different places on the bin to get better placements for the components they connect to.

A slight change with the CV system is that we are constraining the output of our model to various types of recyclables, trash, and special waste. This way we will only recognize items of interest to a recycling bin. If the model can’t classify an image with high confidence, we will still default to classifying it as trash.

There are no changes made to the schedule.

Here is a YOLO output classifying a water bottle as PET plastic (the label is very small):

This week was spent training a YOLOv7 model with a custom dataset. I set up a fork of the YOLOv7 Github repo with our training data to be cloned into Google Colab, and was able to successfully run training. I was worried that Colab usage limits would mean that we would have to partially train the model over multiple times, but it seems like we can train for ~50 epochs before the runtime is terminated, which should offer pretty good accuracy based on other people’s experience custom training YOLO. If not, it is also possible to continue training the model from the weights file, or we can use the Jetson’s GPU to train. I found a another dataset online that contains more varied waste categories. I want to try training a model with this dataset, and figuring out training parameters and evaluating models is my plan for the next week. I’ve also found a dataset of battery images, and will further train our model on that to identify something that should be rejected. This should be enough for an interrim demo. I’m hoping in the next week to have a model that is good enough to deploy for project to at least identify recycling, since the schedule specifies that we should be done with the model next week. If needed, I could continue working on more training, since deploying another model on the Jetson is as easy as saving a new weight file.

My work this week focused on the Jetson. I got the camera connected to the Jetson and got it to capture images through terminal commands to gstreamer that save an image. I could run this command from a python file using os.system(). There is also a way to use opencv with a gstreamer pipeline to capture the image, but I haven’t gotten it to work yet. I will focus on other things for now, but the terminal command takes a bit longer since it has to set up and tear down the gstreamer pipeline every time, and the image seems to come out slightly orange, but we can at least capture images.

I also got the necessary dependencies to run pretrained yolo models on the Jetson with GPU acceleration. The dependencies were more complicated than I thought, with for example installing TensorRT (NVIDIA library for GPU-accelerated inference) required choosing the right installation for your Python and CUDA versions, but it worked out. After some basic testing, it seems like the system can perform inference on a jpg in ~50 ms, which should be more than fast enough.

Next steps are to train a YOLO model on our custom dataset. I found a dataset of recyclable waste, and split it into train/test/validation, and now that the dependencies are all set, we should be able to use the Jetson’s GPU to train.

Progress is on track.

We predict some trouble with the implementation of the backend of the web application. Because of this, the backend section may take a little bit longer than previously thought to complete. In the case that this happens, we will simplify some of the features of the web app, such as only displaying the current week’s statistics instead of being able to scroll to see all of the past week’s statistics as well.

We are also unsure about approaching the mechanical construction. We have a design in mind, but none of us have much experience with woodworking or anything like that. We plan to ask TAs and professors for guidance, and Techspark workers for help with the building process.

We did not make any changes to the existing design of the system. Currently, all of the parts of the project are more or less on schedule. We will be able to start working on integrating some systems together, such as the CV and hardware.

Some testing on inference time:

Fun with Jetson camera: