Month: December 2024

I evaluated the YOLO model’s performance with unit tests as described in the final presentation and team status report (using a dataset of test images), and found that the model’s performance was below what we initial set for our use-case requirements. Examining the images that the model misclassified, I found that it would fail to classify certain objects that weren’t well-represented in the training dataset, like milk jugs. I collected a larger dataset of recyclables to train on, and also added some trash images, and trained new version of the model. This one performed much better, with accuracy scores pretty much in line with the use case requirements.

I also made progress with the mechanical build. The shelf that we wanted to use was ordered very late, and might not arrive in time. After discussing with the group, we decided to make the structure ourselves. I figured out the dimensions for the table, CADed a design, and enlisted the help of a friend with woodworking experience to build it. We used scrap wood from Techspark, so the table won’t cost anything in our budget. Some adjustments have to be made, like the camera holder (more details in the team status report), but the build looks good, and I’m glad we were able to put it together on such short notice.

My plan for next week is most finishing up the build. The camera holder will need to be raised, and the door needs to be laser cut to a size that will fit the hole. We will also be wiring everything together.

Schedule looks good, we should be able to get a working project done for the video.

Here is the CAD of the table:

One thing that we are concerned about is that the acrylic that we are using as the swinging door is clear, which we think could interfere with the CV’s ability to categorize the objects. In order to manage this risk, we are planning to either paint the acrylic, or cover it in some sort of paper or tape to make it a solid color.

We have also built the structure of the recycling bin, as well as the post to which we are planning to attach the camera. However, we have noticed that the height at which we are placing the camera may not be far enough from the platform for the entire object to fit in frame. In order to fix this issue, our solution is to rebuild the post so that the camera will be placed higher. We have tested with a few objects to find the optimal height to place the camera.

We decided to switch back to woodworking instead of the pre-built structure because the shelf did not arrive on time. Fortunately, we were able to get some help from someone with woodworking experience and finished building the structure of our product within a couple of hours.

We are also moving the ultrasonic sensor to the door rather than being next to the camera above the table. We found that the camera will need to be higher than the holder currently is (12 inches), but that distance will make the ultrasonic sensor less accurate. We plan to move the sensor in front of the motor, facing sideways across the door. This will allow it to detect items as they are placed.

We had a little bit of delay in the schedule for building our mechanical part, but we are in progress of finishing our build and should be able to start final testing next. This had no severe changes to our schedule.

CV Units Tests:

I tested the YOLO model’s performance by running detection on a test dataset of 133 trash/recycling images. These were images that weren’t used in the training or validation steps, so the model had never seen them before. I evaluated the model’s performance using the percentage of images classified correctly. I initially found that the model was not able to detect certain objects that were not well-represented in the training dataset. For example, the original training dataset did not include many milk jugs, and the model failed to classify milk jugs as plastic in testing. I retrained a new version of the model on a much larger dataset, and accuracy improved dramatically. I also used this testing to tweak the threshold confidence score at which the model would classify an item.

Web App Unit Tests:

I tested different components of the web app to make sure that everything was displayed accurately on the screen through written unit tests, and also by user testing. The tests showed that everything was displayed correctly, but sometimes waiting for the information to load on the screen would take a few seconds. I also tested the http call requests to make sure that they would be able to consistently make accurate requests to the backend and the jetson, and they responded to the calls consistently.

Hardware Unit Tests:

I tested each component of the hardware (ultrasonic sensor, servo motor) separately with different scripts to make sure that they work as expected before combining them and implementing the serial communication with the Jetson. I tested with different distances and angles to see what’s ideal for our use case requirement. I also adjusted the delay time between each operation based on how they performed when all the components were combined.

Overall System Tests:

Mechanical construction of our project has started, so once we are finished and have secured all the components together, we will start the overall system tests. This includes taking different objects and placing them on the table and timing how long it takes for the object to be detected, classified, and sorted. We will also test the accuracy of our overall project by hand classifying the objects that we are testing with, and checking how many of them are sorted into the correct bin. We will also make sure that items that are sorted into a particular bin (ie. recycling/trash) actually make it into the bin without falling out.

The most significant risk to the project right now is the mechanical structure. We ordered a shelf to use as the main structure, but it may be arriving later than expected. While that is arriving, we will build what we can with the parts we have. We can laser cut our tabletop and door, and once the mounting brackets for our motor arrive we can drill holes to mount our components. The shelf should arrive with plenty of time for us to complete final assembly for the video and demo.

We have decided to not implement the rejection system for the recycling bin. Our original plan was to have the bin reject any items that were unable to be recycled or thrown away, such as batteries, paint, or chemicals. Unfortunately, datasets for these kinds of special waste are difficult to find online, and the nature of the materials makes it difficult to make the dataset ourselves with actual photos. We explored making a dataset out of images online, but most of the images we found were stock images, or didn’t have the items in the sort of photo conditions that we will have for EcoSort. We did find a dataset of battery images, and trained a version of our final model to recognize batteries, but we have decided that the main focus of our project is to be able to differentiate between recyclable and non recyclable items, and only recognizing one type of special waste wouldn’t add too much utility to EcoSort.

No scheduled changes have occurred.

Our color palette for our web app:

I decided to order a usb camera to test if the image quality would be better, and if the integration with the Jetson would be less headache-inducing. It looks like the camera (a Logitech webcam) has autofocus functionality, which is very nice, and the pictures aren’t tinted in any way, unlike our previous camera, where the feed would be tinted orange for a few tenths of a second. In addition, the integration with the Jetson (particularly the ability to control the camera with a Python script) was much easier. We will be using this camera moving forward. I was also able to get TensorRT working. Essentially by converting the model to another format (.pt weights file to .trt model), we can run inference faster, and without having the build the model each time. This means that inference can run faster. I tried training a version of our recycling detection model to also detect batteries, and it seemed to perform decently. However, the team decided that since we were only able to recognize batteries (datasets for other types of special waste were hard to find), the rejection system would be too specific to be worth pursuing for the project. From testing, overall model accuracy is at around 75% on our test dataset, below the 90% we set as a goal. This could partially be explained by the dataset using lower quality images, and so maybe the model will perform better when actually used in EcoSort. Either way, the current version of our model is accurate enough to be deployed, so I will devote most of my attention to integration and mechanical construction, in accordance with the schedule. We have the acrylic that we plan to use for our tabletop. Next week I will laser cut the door out, and drill holes so we can attach the motor and mounting brackets. Once the shelf arrives,  I will work on assembling that.

Most of my coursework in machine learning was theoretical, discussing how models worked, so I didn’t have as much experience training and validating models for a specific task, and that was something I gained a lot of experience on in this project. I had to figure out how to gather datasets, choosing the right one for the task, and evaluating the model after training. It was definitely a lot of trial and error, as I ended up trying multiple combinations of different datasets and training parameters. I also had to get familiar with some libraries for the project, like Pytorch and OpenCV. Luckily, there are a lot of resources available online for this kind of “applied” machine learning. I also learned a lot about the Jetson. I didn’t really know too much about the Jetson’s capabilities before capstone, but a semester of working with it has showed me what a powerful platform it is. I consulted a wide variety of resources, from NVIDIA documentation to forum posts from other Jetson users.