Justin’s Status Report for 10/5

I spent time in the beginning of the week working on the design presentation, which went well, and we got some useful feedback. I have landed on a suitable dataset for our project: https://www.kaggle.com/datasets/arkadiyhacks/drinking-waste-classification. This dataset not only contains thousands of images to train on, but the images show the images as they might appear when captured by our camera (ie. the item is photographed from above), rather than a stock image. Unfortunately, I was out of Pittsburgh in the second half of the week to attend a wedding, so I was not able work with the Jetson and camera (which arrived after I left), so next week I am hoping to hook up the camera to the Jetson, load YOLO (off the shelf), and test out that setup. I will also start setting things up to train YOLO with a custom dataset. Luckily YOLO has many resources online, I’ve already watched some videos, and it should hopefully be easy to set up. I will also be working with my team on the design review. Being away from Pittsburgh has delayed my progress beyond what I initially hoped, and I anticipate the design report and midterms to take time next week, so I am also planning to work on this project over fall break, bringing the camera and Jetson back home with me if necessary (I should not need any other hardware to work on the CV).

Justin’s Status Report for 9/28

This week I was able to get YOLOv7 up and running on my computer to test its performance on some sample images. This was purely to gauge the model out of the box, and as expected, the classification was not very good on recyclable materials.

This was expected, and confirms our need to find a suitable dataset of recyclable materials to fine tune the model. Our original plan was to train our model on some labeled recyclable materials (ie. aluminum can, plastic bottle), and have some sort of lookup table for each classification to determine if the item is recyclable. This could potentially lead to cases where an item isn’t recognized with high confidence, and we would have to make a decision on what to do in such a case. An alternative we discussed was customizing our dataset so items are labeled as “recyclable” or “non-recyclable”, but currently I am leaning towards the original idea, with unrecognized items being simply classified as non-recyclable.

Aside from CV work, most of my time was spent on the design presentation, and working out the details of our new game system. The game system won’t interact directly with the CV, but adding a new system to the project felt like something for the whole group to discuss.

Progress has been good so far. I will be (temporarily) leaving the country in the middle of next week. I should (and plan to), work on assignments while I am gone, and luckily I should be able to do CV work remotely, but between being gone, and making sure that I’m keeping up with other classes. I’m anticipating that I may not get as much work done next week. I’m hoping to find a suitable dataset and figure out how to train the existing model on this dataset. One that I have found is this: https://www.kaggle.com/datasets/arkadiyhacks/drinking-waste-classification , but plenty more are available, and I’d like to find a dataset that may include items that are often thought to be recyclable but actually aren’t (ie. plastic bags). If the Jetson camera arrives before I leave, I also plan to hook it up to get the Jetson to capture images.

I will make sure to communicate with my group if I have any issues.

Team Status Report for 9/28

As we are finalizing our design, the most significant risk we are facing is figuring out how our system will handle the items that the model does not recognize. If the model is unsure about the item and cannot classify it, we weren’t sure if we should make the user retake the picture of the item for a more accurate result, or if we should have a rejecting system that will not place the item into any of the bins. We decided that we will place the item into the trash bin because it is safer to have something mislabeled as trash rather than contaminating the recycling bin with non-recyclable items.

Since we have switched from four categories of recycling, to just recycling and trash, we decided that having a sliding gantry was no longer necessary, and chose to go with just a single door that tilts right or left depending on what the item is sorted as. This change allows us to minimize the amount of time it takes for items to be placed into their respective categories, and reduces possible mistakes that could happen when placing the item. The cost of the change would also decrease the overall budget, as we would not have to purchase a gantry.

We are also planning to add a new system to our project, which is a game where the bin tracks the number of items recycled in a week, and compares it to the past week, as well as a running average of items recycled per week. This tracking system will work similarly to the setting in iOS that tracks weekly screen time, and serves to track overall recycling “performance” for the household. The statistics will be accessible through a display on the bin that connects to our Arduino. Users will be able to control the display with buttons to view different statistics (ie. this week’s total, average recycling per week, % of items in bins that were recycled).

Some pictures of running YOLOv7 off the shelf:

It works quite well for some images/objects, like this image provided in the repository

But not so well in detecting recyclable materials. Fine-tuning will be needed.

Part A-Ashley Ryu

With respect to considerations of public health, safety, and welfare, our project aims to address these aspects by encouraging better recycling habits. Public health is addressed by reducing waste and limiting exposure to hazardous substances, which can be caused by improper management of trash. Safety is addressed by educating users about proper recycling, and reducing the risk of people handling hazardous materials improperly. For example, batteries should be handled separately instead of going into the trash/recycling bin, because they contain toxic chemicals that may cause fire. Our project aims to identify these hazardous materials in addition to trash vs. recyclable. Lastly, we address welfare by promoting a sense of responsibility and awareness of environmental issues through better recycling practices.

Part B- Mandy Hu:

We intend for our project to impact society by making recycling a more prominent part of people’s daily lives. When doing research for our project, we compared recycling in America, where most people don’t care about it too much, to South Korea, where sorting recycling is normalized, and people who don’t recycle are looked down upon by society. By keeping track of recycling statistics, people will become more aware of how much trash they are consuming on a daily basis, and become more diligent about what products they are purchasing. In addition, by simplifying the recycling process and making it quick and easy, we want to make it so that even people who are busy don’t have an excuse to not recycle.

Part C – Justin Wang:

Since our project aims to be a household product, it should not be incredibly expensive. To that end, we are using mostly parts that can be purchased off-the-shelf and are easily accessible. The custom parts of the project are mostly software and system design, and thus can easily be replicated. The design aims to minimize complexity and parts required, which should reduce purchasing and production cost and make assembly easier. The bulk of the price will come from the Jetson, which does cost a significant amount, but the simplicity of the rest of the design should ensure a scalable production process that would make EcoSort easily accessible to those who want it.

Team Status Report for 9/21

A potential issue that was raised was the accuracy of a YOLO model, which tends to be used for applications where speed is more important than accuracy. We are currently looking into the possibility of more powerful object detection models, but are leaning towards a more recent version of YOLO, since it is easier to set up and run, and its wide use in past 18500 projects means that it is proven to work. Aside from the CV, a potential area of concern is the system to drop items into the bins. We’ve come up with an arrangement which should simplify the system as much as possible, with a linear gantry only requiring control to a stepper motor which will “select” a bin, and a servo to drop it in. In this way we avoid complex robot kinematics. If we end up switching to a single-stream system (more on that in the next paragraph), we will have only two categories, meaning the system could be further simplified.

Based on the feedback we received from our proposal presentation, we are considering changing our design to sorting into two bins, recyclable vs. trash, instead of four. Pittsburgh (and many other American cities) use single-stream recycling, which means all recyclable materials are collected together then sorted at the processing plant. Some places may collect sorted materials (ie. CMU has individual bins for different categories of recycling), but some places may simply collect our sorted materials and dump them together. Sorting into two bins will likely not have much impact on our computer vision system, which will still recognize and classify objects as we planned, the only difference being that classified objects will then be grouped based on whether or not they are recyclable, instead of the specific recyclable material (ie. water bottle → recyclable, rather than water bottle → plastic). There isn’t too much change on the hardware side, the system design will remain the same, only we have fewer categories. The biggest changes are in the mechanical design, primarily in part selection and physical size. With half as many categories, our bin can be physically smaller, and we can use a smaller gantry for item placing.

Since this change doesn’t impact the overall design of our project, we don’t anticipate our schedule to be impacted too much. This could have the potential to push back the hardware assembly part of the project, since we may be delayed in ordering the gantry, but we still have to go through the design process before starting assembly, so the delay should be small, if there is one.

Justin’s Status Report for 9/21

Most of my time this week was spent preparing for the proposal presentation, with some research in the CV system, as well as looking into parts we may want to order. We received some feedback from the presentation about the YOLOv4 tiny model that we planned to use, with a suggestion that we could try a more recent YOLO version, or even a different model entirely that is more accurate (ie. Faster R-CNN). Since YOLO is rather easy to set up, I am currently working on setting up YOLOv7 to try training on a custom dataset. Our order for a Jetson from the 18500 inventory has been approved, so hopefully I can try deploying YOLO on it soon. I have found multiple datasets of recyclable materials (ie. https://www.kaggle.com/datasets/alistairking/recyclable-and-household-waste-classification), so we have plenty of training data available.
I also looked into serial communication between the Arduino and Jetson. Both devices have serial communication pins that can be used to talk with each other. It is also possible to connect the two with a micro usb cable, which will likely be our approach. Communication between the two devices is an important part of meeting our project’s use case, so it’s good to see that this is possible and has been done before.