Last Sunday, I worked on finalizing our design slides for the presentation. Later in the week on Thursday I picked up the ultrasonic sensor that was delivered. I wrote a piece of code in Arduino that detects the distance of the nearby object with the ultrasonic sensor, which can be incorporated in our product later. For now, it prints the distance to the object on the console but later this would trigger the camera to capture an image. I simulated this with the Tinkercad that I was using last week, and once I have all the physical components to connect it with the Arduino I will test to see how accurate our ultrasonic sensor is with the object detection. I also looked more into setting up serial communication between Arduino and Jetson, in order to implement the image capture triggering feature. And since our camera is now delivered (which is not picked up yet), I looked into how the Jetson can capture the image when it receives the signal that an object is detected. I found some Python code online that does a similar job, which does not look too complicated. By next week, along with working on our design report, I hope to start setting up all the hardware components and perform some basic tests.
Mandy’s Status Report for 10/5
This week, I spent most of Sunday rehearsing for the design presentation. I also did more research about the types of display screens that we could use, and decided to switch to an OLED display screen instead of the arduino display shield, because it seems to be more compatible with our project. I also ordered the screens, and started writing out the code for the displays, although I have been unable to test it since the screens have not arrived. In addition, I have started working on the design document by writing out information from our presentation and compiling it into the proper format.
I am currently on schedule for my part of the project. The schedule has me completing this display portion of the project by the end of next week, which I am currently working on. However, since I sent in the request for the screens late, I’m don’t think that they’ll arrive until Thursday of next week, and I am unsure of what kinds of complications will come up once I actually test my code with the screens. If I am unable to finish by the end of the week, I will catch up by working on it during spring break.
In the next week, I hope to have a fully functioning display screen that works with stubbed responses from the jetson.
Team Status Report for 10/5
This week, we presented our design review on Monday and started working on our design report.
One risk that we are currently facing is the YOLOv7’s ability to recognize recyclable objects. While we haven’t started training the model yet, we realize that there are many different possible recyclable objects that we have to take into consideration, especially for types of paper. We are mitigating this risk by finding a dataset that covers all of the different types of materials that we need to recognize.
After doing more research into display screens, we realized that the display screens that we wanted to use were very small, and may not be able to fit all of the information. Instead, we plan to use multiple displays instead and piece them together, as we still think that this specific display screen is the most compatible with the rest of our project. This would not incur a lot of new costs, as the screens that we originally wanted to purchase came with multiple screens, and the only extra costs would come from wires to connect the extra screen.
There have been no changes made to the schedule this week. We are on time and wrapping up the research that each of us have been doing. Next week, as soon as we receive all the parts that we ordered, we plan to start building our hardware components and test the CV.
Justin’s Status Report for 10/5
I spent time in the beginning of the week working on the design presentation, which went well, and we got some useful feedback. I have landed on a suitable dataset for our project: https://www.kaggle.com/datasets/arkadiyhacks/drinking-waste-classification. This dataset not only contains thousands of images to train on, but the images show the images as they might appear when captured by our camera (ie. the item is photographed from above), rather than a stock image. Unfortunately, I was out of Pittsburgh in the second half of the week to attend a wedding, so I was not able work with the Jetson and camera (which arrived after I left), so next week I am hoping to hook up the camera to the Jetson, load YOLO (off the shelf), and test out that setup. I will also start setting things up to train YOLO with a custom dataset. Luckily YOLO has many resources online, I’ve already watched some videos, and it should hopefully be easy to set up. I will also be working with my team on the design review. Being away from Pittsburgh has delayed my progress beyond what I initially hoped, and I anticipate the design report and midterms to take time next week, so I am also planning to work on this project over fall break, bringing the camera and Jetson back home with me if necessary (I should not need any other hardware to work on the CV).
Ashley’s Status Report for 9/28
This week I worked with my teammates on the design presentation and we made some changes to the design. The hardware part will mostly remain the same, so I continued with my research on the hardware components I will be working on. We finalized which model we wanted to use, and placed the orders for the ultrasonic distance sensor and the Arducam camera.
I also found a software called Tinkercad that allows you to simulate circuits. I saw that it includes the exact ultrasonic sensor that I wanted to use, and it allows you to write code. I played around with it a little and hopefully this will help me test and verify that it works before I start working with the actual components. Tinkercad lets me do something like this:
I am a little bit behind schedule, due to many overlapping deadlines and upcoming exams. I have done research on how the components work and what pieces of code I would have to write, but I don’t have any solid components yet. In the next week, since I expect to have a bit more time, I hope have a more solid design of the hardware and start setting them up once the orders arrive.
Mandy’s Status Report for 9/28
This week, I started by preparing for the design presentation. Since we made a lot of changes to our project, we had to add these changes to the presentation. In addition, since I am presenting for the design presentation, I spent the majority of our time writing out a basic script of things that I need to say, and spent time rehearsing it.
Since we also added a new section to our project, I spent some time researching equipment and materials that we could use, including the type of display. Originally, I thought about using a raspberry pi, since raspberry pis have more compatible display screens. However, I decided on using an Arduino GIGA display shield, as it is a display that is compatible with Arduino, which we were already using to control the door. In addition, it is a touch screen display, which can allow users to interact with it, making it feel more like a game.
I am slightly behind on my schedule, since we just decided to add an extra part to our project a few days ago. I have been able to do some basic research and brainstorming about it, but in the following week I must finalize the equipment that I need to use, and place orders for it within the first few days of the week.
In the next week, I plan to place orders for the items that I need, draw out specific diagrams for what the display would look like and what it kinds of statics need to be shown. I will also begin the coding for the assignment.
https://store.arduino.cc/collections/displays/products/giga-display-shield
Justin’s Status Report for 9/28
This week I was able to get YOLOv7 up and running on my computer to test its performance on some sample images. This was purely to gauge the model out of the box, and as expected, the classification was not very good on recyclable materials.
This was expected, and confirms our need to find a suitable dataset of recyclable materials to fine tune the model. Our original plan was to train our model on some labeled recyclable materials (ie. aluminum can, plastic bottle), and have some sort of lookup table for each classification to determine if the item is recyclable. This could potentially lead to cases where an item isn’t recognized with high confidence, and we would have to make a decision on what to do in such a case. An alternative we discussed was customizing our dataset so items are labeled as “recyclable” or “non-recyclable”, but currently I am leaning towards the original idea, with unrecognized items being simply classified as non-recyclable.
Aside from CV work, most of my time was spent on the design presentation, and working out the details of our new game system. The game system won’t interact directly with the CV, but adding a new system to the project felt like something for the whole group to discuss.
Progress has been good so far. I will be (temporarily) leaving the country in the middle of next week. I should (and plan to), work on assignments while I am gone, and luckily I should be able to do CV work remotely, but between being gone, and making sure that I’m keeping up with other classes. I’m anticipating that I may not get as much work done next week. I’m hoping to find a suitable dataset and figure out how to train the existing model on this dataset. One that I have found is this: https://www.kaggle.com/datasets/arkadiyhacks/drinking-waste-classification , but plenty more are available, and I’d like to find a dataset that may include items that are often thought to be recyclable but actually aren’t (ie. plastic bags). If the Jetson camera arrives before I leave, I also plan to hook it up to get the Jetson to capture images.
I will make sure to communicate with my group if I have any issues.
Team Status Report for 9/28
As we are finalizing our design, the most significant risk we are facing is figuring out how our system will handle the items that the model does not recognize. If the model is unsure about the item and cannot classify it, we weren’t sure if we should make the user retake the picture of the item for a more accurate result, or if we should have a rejecting system that will not place the item into any of the bins. We decided that we will place the item into the trash bin because it is safer to have something mislabeled as trash rather than contaminating the recycling bin with non-recyclable items.
Since we have switched from four categories of recycling, to just recycling and trash, we decided that having a sliding gantry was no longer necessary, and chose to go with just a single door that tilts right or left depending on what the item is sorted as. This change allows us to minimize the amount of time it takes for items to be placed into their respective categories, and reduces possible mistakes that could happen when placing the item. The cost of the change would also decrease the overall budget, as we would not have to purchase a gantry.
We are also planning to add a new system to our project, which is a game where the bin tracks the number of items recycled in a week, and compares it to the past week, as well as a running average of items recycled per week. This tracking system will work similarly to the setting in iOS that tracks weekly screen time, and serves to track overall recycling “performance” for the household. The statistics will be accessible through a display on the bin that connects to our Arduino. Users will be able to control the display with buttons to view different statistics (ie. this week’s total, average recycling per week, % of items in bins that were recycled).
Some pictures of running YOLOv7 off the shelf:
It works quite well for some images/objects, like this image provided in the repository
But not so well in detecting recyclable materials. Fine-tuning will be needed.
Part A-Ashley Ryu
With respect to considerations of public health, safety, and welfare, our project aims to address these aspects by encouraging better recycling habits. Public health is addressed by reducing waste and limiting exposure to hazardous substances, which can be caused by improper management of trash. Safety is addressed by educating users about proper recycling, and reducing the risk of people handling hazardous materials improperly. For example, batteries should be handled separately instead of going into the trash/recycling bin, because they contain toxic chemicals that may cause fire. Our project aims to identify these hazardous materials in addition to trash vs. recyclable. Lastly, we address welfare by promoting a sense of responsibility and awareness of environmental issues through better recycling practices.
Part B- Mandy Hu:
We intend for our project to impact society by making recycling a more prominent part of people’s daily lives. When doing research for our project, we compared recycling in America, where most people don’t care about it too much, to South Korea, where sorting recycling is normalized, and people who don’t recycle are looked down upon by society. By keeping track of recycling statistics, people will become more aware of how much trash they are consuming on a daily basis, and become more diligent about what products they are purchasing. In addition, by simplifying the recycling process and making it quick and easy, we want to make it so that even people who are busy don’t have an excuse to not recycle.
Part C – Justin Wang:
Since our project aims to be a household product, it should not be incredibly expensive. To that end, we are using mostly parts that can be purchased off-the-shelf and are easily accessible. The custom parts of the project are mostly software and system design, and thus can easily be replicated. The design aims to minimize complexity and parts required, which should reduce purchasing and production cost and make assembly easier. The bulk of the price will come from the Jetson, which does cost a significant amount, but the simplicity of the rest of the design should ensure a scalable production process that would make EcoSort easily accessible to those who want it.
Ashley’s Status Report for 9/21
This week, I met with the team on Friday to reflect on our proposal and project design. I did some research on the parts that we need to buy and the software that we will be using. I looked into the ultrasonic sensors that could be used to detect the object, and read through some Arduino code tutorials. I found some parts online that we could buy for the mechanical part of our system. I am on schedule for now, and for next week, I will continue with my research and hopefully finalize the items that we will be buying. I will also work with my teammates on the design presentation.
Team Status Report for 9/21
A potential issue that was raised was the accuracy of a YOLO model, which tends to be used for applications where speed is more important than accuracy. We are currently looking into the possibility of more powerful object detection models, but are leaning towards a more recent version of YOLO, since it is easier to set up and run, and its wide use in past 18500 projects means that it is proven to work. Aside from the CV, a potential area of concern is the system to drop items into the bins. We’ve come up with an arrangement which should simplify the system as much as possible, with a linear gantry only requiring control to a stepper motor which will “select” a bin, and a servo to drop it in. In this way we avoid complex robot kinematics. If we end up switching to a single-stream system (more on that in the next paragraph), we will have only two categories, meaning the system could be further simplified.
Based on the feedback we received from our proposal presentation, we are considering changing our design to sorting into two bins, recyclable vs. trash, instead of four. Pittsburgh (and many other American cities) use single-stream recycling, which means all recyclable materials are collected together then sorted at the processing plant. Some places may collect sorted materials (ie. CMU has individual bins for different categories of recycling), but some places may simply collect our sorted materials and dump them together. Sorting into two bins will likely not have much impact on our computer vision system, which will still recognize and classify objects as we planned, the only difference being that classified objects will then be grouped based on whether or not they are recyclable, instead of the specific recyclable material (ie. water bottle → recyclable, rather than water bottle → plastic). There isn’t too much change on the hardware side, the system design will remain the same, only we have fewer categories. The biggest changes are in the mechanical design, primarily in part selection and physical size. With half as many categories, our bin can be physically smaller, and we can use a smaller gantry for item placing.
Since this change doesn’t impact the overall design of our project, we don’t anticipate our schedule to be impacted too much. This could have the potential to push back the hardware assembly part of the project, since we may be delayed in ordering the gantry, but we still have to go through the design process before starting assembly, so the delay should be small, if there is one.