Ting’s Status Report 4/29/23
This week we worked with TA Samuel to come up with multiple options for our ML model which was detecting everything as trash. The options are listed in our team status report. As of right now, we are choosing the path of combining the old drinkables dataset into the new trashnet dataset. We are cleaning up the dataset, replacing the plastics in trashnet with the old dataset. This is because according to Pittsburgh recycling rules only bottles without caps can be recycled but this isn’t reflected in the new dataset. Over the weekend I will work on cleaning up the datasets so that next week we can train. If we can get this option to work early next week we will be ok, but otherwise we will be behind schedule.
The mechanics are on track to be finished, we just have to install the side lock servos and redo some of the legs. Vasudha and I cut the wood for the backboard and screwed it into the frame with the main servo. The frame of the whole system is now integrated together and we can test with the jetson. We discovered that angle brackets are needed on the legs and everything else we screw at a 90 degree angle because the legs started to warp in on itself. Next week we will work on these adjustments and finish up servo locks, and the mechanics will be finished.
Vasudha’s Status Report for 04/29/23
This week, I spent time working more on the mechanics and helping out on the software model side. On Sunday, I worked on assembling and testing out a temporary set up that would allow me to perform end to end testing from the capture to platform turning. This setup was not completely built, with temporary parts such as a taped paper platform and the camera capture mounted on a separate table, but showed me what the results would look like once the mechanics were fully built. I then ran multiple tests to get last minute records of our device’s time and accuracy performance to put in our presentation slides.
Later in the week on the mechanics side, I connected the acrylic door to the axle, powered the hardware, and ran tests to make sure everything was working. I also tested this set up with the newer clamps and ended up going with the older one as that was easier to integrate in terms of alignment. After testing the acrylic door, I found that it was definitely on the heavier side, and the clamp setup is not very stable. However, it works, and hopefully should be enough if we are not able to further develop the setup by the demo. I also worked with Ting to cut and build the camera mount, which was attached to the back of the servo frame, giving that frame some additional support at the same time. The overall structure of the device is not very stable, with the wooden parts often warping/disconnecting from the acrylic. Now that I have a better idea of how to assemble the frame, I have been thinking about redoing the connections and using bolts instead of screws to connect the wooden parts to the acrylic if there is time before the demo.
On the software side, our model implementation has not been performing well. Therefore, I spent some time helping out with debugging, looking for newer datasets, and sorting through them to integrate into our current implementation.
Schedule wise, I am somewhat on track in the sense that progress is being made, and our device is almost fully constructed and integrated even if the individual parts are not developed to their optimum levels. In terms of mechanics, the only crucial part that has not been implemented is the installation of the locks, which as a last resort solution we might end up taping to the system.
Given the remaining time in the course, next week I plan to try and finish off the required finishing touches for the mechanics, help out with the software, go back and see if any mechanic parts could be improved, and work on the final submissions.
Team Status Report for 4/29/2023
For this week, the biggest risk lies in our YOLO model. With the new dataset added and trained on top of the old weights, its classifications are mostly showing up as “Glass Bottle” while the class number is 0 (which maps to trash). We believe that there is both a mapping issue and a pure performance issue. During debugging, we did realize that the performance, based on the confusion matrix, is quite low and it is largely due to imbalance distribution in training images. After we changed the yaml file and fixed other missing steps, however, the training performance is still not good.
Right now we have listed a few options:
- Use original weights and CV detection; for all images recognized by CV but not YOLO, consider them as trash.
- Find another dataset containing both trash and recyclables, train weights entirely from that
- Find another dataset that contains both trash and recyclables, train on top of old weights.
- Use images from two existing datasets and manually rebalance each class and re-train.
Among these options, we are currently pursuing the first two at the same time. For #1, we also implemented an alternative; instead of 2-stage FSM, the system accepts user input through the keyboard and takes pictures as users ordered. Because 2-stage FSM is quite time-sensitive, having this prototype is smoother. For the demo, we could ideally perform both.
Simultaneously, we are also trying to find new datasets that contain all classes and have a more balanced distribution. We have already found a few potential datasets and are focusing on one for training now. If the performance is good (depending on confusion matrix), we could do trials using that model.
Earlier this week, we have done an all integrated testing with #1. The overall accuracy is about 60% from 20 tests run. Because YOLO is no longer constantly running, the model boot time is present for each classification attempt. The overall operation time is around 20 seconds now.
We made progress on the wooden frame, screwing in the back piece and platform that the camera sits on. We tested the height of the platform with respect to the main swing platform that the items will sit on. We discovered that we may have to redo some of the legs since we didn’t use angle brackets and they are angled at non 90 degrees now. Overall the main thing left to do with mechanics is to mount the side lock servos. We forgot to consider holes in the wood for this, but it seems fine to have the servos under the platform instead of inside the wood. For now the plan is to tape them on and test the hardware code integrated with the mechanics, then find a more permanent solution.
Our unit tests on the mechanics side included placing different objects on the platform to see if its weight would hold up, such as light pieces of trash to heavier things like phones. From this we discovered we need to have the platform be very balanced in terms of the materials we were using to attach it to the shaft, or else the platform will flip easily. For overall system testing, we discovered that we had to keep holding the trash up to the camera when we tried putting it on the platform, so when we cut the wood for the backing remeasured and made it slightly shorter to account for that.
For the hardware, unit tests were conducted for each component from the Arduino – each of the 3 servos for direction and amount of turn, Neopixels for color, and Piezo for audio output based on an input of 0 or 1 from the model. All binary output combinations were tested (00011011) as well as an initial alternate check. The components were then tested together with a similar input testing scheme. Next, this same test was done this time sending the inputs to the hardware from the Jetson. Lastly, after fully integrating sending the output from the model on the Jetson to the Arduino, the overall system was tested (ie. placing an object under the camera, running detection and classification, then checking the output). While at this point we were having issues with our model output, we made sure that the output being defined by the model was in fact resulting in the correct hardware reactions. Timing was measured for each of these tests along with accuracy.
Our model issue was found through model unit tests, as we attempted to test a plastic bottle, a metal can, a glass bottle, a plastic clamshell box, a plastic bag, and a napkin individually with our detection system and eventually our detection + classification system.
We don’t have any changes to our schedule. On the mechanics side we are on track to finish, with there being only one aspect left to implement. On the ML side, things are less certain, but we have many options in the courses of options that we can take so I believe we are still able to do well.
Aichen’s Status Report for 04/29/2023
Earlier this week, I rehearsed for the final presentation and wrapped up the last presentation before the demo. Simultaneously, we troubleshooted the weird performance of the most updated weights by tracing on Colab and added missing steps. After retraining, however, the performance is still quite disappointing. Following that, we came up with a few contingency plans as discussed in our team status report. I was focusing most on #1 and I implemented the alternative FSM design. I have tested both FSMs design first on local and then in integration with the team. The results of the testing are also covered in the team report. Right now, I am almost comfortable with that as the backup plan for the demo.
Schedule-wise, the ML section is behind our expectation as the second dataset seems to be highly imbalanced and there is almost no way to improve its performance based on our needs. Because the two previous datasets all took considerably long to set up, having to find a new dataset now is quite tight on time. But at least we have a working MVP.
Right after Monday, we will wrap on mechanics and determine if we want to pursue track #2 and #3 listed in the team report. Also, we would need to organize testing data and demo videos for the final few deadlines.
Ting’s Status Report 4/22/23
This week I worked on both the mechanic and software portion. On the mechanic side, I helped Vasudha with constructing the swinging door, we discovered that attaching the servos with the wood first may not have been the best idea, since the legs are curved in and we probably have to redo it when we attach the shaft.I think we should work with the lid and the bin directly instead of working with the legs for now. The platform itself seems too heavy, we are thinking of switching to plywood. Also the clamps we ordered were too big and we have to order smaller ones for the shaft to fit better. Schedule wise we probably have to figure out the swinging mechanism by the end of this week to be comfortable.
On the software side, I trained a trash dataset on top of our original model. Our class names were different for the same object for some of them, so the model was finding minute differences that it wasn’t supposed to learn. To fix this, Aichen wrote a script to change classes into 4 classes: trash class, 3 recyclable classes. Using this script, I retrained the trash dataset on top of our original model. We are testing this now, and currently it’s having trouble classifying recyclables as the right class. We will have to look into this after final presentations.
Team Status Report 4/22/23
This week on the software side, we were able to train a trash dataset containing 58 classes on top of our original model. When we tested it, it was giving us “no detections”, which is when we realized that we needed to change the class names to be both the same between both trash sets, and also we needed less classes since we really eventually only need two classes in the end, recyclable and non-recyclable. When there were two different names for the same thing, the model was learning minute differences that it should not have been. Aichen wrote a script that would change the label txt files that we were training with to have only 4 classes, trash and 3 types of of recyclables.
On the mechanical side, we started building a major portion of the structure, cutting out the door and making holes in the acrylic using the laser cutting machine, screwing in the wooden legs and frame together, then to the acrylic, installing the servo and the axle, and attempting to connect the door to the axle.
While doing this, we ended up facing a few issues. For one, the spots where we needed to screw overlapped, resulting in somewhat uneven placements of pieces. Additionally, when attempting to use the larger clamps to hold our acrylic door, it was hard to stabilize them, and from a first glance the door seems too heavy for the servo to handle. Therefore, as a first step we ordered new clamps. If after this change, adding the locks, and powering the servo it is still too heavy, we will order a new servo (by the end of next week). Due to all these uncertainties, the mechanical portion is once again one of our biggest risks. To overcome the stability portion, we are also planning on removing the somewhat unevenly installed legs, as the door can stand on the bin without warping. Once everything on the main frame is installed, we will attempt to put the legs back.
Currently, the ML performance also poses a risk for the whole classification system. As we added the new datasets and took the “normalized classes” approach trying to make the model not too fine-grained on bottles, the performance it has is quite unsteady on various classes of items. For items that it is identifying correctly, its confidence value is not as high as our 0.85 threshold. The mapping between the numbers and words (both to label classes) is also somewhat skewed. However, we believe this is still the move to take and we’ll dive deep into it right after the presentation. The good thing is that the model seems to do well at classifying trash vs. recyclables even though it might not always get the actual recyclable class right. (there are multiple recyclable classes and a single trash class in the training dataset)
We are somewhat behind on schedule, since our respective parts are not working as well as intended after some initial testing, and need more development work. Our plan is to scale back on the mechanical building, installing in terms of minimum priority to get the device working, then go from there, specifically having the door portion done by the end of next week, and start installing the rest of the hardware parts that weekend. On the ML side we are also somewhat behind schedule, since we are not exactly sure what the next steps are for fixing the classification situation, and we’ll have to take a look closer after the presentation is finished.
Vasudha’s Status Report for 04/22/23
This past week, I mainly worked on the mechanical portion of the project. First, I spent some time marking out my pieces. I marked out the door and places to make holes for the acrylic piece, then the specific measurements for the wooden legs and door frame. I then learned how to use the laser cutting machine and the drawing tool, and marked out my specific measurements for cutting the acrylic on it. I cut the acrylic in three different sessions after seeing what I needed to connect the parts, one initially for the door, one for the frame holes, and one for the door holes. I then got some of the wooden pieces cut with the help of the TechSpark staff, and then cut the remaining pieces by hand using a hand saw. I then learned how to use a power drill, drilled holes into the wooden parts, and then tried screwing them together. I also added corner braces to the connected legs, and then connected the wooden frames to the servo and to the acrylic frame. After this, I tried working on the door, and while alignment was okay with the larger clamps to hold the axle, it was quite loose and hard to stabilize (used a lot of popsicle sticks and tape). To account for this, we have ordered smaller clamps to see if it will be easier to install. Additionally, when trying to put the servo frame + legs + door + acrylic frame all together, the overall frame was not very stable, and seemed better off if initially assembled without the legs (which were uneven and short due to the last wooden piece holding them up not being installed yet). Due to this, along with the fact that I have yet to cut holes to install the locks, and other complications with alignment, I will probably have to uninstall the legs and focus on fixing the rest of the pieces.
Next week, I plan on finishing the basic lid assembly and start integrating it completely with the hardware so the overall testing of the mechanics can begin. This has already been done with the main servo, and I will focus on installing the locks and the door to the axle so everything can be tested together.
In terms of schedule, while I was initially on schedule, due to complications and the learning curve for the tools, I think I have now fallen behind in terms of the mechanical building. However, now with a better idea of what I need to do and the remaining parts needed, I think I can make significant progress before the final demo.
Aichen’s Status Report for 04/22/2023
We had a discussion before the weekend about “normalizing classes” so that multiple classes in the new dataset could have the same label based on a shared characteristic. For example, we assign paper bags, paper boxes, normal paper, etc. all to one class. This week, I have written the script to go through all labeling files of the old and new dataset to change their classes based on their original label while leaving the box coordinates unchanged. After testing that locally, Ting applied that to our Colab workspace and it processed all labeling files as we expected.
After training with the new dataset, Ting and I did some testing with the updated weights. Our camera, very unfortunately, broke again on Wednesday so we had to test with still images. Later, after quickly ordering a new camera usb and setting it up, the model could detect items correctly and could generally tell whether it is trash vs. recyclable. However, two major problems remain: 1. For actual items, the output confidence value is barely over 0.85 even though the classification is correct (our threshold for a positive answer is 0.85). 2. As there are multiple recyclable classes (single trash class), the output label is often wrong among recyclables (for example, metal might be identified as plastic).
Besides that, I have worked mostly on the presentation where we revisited and condensed materials from before and did more testing based on presentation outlines. Our schedule is slightly behind on ML training as we doubt the mapping between labeling classes (numbers) and actual classification results (word) might be skewed. Looking into that might take some time and we also need more integration testing. “FSM” is shown to be working but more testing needs to be done before the demo.
For next week, there are 3 major tasks for me:
- Finish the slides and do the presentation
- Improve ML accuracy and do more testing
- Help building the mechanics
Ting’s Status Report 4/8/23
This week we had our interim demo, and we showed our system which is a camera that captures the object, shows the classification on the monitor, then decides whether it sends a 1 or 0 to the Arduino which then turns servos and lights/sounds up accordingly. From the feedback we received, we worked on refining the ML as well as adding a CV script to check for object detection separate of YOLO. So our system was classifying bottles very well, but it had trouble detecting anything that wasn’t a bottle. This is because the model was only trained on the four types of drinking waste, and if it doesn’t consider an object a bottle then it won’t detect it at all. So I worked on finding a trash dataset that I could further train the model on. It was more difficult than expected, since we were looking for a very specific file structure for the images and labels to be in. So far I was almost able to integrate a trash dataset I found to start training, but all the labels are in json form and the code that is supposed to convert them to txt works in recreating the file but leaves the files empty, which is something that I will work on further to figure out. After this is fixed though I will be able to start training on the trash dataset. We will use this in adjacent to the CV script that Aichen is writing, which will give a score based on how different two images are.
This is an example of a classification instance that sends false to the arduino.
Next week I will keep working on training the model with the trash dataset and integrate it with the CV portion, which will be on schedule if i finish within the week.
Weekly question:
We did integration testing by using the hand santitizer bottles in Techspark, It worked well, but not when we tested a piece of supposed trash, like a plastic wrapper or a phone. We will continue testing with different kinds of bottles and common pieces to trash, to make sure that the dataset can handle more different kinds of waste.