For this week, the biggest risk lies in our YOLO model. With the new dataset added and trained on top of the old weights, its classifications are mostly showing up as “Glass Bottle” while the class number is 0 (which maps to trash). We believe that there is both a mapping issue and a pure performance issue. During debugging, we did realize that the performance, based on the confusion matrix, is quite low and it is largely due to imbalance distribution in training images. After we changed the yaml file and fixed other missing steps, however, the training performance is still not good.
Right now we have listed a few options:
- Use original weights and CV detection; for all images recognized by CV but not YOLO, consider them as trash.
- Find another dataset containing both trash and recyclables, train weights entirely from that
- Find another dataset that contains both trash and recyclables, train on top of old weights.
- Use images from two existing datasets and manually rebalance each class and re-train.
Among these options, we are currently pursuing the first two at the same time. For #1, we also implemented an alternative; instead of 2-stage FSM, the system accepts user input through the keyboard and takes pictures as users ordered. Because 2-stage FSM is quite time-sensitive, having this prototype is smoother. For the demo, we could ideally perform both.
Simultaneously, we are also trying to find new datasets that contain all classes and have a more balanced distribution. We have already found a few potential datasets and are focusing on one for training now. If the performance is good (depending on confusion matrix), we could do trials using that model.
Earlier this week, we have done an all integrated testing with #1. The overall accuracy is about 60% from 20 tests run. Because YOLO is no longer constantly running, the model boot time is present for each classification attempt. The overall operation time is around 20 seconds now.
We made progress on the wooden frame, screwing in the back piece and platform that the camera sits on. We tested the height of the platform with respect to the main swing platform that the items will sit on. We discovered that we may have to redo some of the legs since we didn’t use angle brackets and they are angled at non 90 degrees now. Overall the main thing left to do with mechanics is to mount the side lock servos. We forgot to consider holes in the wood for this, but it seems fine to have the servos under the platform instead of inside the wood. For now the plan is to tape them on and test the hardware code integrated with the mechanics, then find a more permanent solution.
Our unit tests on the mechanics side included placing different objects on the platform to see if its weight would hold up, such as light pieces of trash to heavier things like phones. From this we discovered we need to have the platform be very balanced in terms of the materials we were using to attach it to the shaft, or else the platform will flip easily. For overall system testing, we discovered that we had to keep holding the trash up to the camera when we tried putting it on the platform, so when we cut the wood for the backing remeasured and made it slightly shorter to account for that.
For the hardware, unit tests were conducted for each component from the Arduino – each of the 3 servos for direction and amount of turn, Neopixels for color, and Piezo for audio output based on an input of 0 or 1 from the model. All binary output combinations were tested (00011011) as well as an initial alternate check. The components were then tested together with a similar input testing scheme. Next, this same test was done this time sending the inputs to the hardware from the Jetson. Lastly, after fully integrating sending the output from the model on the Jetson to the Arduino, the overall system was tested (ie. placing an object under the camera, running detection and classification, then checking the output). While at this point we were having issues with our model output, we made sure that the output being defined by the model was in fact resulting in the correct hardware reactions. Timing was measured for each of these tests along with accuracy.
Our model issue was found through model unit tests, as we attempted to test a plastic bottle, a metal can, a glass bottle, a plastic clamshell box, a plastic bag, and a napkin individually with our detection system and eventually our detection + classification system.
We don’t have any changes to our schedule. On the mechanics side we are on track to finish, with there being only one aspect left to implement. On the ML side, things are less certain, but we have many options in the courses of options that we can take so I believe we are still able to do well.