Posted on

Aichen’s Status Report for 04/29/2023

Earlier this week, I rehearsed for the final presentation and wrapped up the last presentation before the demo. Simultaneously, we troubleshooted the weird performance of the most updated weights by tracing on Colab and added missing steps. After retraining, however, the performance is still quite disappointing. Following that, we came up with a few contingency plans as discussed in our team status report. I was focusing most on #1 and I implemented the alternative FSM design. I have tested both FSMs design first on local and then in integration with the team. The results of the testing are also covered in the team report. Right now, I am almost comfortable with that as the backup plan for the demo. 

Schedule-wise, the ML section is behind our expectation as the second dataset seems to be highly imbalanced and there is almost no way to improve its performance based on our needs. Because the two previous datasets all took considerably long to set up, having to find a new dataset now is quite tight on time. But at least we have a working MVP.

Right after Monday, we will wrap on mechanics and determine if we want to pursue track #2 and #3 listed in the team report. Also, we would need to organize testing data and demo videos for the final few deadlines.

Posted on

Aichen’s Status Report for 04/22/2023

We had a discussion before the weekend about “normalizing classes” so that multiple classes in the new dataset could have the same label based on a shared characteristic. For example, we assign paper bags, paper boxes, normal paper, etc. all to one class. This week, I have written the script to go through all labeling files of the old and new dataset to change their classes based on their original label while leaving the box coordinates unchanged. After testing that locally, Ting applied that to our Colab workspace and it processed all labeling files as we expected.

After training with the new dataset, Ting and I did some testing with the updated weights. Our camera, very unfortunately, broke again on Wednesday so we had to test with still images. Later, after quickly ordering a new camera usb and setting it up, the model could detect items correctly and could generally tell whether it is trash vs. recyclable. However, two major problems remain: 1. For actual items, the output confidence value is barely over 0.85 even though the classification is correct (our threshold for a positive answer is 0.85). 2. As there are multiple recyclable classes (single trash class), the output label is often wrong among recyclables (for example, metal might be identified as plastic).

Besides that, I have worked mostly on the presentation where we revisited and condensed materials from before and did more testing based on presentation outlines. Our schedule is slightly behind on ML training as we doubt the mapping between labeling classes (numbers) and actual classification results (word) might be skewed. Looking into that might take some time and we also need more integration testing. “FSM” is shown to be working but more testing needs to be done before the demo.

For next week, there are 3 major tasks for me:

  1. Finish the slides and do the presentation
  2. Improve ML accuracy and do more testing
  3. Help building the mechanics
Posted on

Aichen’s Status Report for 04/08/2023

For the first half of this week, I’ve worked on integration and testing of Jetson (and camera) and the Arduino. By demo, feeding real-time pictures to the YOLO model and parsing classification results and sending those to Arduino could all run smoothly. By far, the integration between software and hardware is pretty much done.

One major issue we’ve found is that due to the limited variety of our current training datasets, the model could barely detect and classify any item that is not in a shape close to bottles. As we are getting a lot of “no detections”, more than we expected, the major next step is to train with more datasets containing items of all other types. 

After the demo, I have also integrated the “FSM” implementation so that YOLO is only run when a significant change happens under the camera. This change is to save resources and avoid unexpected classification (i.e. when the platform is not “ready”). When I tested with my local laptop, the code works all right. There are still some capture issues when I tried it on Jetson on Friday, which might be due to the usb camera. A lot of other possibilities are thought out with Ting and Samuel, and my biggest guess now is that the “usb camera” needs a longer turnaround time in between two captures and I would use a wait loop before each capture instead of a fixed sleep time as of now. Before, this was not an issue as each capture is followed by an inference call which is way longer than the sleep times I use now. More debugging will be done on Sunday and Monday.

By next week, the FSM part should be fully integrated and I will also help Ting with training a new dataset. Training of the new dataset will hopefully be done within next week too.

Posted on

Team Status Report for 4/8/2023

This week, the main tasks we focused on were prepping a basic set up for our demo, starting the mechanical building, further training our model, and improving our detection algorithm. 

 

For our demo, we presented the overhead camera capturing images, our model detecting and classifying the objects in the image and then sending a value that the hardware circuit then acted upon. The hardware circuit included all the parts for the baseline, the neopixel strip, speaker, and servos. Some issues that we had during the demo was the Aruduino controlling the hardware kept switching ports, stopping the process midway, and the model not detecting anything that was not “recyclable” according to the model. Our plan for the next few weeks is to fix these two major issues by writing a script for the port switching issue and further training our model with a trash dataset so that we can detect both. On the side, we will start developing the mechanical building so that everything is hopefully built by the deadline. 

 

Our biggest risk is the mechanical portion once again. This is mainly due to the fact that all of the other requirements for our baseline have already been somewhat developed/have a basic implementation currently running, while the mechanical aspect has yet to be physically developed. However we started working on this later this past week once all the parts arrived, and our plan to deal with this risk is to in parallel develop the different remaining parts so that our overall integrated system operates better than during our internim demo. 

 

In regards to our schedule, we realized that we needed to train our model more and improve our detection algorithm, and so will need to allocate more time for that this week. However, in regards to the hardware, we are pretty much on track. We are on track for the mechanical building as per our latest schedule, and will need to make significant progress in the next one or two weeks to have a working system. 

 

Next week, our plan is to have our model detecting both recyclables and waste, and have the basic mechanical frame start taking shape so that we can start integrating the hw to it the following week. Integration of the FSM part, before running YOLO inference, is under testing now and will be working by next week.

 

While we did complete some basic tests and integration for the internim demo, as we enter the verification and validation phase of the project, we are planning to complete some more individual tests on each of our subsystems, and then test the overall integration. 

Posted on

Aichen’s Status Report for 04/01/2023

After some cross debugging (setting up backup jetson and trying different MIPI/USB cameras), we have finally sorted out the camera issue. On Sunday I was able to get the usb camera working and taking videos/images. On Monday, the whole team double checked and we decided to switch to usb cameras now. During labs, we have done measurements to determine the height and angle of the camera. Ting and I also worked on integrating the camera code with inference YOLO code (detect.py). We are able to capture images and trigger detect.py from the script that makes the camera captures images. We have done initial rounds of testing and the result makes sense. We are working on transforming image results to text files so that we could parse detection results with code and pipeline it to Arduino. We are able to find the classifications and confidence values and we could also direct them to text files. Right now as I am writing on Friday, we are still working on the logic of outputting a single result when there are multiple items identified. Overall, progress is much smoother this week compared to the last. We have also done initial mechanical measurements to better prepare for the interim demo.

Posted on

Aichen’s Status Report for 3/25/2023

This week, the absolute of my efforts went into setting up/debugging the camera. After being able to run inference on Jetson, we started setting up the camera (MIPI connected to Jetson), which could be “found” but not able to capture using different commands as well as cv2 module. After the first camera got burned, we switched to the backup camera which threw us into the same trouble. After a few hours of debugging, we decided to go with different routes now. First, I’m currently setting up the backup Jetson Nano to see if it could work with a USB camera. Second, if that works, we could also try it with the MIPI camera. If that still works, then it might be the Jetson Xavier’s problem and we would have to set up YOLO environment on Jetson Nano.

Schedule wise, the camera’s issues are a blocker for me, but we will do the best we could to have the camera subsystem functioning.

There’re some code updates in the github repo (the same link as posted before).

Besides that, I worked with Vasudha to get the “communication” between Jetson and Arduino working. I used a script in Python to send serial data (which will later be run on Jetson) to Arduino. After the whole team’s working, our HW circuit could behave based on the serial input it receives.

Posted on

Aichen’s Status Report for 3/18/2023

This week, after finishing the ethics assignment, I started setup of Jetson. There were a few hiccups such as missing the micro SD card reader and the very long downloading of the “image” (basically the OS for Jetson). However, my team and staff have all been very helpful, Ting and I were able to fully set up the Jetson by Thursday and we were able to run inference on Jetson on Friday.

I also helped during debugging as we migrate code to GPU on both Google Colab and HH machines. Those could all work now and training results proved to be exceeding expectations.

On Friday, we were able to connect the CSI camera to Jetson and I am currently working on capturing images using the CSI camera and integrating those with the detection code that I wrote earlier. On Monday, we would test this part on Jetson directly. If everything stays on track, Jetson would be fully integrated with the software subsystem by interim demo, which is about two weeks away from now.

For working with Jetson’s camera, I am planning to use the PyPI camera interface, implemented in Python3. This, I believe, is the best choice as our detection & classification code are all written in Python and Jetson naturally supports Python. Here’s a link to the module: https://pypi.org/project/nanocamera/

 

Posted on

Team Status Report for 3/18/2023

This week, we started with setting up YOLO running on GPU with both Google Colab and HH machines. Along the side, we have also started setting up Jetson. After the YOLO code could be run on Google Colab, we trained with 20 epochs and the results are shown with images in the end. In short, the code exceeded an accuracy of 90% for all 4 types without fine tuning. Using the weights learned by the training, we have done test runs of inference using Jetson which could also successfully run to finish. The reason we decided to pursue both tracks is that gcp is quite expensive where 10 epochs cost $40. Fortunately, we are able to reach a decent accuracy for now with 20 epochs.

For the images shown at the end of this report, the first one shows a sample result where an item is detected and classified. The text shows its classification result and the number is the confidence level of the classification. Our code could also run on the HH machines now, but because it is already working on Jetson, we decided to pause that route for now.

Our biggest risk still lies in our limited experience with mechanical design. However, we have communicated the concern with the staff in meetings and we have professors and TAs who could help us along the way, especially the woodwork. For now, we are working on design graphics with specific measurements and will share that with staff soon. Besides that, there are no design changes made.

Due to delayed shipping of parts, we are slightly behind on building the hardware part. Besides that, the software system and integration with Jetson is going fine. We are planning to finish implementation of detection & classification on Jetson in two weeks and make sure basic communication between Jetson and Arduino will work by the interim demo. For communication, Arduino needs to receive a boolean value in real time that Jetson sends. The Serial module (same name for Arduino and Python) would be used for that.

Posted on

Aichen’s Status Report for Mar 11th

Good news before spring break! After writing scripts to process the labeling files as Ting and I have discussed, our model is finally able to run through the whole training process! Using Google Colab without GPU, training for a single epoch took more than half an hour (while the default number of epochs is 150). Therefore, we have worked on using the HH 1305 machines as well as GCP credits to accelerate the training process. Once running on GPU, training will be done soon and we will use backup datasets to practice fine tuning.

As we were starting to set up Jetson, we realized that the Jetson Xavier NX (paired with a microSD memory card) is a better choice in terms of computing power than the Jetson Nano that we’ve chosen before. After a brief research, we decided to switch. After spring break, we will set up Jetson & camera and deploy image capture and (change) detection code to Jetson.

Besides that, this week I have used the most of our time on the design doc. Alas, simply moving stuff from a google doc to the latex version took so long. On Friday (Mar 3rd) as I am writing now, I have just finished 2 hours of proofreading and reformatting and we are finally ready to submit.

In the coming weeks, setting up Jetson and integrating what we have now will be a major task. As none of us has worked with Jetson extensively before, there would be challenges. Either way, I am excited to transform work from computers to reality.

 

Link to code (update mostly on scripts organizing training data):

https://github.com/AichenYao/capstone-scripts

 

Posted on

Aichen’s Status Report for Feb 25th

I researched about detecting image change and implemented an image detection algorithm using the CV2 module (open-cv library). I took a few images on the spot where each pair captures the same objects, one with hands and one without hands. I ran the script and recorded their matching error and runtime. The runtime is 0.02s, much lower than what is expected for YOLO, which is exciting. More experiments need to be run to set a reasonable threshold for declaring an “image change”. The code and sample images are linked below.

Besides that, most of the work done this week is for the design review document due next week. I have solidified the introduction, use case requirements and design requirements and written them on the design doc. I have also updated the design of the FSM by setting a timeout at the waiting state where we wait for the second, consecutive change. I am also writing about that in design trade studies as well as system implementation.

Earlier this week Ting and I met again to talk about the ML model and we are understanding the labeling and images structure better. We are planning to add to the “partition” script and partition labeling files to train, test and validation folders and run the model again. Hopefully it could run until fine tuning this time. This should be achieved before the spring break. 

 

Image detection code & sample pictures (results logging and script instructions can be found in the code file):

https://github.com/AichenYao/capstone-scripts/tree/main/image_detection