This week, I have taken a dive into the pretrained ML model and the datasets that we are planning to use. I have written scripts to randomly partition our training datasets into subsets of training, testing, and validation and setting up our project structure accordingly on Google Colab notebook with Ting. Because we do not have access to the datasets that the pretrained model uses, we are still wrestling with where to place the “labeling” files that YOLO models need to detect where (multiple) items are exactly on an image. Right now, the model could run after classification, until it needs to assign labels to different objects.
After discussing with TA and our team, I have done more research about integrating Jetson, camera, and Arduino. One important progress is the object detection algorithm. We are basically implementing a “two-state” waiting algorithm to detect items placed and to wait for hands to be removed before running YOLO classification. The more detailed explanation could be found on the FSM graph I drew on our design presentation slides. The connection of Jetson Nano, camera (IMX219), and Arduino, as well as the actual software modules (Serial) to send and receive data, are all more settled now,
Besides that, I have ordered the items mentioned in last week’s report and have spent the later of this week working on the presentation slides. For ECE courses, the FSM concept I learned in 18240 was helpful for us to understand the “detect-and-wait” algorithm, as well as presenting. And ML (10301) was also crucial for understanding the ML model and dataset structure that we are using. Since we have worked with Arduino in 18220 too, that also guided our research well.
Colab Code: https://colab.research.google.com/drive/1inMk5b39dCFxSdzQ6_ReY75MkQTkyQ1Z#scrollTo=MsLqRPqxeAK5
Dataset Processing Script (will keep updating this for partitioning labeling files later):
https://github.com/AichenYao/capstone-scripts/blob/main/fileUtils.py