Posted on

Team Status Report for 02/18/23

This week our team has explored materials and measurements of the mechanical parts including the bins (one big bin and two small “sub-bins” for recyclables and trash). We updated our trapdoor design from a single door operated by an actuator to a swinging door with two openings, one for recycling and one for trash. This way, instead of the user having to wait for the device to respond and having to take the waste off the platform if not recyclable, the device can educate the user with its alerts while still taking care of the organization, making its use easier and more practical overall. Since we were still in the design stage and hadn’t ordered parts for the old design, there wasn’t much of a cost to account for due to this change. From there, we have designed and prototyped where and how to connect each part including the platform and the servos that controls the closing and opening of the lid. We have also added the idea of using ultrasonic sensors to detect if the bins are full. The fill indicator is not a part of our MVP but we will look into how the sensors could read and transfer the capacity status. Simulations of arduino are performed on tinkercad and more details could be found on our presentation slides. 

Our biggest risk right now that could jeopardize success is the operation of the ML waste classification model, since we are having some initial set up setbacks and the model not working could mess up the entire classification part of our project, which is one of our biggest requirements. In terms of software, we have delved into the ML code and tried to run it with our drinking waste classification datasets. Right now the training could be run halfway and we still need to adjust our dataset structure. Since the size of datasets is huge and Google colab takes fairly long to run, this part is taking more time than expected. However, as we definitely have a better understanding of the model and we have written scripts to deal with raw data and set up our project structure on colab, we are able to deal with the risks without it not leaving us too much behind and we are confident we can get this working soon. As a contingency plan, we do have other backup models/datasets from our research earlier this semester that we can switch in if the current set up does not work out. 

Over the weekend we are ordering more mechanical parts such as the bins, servos, and lids; and after the design report, we could transfer code from tinkercad to the devices and integrate them.

So far, HW wise we are on/ahead of schedule (design + simulation complete), however, the delayed part ordering may set us back a bit this upcoming week. Software wise we are a little behind due to the difficulty in debugging the training model, and therefore will focus more of our efforts here to get back on track.

Some principles of engineering, science and mathematics that our team used to develop the the design solution include:

  1. ML (detection and classification algorithms): ML Model
  2. FSMs, engineering design charts: design diagrams
  3. basic mathematics/geometry: mechanical measurements and camera distance calculations (focal point + triangle calculations)
  4. Algorithms, Programming, and Control flow: training model debugging, hardware programming for simulation
  5. Circuit design: Arduino + components hardware design for speaker, neopixels, servos, etc.
Posted on

Aichen’s Status Report for Feb 18th

This week, I have taken a dive into the pretrained ML model and the datasets that we are planning to use. I have written scripts to randomly partition our training datasets into subsets of training, testing, and validation and setting up our project structure accordingly on Google Colab notebook with Ting. Because we do not have access to the datasets that the pretrained model uses, we are still wrestling with where to place the “labeling” files that YOLO models need to detect where (multiple) items are exactly on an image. Right now, the model could run after classification, until it needs to assign labels to different objects.

After discussing with TA and our team, I have done more research about integrating Jetson, camera, and Arduino. One important progress is the object detection algorithm. We are basically implementing a “two-state” waiting algorithm to detect items placed and to wait for hands to be removed before running YOLO classification. The more detailed explanation could be found on the FSM graph I drew on our design presentation slides. The connection of Jetson Nano, camera (IMX219), and Arduino, as well as the actual software modules (Serial) to send and receive data, are all more settled now,

Besides that, I have ordered the items mentioned in last week’s report and have spent the later of this week working on the presentation slides. For ECE courses, the FSM concept I learned in 18240 was helpful for us to understand the “detect-and-wait” algorithm, as well as presenting. And ML (10301) was also crucial for understanding the ML model and dataset structure that we are using. Since we have worked with Arduino in 18220 too, that also guided our research well.

 

Colab Code: https://colab.research.google.com/drive/1inMk5b39dCFxSdzQ6_ReY75MkQTkyQ1Z#scrollTo=MsLqRPqxeAK5

Dataset Processing Script (will keep updating this for partitioning labeling files later):

https://github.com/AichenYao/capstone-scripts/blob/main/fileUtils.py

 

Posted on

Aichen’s Status Report for Feb 11th

For the early half of this week, I have worked on the proposal representation and I was assigned to use case & use case requirements. I went through previous communications with the staff team, as well as our proposal planning doc and abstract doc to further clarify our project scope. I first wrote down all use case requirements associated with each technical component in a planning document and created slides from there. The detailed requirements are used as speaker’s notes for our actual presentation. Over the weekend, I met with my team to polish slides and rehearse our proposal presentation.

After the presentation, I have discussed with our team and TA about whether the goal is to identify single or multiple objects at once. I spent two hours on Tuesday night watching tutorials of YOLO, ResNet, etc. and reading documentation about pre-trained CV models that our team will potentially use (references attached at the end). 

After Wednesday’s class, we have decided to also start researching hardware to confirm parts that we need to order. I have looked into data transfer between Jetson and Arduino and the messaging read (Serial) mechanism of Arduino. I have also watched setup guides of Jetson, as well as connecting an Rpi camera with Jetson. At this point as I am writing on Friday, I have already ordered Jetson from the class inventory and we will request cameras soon. We are most likely getting an Rpi camera that is known to be compatible with Jetson. 

Tomorrow (Saturday, 2/10) I am planning to spend 2 more hours looking into the pre-trained model and hopefully run it on datasets that we have already found.

Here is a list of guidances I believe we will use after the hardware parts arrive:

Jetson Nano (with camera): https://automaticaddison.com/how-to-set-up-a-camera-for-nvidia-jetson-nano/#:~:text=Connect%20the%20Camera%20to%20the%20Jetson%20Nano,-Make%20sure%20the&text=Grab%20the%20camera.,away%20from%20the%20heat%20sink.

Arduino (data transfer):

https://www.programmingelectronics.com/serial-read/#:~:text=USB%20is%20one%20of%20the%20most%20common%20ways%20to%20transmit,built%2Din%20Arduino%20Serial%20Library.

(Multiple) Object Detection CV documentation:

https://medium.com/analytics-vidhya/object-detection-with-opencv-step-by-step-6c49a9cc1ff0