Posted on

Vasudha’s Status Report for 02/25/23

This past week, I worked on finishing up the slides for the design presentation, adding last minute information regarding the hardware and mechanical design that I had been working on and helped Ting practice (since she was the speaker for this phase). After this, I updated our materials list with specifics needed after some design changes (ex. neo-pixel strip instead of singular pixel, dowel structure, etc.). I also took time this week to look into the design report, taking the presentation feedback into consideration to better define the design and update the diagrams accordingly. I looked into fine tuning the swing door design support of the swing door after realizing our last plan had the door being held up and controlled solely by a single servo. After looking into potential axle setups and talking to our TA about how we could implement this, we decided to go with a dowel supporting the ends of the door, with other designs in mind in case this support is not enough (ex. gluing an axel to the bottom of the door and having it lie on a loop connected to each side of the main frame, getting a new lid material that is thicker so that we can drill further into it, etc.). I updated the circuit simulation to reflect the hardware component changes. Additionally, I tried planning out a more detailed schematic regarding mechanical building by drawing out where connections with the hardware would need to be made, and then looking up possible implementations (screw placement, frame dimensions, placement, etc.).

As mentioned last week, my progress is now slightly behind on the hw/mechanical side due to the fact that our materials order was placed quite late. However, to account for this, I spent more time on simulating, improving our current design, and the design report to save future time and be more prepared for when the parts actually arrive. With these proactive actions, hopefully the time spent for building will be reduced so that we can still stay on track. 

In regards to actions for the next week, since our team has been struggling on the software side, I plan to help debug the model set up so that progress can be made while waiting for materials to arrive. Additionally, I plan to finish off the report early on in the week so that more time can be spent on the technical end before Spring Break.

Posted on

Team Status Report for 2/25/23

This past week, our team worked on fine tuning our design, firstly for the presentation slides due last Sunday, then further details based on questions that came up during the presentation and feedback that we received afterwards. 

Looking at our current status in terms of progress and design improvements, the most significant risk that we have as of now is once again the mechanical operation of the swing trap door, as if this operation does not work, it takes away a large portion of our project goals, which advocates for a hands free recycle sorting system. Since we don’t have much experience with mechanical building, it has been a little difficult figuring out what exactly we need to make sure the door operates properly. However, to mitigate this risk, we have worked towards improving our design over the last few weeks with the help of our TA(Samuel) and Professor Fedder. We have multiple contingency plan designs such as a single open/close door or a non automatic but locked flap door rather than a swing door that while do not completely achieve our goal, can serve as a backup in case the current design doesn’t work out. 

After ordering parts, researching, and further detailing the swing door (ex. looking at measurements, operation, etc.), a flaw we found with the design we had lastly settled on is the lack of support for the door itself, which was being held up solely by the servo in charge of turning the door. Since we had already ordered other parts including the acrylic for the door itself, our solution had to account for this. Specifically, while we thought adding an axle would fix our issue, a problem with this solution is that the door material might be too thin to drill a hole and insert an axle all the way through. Therefore, the solution we have currently decided upon is drilling a small hole into the side of the door in which a small dowel on which the door can rotate upon while still having some support. Since we haven’t started building yet and the change only builds upon our current design, there isn’t too much of a cost except for the addition of the dowel itself. 

For the FSM, we implemented a detection algorithm (not YOLO which is only for classification) and did test runs. The code as well as logging of runtime could be found at the link under Aichen’s report. Overall, the runtime is much lower than what is expected of YOLO and we will do more test runs to set a threshold for declaring image change.

We also took in Samuel’s advice for setting a timeout and updated our design based on that. So right now, if no image change is detected at the waiting state for a certain period of time (set to be 5 seconds now), it will go back to the initial state.

Jetson has arrived this week; once the camera is also here, we are excited to set up continuous image capture and perform detection. We will then also verify if the camera is set up properly to capture the whole platform. 

We also decided on a number for the confidence threshold, which is referring to the number given by YOLO algorithm to show how confident it is that the object is what it says it is. This was a question that was brought up after our design presentation. We wanted a number that would balance not contaminating the recycling, while also not throwing too many items out that could have been recycled. We decided that .85 is a number that would allow for a good balance of these considerations. 

In terms of schedule, since we haven’t received our hardware and mechanical materials yet, we have not started to build. To account for this setback, we proactively simulated our hardware and fine tuned the design of the structure overall to make it easier to set up. We will be receiving a lot of things next week I expect, since we placed our orders this past week, so we look to start building this upcoming week. We are still behind schedule in terms of the ML as we were planning to start fine tuning this past week. We seem to still have some issues regarding debugging the training model, but we will take the next few weeks (before and after spring break) to really focus up on this aspect and get the code running and fine tuning started. After the code runs we don’t think it’ll be too much to train. 

To summarize our schedule updates:

  1. Moved construction of hardware from the week of 2/20 to 2/27 (shortened the construction time since a lot of the work has been covered through simulation).
  2. Moved construction of mechanical parts to the week of 3/13.
  3. Moved report writing from the week of 2/27 to starting the week of 2/20 and fine tuning the week of 2/27.
  4. Stretched the debugging + training of the ML model until the end of the week 3/13 at the latest.
Posted on

Aichen’s Status Report for Feb 25th

I researched about detecting image change and implemented an image detection algorithm using the CV2 module (open-cv library). I took a few images on the spot where each pair captures the same objects, one with hands and one without hands. I ran the script and recorded their matching error and runtime. The runtime is 0.02s, much lower than what is expected for YOLO, which is exciting. More experiments need to be run to set a reasonable threshold for declaring an “image change”. The code and sample images are linked below.

Besides that, most of the work done this week is for the design review document due next week. I have solidified the introduction, use case requirements and design requirements and written them on the design doc. I have also updated the design of the FSM by setting a timeout at the waiting state where we wait for the second, consecutive change. I am also writing about that in design trade studies as well as system implementation.

Earlier this week Ting and I met again to talk about the ML model and we are understanding the labeling and images structure better. We are planning to add to the “partition” script and partition labeling files to train, test and validation folders and run the model again. Hopefully it could run until fine tuning this time. This should be achieved before the spring break. 

 

Image detection code & sample pictures (results logging and script instructions can be found in the code file):

https://github.com/AichenYao/capstone-scripts/tree/main/image_detection

 

Posted on

Ting’s Status Report for 2/25

This week i presented on the design presentation. one question that was raised was about the confidence threshold. The YOLO algorithm will give a confidence calculation, from 0 to 1 determining how likely it is that the object is actually what it says it is. We decided that we want a 85% threshold. Aichen and I met up and we kept looking into the file structure of the YOLO code, and found that the given YOLO_imgs text files in the dataset are actually the labels that the train.py code is looking for. We will have to modify the script that segmented the dataset images into train/val/test and also segment the corresponding label text files. We have ordered all the materials and some of them have started arriving. We will definitely start building next week as our materials start arriving. We will also be spending the week working on the design document. I think we are still behind on the ML side, as our file structure is still having issues. But there has been development in terms of the detection portion of the software. 

Posted on

Vasudha’s Status Report for 02/18/23

This week, I mainly focused on the hardware and mechanics side of the design of our project. Specifically, I fine tuned our trapdoor design, planning out specific measurements for the bins, lid, trapdoor/platform door, camera-jetson frame, etc. I went through multiple iterations of the trapdoor design, some as seen below:

https://docs.google.com/drawings/d/1vTtQRCfDoRzRX9KN5lw0bdCIRwlpby7mf9q2U1XyG2s/edit?usp=sharing 

https://docs.google.com/drawings/d/113cMvrOCkN0lnGb3p7Mp-p-1ntxNbYKT5Z-GDhDywOc/edit 

https://docs.google.com/drawings/d/1SFQX2WVGmJm90RJwDvzh1R_OuZj7p4rXcLf2yBIn_wA/edit?usp=sharing 

https://docs.google.com/drawings/d/1Q1V-tIaLU_egyhBnFnMCREbfhr12IFpmU5ED0h5apkc/edit?usp=sharing 

The first design above was the design as of last week, involving an actuator controlling a single lid for recycling. After some research and suggestions from the TA, we decided to switch to a swinging platform controlled by a servo. One servo will be connected to the swing door, controlling the degree of turn towards each bin. Then, on each side of the platform door, a servo will be attached to the lid frame and act as a sort of lock to support the platform to make sure it cannot be forced into the wrong direction. This setup will then allow us to go back to our initial idea of having a bin self-separate incorrect recyclables instead of the user having to wait and remove their waste from the platform if not recyclable. Designs 3 and 4 provide specific measurements and perspectives for the initial swing door design. Side by side, I developed a list of materials needed based on the design changes, and looked up different material types and sizes based on the needed measurements. If only a certain measurement was available (ex. Recycling bins only came in certain sizes), I scaled up/down the other parts of the design to account for this. Updated list of materials here:

https://docs.google.com/document/d/1g7h4UxZviUhH3USrjvgQkKy8eIstabp8K-l7TRypfCo/edit?usp=sharing 

I worked on the arduino circuit design for all the hardware components – neopixels, servos, piezo speaker, and proximity sensors. I used tinkercad to create the design and programmed the simulation with the basic control flow of the circuit based on the output that will be sent to the Arduino from the Jetson. Link to design + simulation code: https://www.tinkercad.com/things/lqaXO4oCuXd?sharecode=SEZayoRTbx7Bb38PGnaV4sheBsci4fS94ztYQ4ntChs 

Lastly, I worked on the design presentation, creating schematics and updating the HW side of the design.

Diagrams worked on: https://drive.google.com/file/d/15j3AFNqRnFr1QK4KOhniSD_vvZ2-SMlD/view?usp=sharing 

While progress so far has been on schedule, with a solid mechanical and circuit hardware design completed, there might be a slight setback due to the fact that not all the materials have been ordered yet. Since our schedule originally had us start assembling the hardware next week, this might be difficult to begin until all parts have arrived. 

Therefore, next week I will work on finalizing design (ex. working out specifics regarding cutting the acrylic for assembly, where to screw parts in, adding more details to the circuit simulation etc.), the report, and help out more on the software side (ex. debugging the training model) until the parts arrive to still make progress and be prepared for when the parts actually arrive. 

Some of the ECE courses that covered engineering, science, and mathematics principles I used for the design include 18-100 and 18-220 for the arduino circuit design. In addition to this I used simple geometry/math topics, and my team used concepts from ML and Programming courses to work on the software side of the design (ie. training model, arduino programming).

Posted on

Ting’s Status Report for 2/18

We continued debugging the object detection code, adding code to clean up and move around the file system. The code is able to run halfway through, but it is still getting stuck on the error of not having the labels file. Our problem is that the file structure is not set up right for the dataset that we’re using with the code that we are modifying. I believe we are behind in this aspect, that we should get the code running.

I also worked on the mechanical design, refining the placement of the neopixel, and also changing it to a strip of neopixels. We thought about making a backboard to support the platform that the camera and jetson to sit on, but we ended up keeping the PVC pipe idea in order to hide the wiring. 

We all worked together on the design presentation, which I will be presenting this week, making block diagrams as well as FSMs and flowcharts to represent the flow of the data throughout the system.

We are a bit behind in terms of mechanical building since we have not finalized the parts to order. But we are finalizing and submitting the order form tonight.

Classes used to develop our design: 18240, 10301 Intro to ML, 18100 Intro to ECE

Posted on

Team Status Report for 02/18/23

This week our team has explored materials and measurements of the mechanical parts including the bins (one big bin and two small “sub-bins” for recyclables and trash). We updated our trapdoor design from a single door operated by an actuator to a swinging door with two openings, one for recycling and one for trash. This way, instead of the user having to wait for the device to respond and having to take the waste off the platform if not recyclable, the device can educate the user with its alerts while still taking care of the organization, making its use easier and more practical overall. Since we were still in the design stage and hadn’t ordered parts for the old design, there wasn’t much of a cost to account for due to this change. From there, we have designed and prototyped where and how to connect each part including the platform and the servos that controls the closing and opening of the lid. We have also added the idea of using ultrasonic sensors to detect if the bins are full. The fill indicator is not a part of our MVP but we will look into how the sensors could read and transfer the capacity status. Simulations of arduino are performed on tinkercad and more details could be found on our presentation slides. 

Our biggest risk right now that could jeopardize success is the operation of the ML waste classification model, since we are having some initial set up setbacks and the model not working could mess up the entire classification part of our project, which is one of our biggest requirements. In terms of software, we have delved into the ML code and tried to run it with our drinking waste classification datasets. Right now the training could be run halfway and we still need to adjust our dataset structure. Since the size of datasets is huge and Google colab takes fairly long to run, this part is taking more time than expected. However, as we definitely have a better understanding of the model and we have written scripts to deal with raw data and set up our project structure on colab, we are able to deal with the risks without it not leaving us too much behind and we are confident we can get this working soon. As a contingency plan, we do have other backup models/datasets from our research earlier this semester that we can switch in if the current set up does not work out. 

Over the weekend we are ordering more mechanical parts such as the bins, servos, and lids; and after the design report, we could transfer code from tinkercad to the devices and integrate them.

So far, HW wise we are on/ahead of schedule (design + simulation complete), however, the delayed part ordering may set us back a bit this upcoming week. Software wise we are a little behind due to the difficulty in debugging the training model, and therefore will focus more of our efforts here to get back on track.

Some principles of engineering, science and mathematics that our team used to develop the the design solution include:

  1. ML (detection and classification algorithms): ML Model
  2. FSMs, engineering design charts: design diagrams
  3. basic mathematics/geometry: mechanical measurements and camera distance calculations (focal point + triangle calculations)
  4. Algorithms, Programming, and Control flow: training model debugging, hardware programming for simulation
  5. Circuit design: Arduino + components hardware design for speaker, neopixels, servos, etc.
Posted on

Aichen’s Status Report for Feb 18th

This week, I have taken a dive into the pretrained ML model and the datasets that we are planning to use. I have written scripts to randomly partition our training datasets into subsets of training, testing, and validation and setting up our project structure accordingly on Google Colab notebook with Ting. Because we do not have access to the datasets that the pretrained model uses, we are still wrestling with where to place the “labeling” files that YOLO models need to detect where (multiple) items are exactly on an image. Right now, the model could run after classification, until it needs to assign labels to different objects.

After discussing with TA and our team, I have done more research about integrating Jetson, camera, and Arduino. One important progress is the object detection algorithm. We are basically implementing a “two-state” waiting algorithm to detect items placed and to wait for hands to be removed before running YOLO classification. The more detailed explanation could be found on the FSM graph I drew on our design presentation slides. The connection of Jetson Nano, camera (IMX219), and Arduino, as well as the actual software modules (Serial) to send and receive data, are all more settled now,

Besides that, I have ordered the items mentioned in last week’s report and have spent the later of this week working on the presentation slides. For ECE courses, the FSM concept I learned in 18240 was helpful for us to understand the “detect-and-wait” algorithm, as well as presenting. And ML (10301) was also crucial for understanding the ML model and dataset structure that we are using. Since we have worked with Arduino in 18220 too, that also guided our research well.

 

Colab Code: https://colab.research.google.com/drive/1inMk5b39dCFxSdzQ6_ReY75MkQTkyQ1Z#scrollTo=MsLqRPqxeAK5

Dataset Processing Script (will keep updating this for partitioning labeling files later):

https://github.com/AichenYao/capstone-scripts/blob/main/fileUtils.py

 

Posted on

Team Status Report for 02/11/23

One significant risk that could jeopardize our project is the trapdoor not working. Specifically, since our group is not very experienced with mechanical hardware, including this component in our project is a risk. If the door does not work, then there is no way to prevent people from putting the wrong waste into the bin, even if our model classifies the trash correctly. Therefore, to mitigate this risk, we have started researching this mechanical portion from the very introduction of this idea, and have simplified it as much as possible. While initially we were planning on using a conveyor belt and pushing the waste into the “correct” bin, we ultimately decided to reduce this plan to a simple downwards opening trap door to limit the number of mechanical components required. Additionally, this risk is being managed by our team members searching up videos to see real working examples of the trapdoor, looking up and comparing the necessary and possible components that could work to operate the door, and looking for swing lid garbage cans as a contingency plan that would eliminate the need for us to build the actual trapdoor but just to add extra components (lock that would prevent the lid from opening) to have the mechanism work as desired. 

Compared to our proposal presentation, some design changes we made include figuring out a more detailed design for the trapdoor (ie. adding an actuator and using the servo as a supporting lock), deciding to use a neopixel rather than 2 separate color LEDs to avoid confusion, and adding an ultrasonic sensor to know when the bin is full. These changes were made based on our research in attempting to fine tune our design, as well as based off of some of the questions/comments asked by peers during the presentation. 

Schedule wise, we are on track. This next week we will need to continue fine tuning our design (hardware and software wise), defining specific parts that we need and having better, quantifiable plans/goals going forward. We could be further in setting up the ML model training,

Our project includes considerations for the education, safety, and cleanliness of the students that will be using Dr. Green recycling bin. Students are learning recycling rules in their local area, the bin will be safe for students and be made of a safe plastic material, with a slow open/close speed. Cleanliness is a component that we are still working on, but the material will be easily cleanable. 

 

Posted on

Vasudha Srinivasan’s Status Report for 02/11/23

During the start of this week, I mainly focused on our team’s proposal presentation, finishing up slides and practicing its delivery. Specifically, I worked on setting up an appropriate theme, format, and visuals for the slides, defining the problem, technical challenges, and testing plan, and adding any missing content to the remaining parts. I then looked over all the information that we had researched, our updated design plan, and any quantifiable values to be as prepared for possible questions that could come up during the presentation. I ran through the presentation multiple times (alone and later in front of others for feedback), making sure that I was covering all the important details while still maintaining the time limit.

After presenting on Monday, I worked on finding ways to incorporate the feedback/comments/questions that came up during the presentation and researched more in terms of our materials to better define our device design, specifically focusing on the mechanical/hardware implementation, as that is the area we are most unconfident about at the moment.  In terms of the visual LED cue that we had originally planned, I thought it might be easier to go with neopixels, as only one would be needed (as opposed to a separate LED for red and green) to avoid confusion for the user and allow for future indicator capabilities if time permits. Based on feedback given during the presentation, I also looked into proximity/ultrasonic sensors that could be used in order to indicate that the bin is full. Lastly, I tried improving our trapdoor implementation, looking up videos of existing designs and tools that could be used (servos vs actuators vs more powerful motors). 

Here is our current version of the Trap door design

My progress is going according to schedule, and I hope that in the next few days, I can narrow down all of the specific materials needed for ordering and have enough design details for our upcoming report deadline. 

Document of research sources of materials: https://docs.google.com/document/d/12br7eD0iqjRmdWXDqOWgGqm1XdGtU76QQc_jtMEdrBI/edit?usp=sharing