Category: Team Status Reports

This week was the last week we could actually put in work before the video, so it was a very crucial week for us since we had to ensure we had at least our MVP working for the video.

The team worked on figuring out how we can attach the servo to the diverter we had printed. We, then, fixed our image segmentation masks to ensure we were detecting the rotten and good parts of the banana correctly. Once we figured that out, we set up our final product and ran several tests to see if we were meeting the metric requirements we had set out to meet. We seem to be meeting the requirements, at least for the MVP, for now. We’re working on image segmentation for carrots to be able to include that into our product, and we’re also working on developing the AlexNet rottenness classifier, so we can see how it’s performing in order to be able to compare it to our current classifier and gain insights. We’re now working on the final video and poster.

Below is a picture of our final product setup. We would like to thank Prof. Savvides and our TA, Uzair, for all of their help and support throughout the semester!

      

This week was very important for our team as it was around the 2-week mark before the final deadline. So, we followed a daily work plan for this week.

Ishita Sinha and Ishita Kumar finished the final set-up for the conveyor belt, including positioning its height and stitching the belt in place. Kush integrated the code onto the Jetson Nano. Then, IK and IS did several conveyor belt + software runs. Initially, the algorithm was too slow to meet our conveyor belt speed requirements. However, IS sped up the code from 3s/fruit to 0.04s/fruit by switching to Numpy arrays. So, now we meet our speed requirements. Some frame analyzer bugs were also fixed and the software+conveyor belt system works well for the most part. We found that sometimes, the hsv filter masks the rottenness detector uses picks up a lot of unrelated noise due to bad lighting conditions. So, it is very important for us to obtain better lighting for more consistently successful runs of our system.

The final part of the MPV left is the gate. Kush and IK designed the first prototype of the gate based on our original design. Unfortunately, the gate from the original design is too short and the banana just gets stuck and starts rotating on the conveyor belt. However, we came up with several new prototypes and have CADed a longer version of our gate which we hope will work.

Having the strict plan for this week helped but we strayed from the plan by mid-week due to unexpected bugs and our original gate design not working. However, we learnt many important things about our design this week during integration and will continue to follow strict plans for the next weeks to finish up our original MVP design (including the gate) with the addition of the conveyor belt.

 

• Implemented image segmentation for good and rotten apples and oranges. Started testing the algorithms on datasets for apples and oranges (to good success).
• Worked on fruit detection (i.e differentiating between rotten and good apples, bananas and oranges) using different models (yolo and pixel analysis).
• Started integration test. Currently, we have been able to take pictures from the cameras, and pass them along to the algorithms and get a final result. Working on tweaking the threshold values and experimenting with lighting and physical surroundings.
• Almost done with servo + shield (should have the system running by monday). After which, we will have full system integrated (i.e from taking picture to controlling gate via the jetson nano).
• Getting close to finishing the project, just need to finish up some physical integration and tweak parameters.

As part of our work on the project this week, on the hardware front, we worked on getting the conveyor belt up and running. We still have some work to complete, but we plan on finishing it up by Monday, in time for the demo. On the software end, we tested our algorithm on a much larger dataset of bananas to ensure it generalizes well. Post that, we started testing on images we took with good versus rotten bananas to see how well it seems to be classifying them. We used a white background for the images since we’ll be having a white background with our conveyor belt. Besides that, we have set up the live stream for the camera, so we now need to work on writing up code for capturing each frame and examining it.

The updated schedule for our project looks as follows:

One of the major concerns for our team, for now, is that we haven’t started working on the gate yet, and we don’t have the setup for the product yet. Our schedule does recognize this and we have it planned accordingly, but realizing we have just 3 weeks until the final demo definitely sounds quite daunting, so we’d need to ensure we really meet the deadlines very well.

• Fabricated all the parts required for the conveyor belt. We decided to 3D print the motor coupler and the shaft couplers instead of using a piece of wood. The couplers seem very sturdy and we are confident they will work.
  • The above picture shows the motor shaft embedded in the motor coupler (yellow piece), which is in turn embedded inside the roller. The 3 black pieces are the other couplers. The whole system is tight and sturdy.
• The CSI camera is able to take pictures in burst mode now, with the timing of the burst being configurable. Here is a sample image taken from the CSI camera. We’re in the process of getting the usb camera to work, but it’s proving to be a lot harder. It’s possible we might need to order a second usb camera since the one we have might not be compatible.
• Ordered the 12V adapter for the conveyor belt and the DC motor speed controller board. We plan to finish constructing the conveyor belt by middle of this week.
• Made a lot of progress on the pixel classification, and the algorithm is now very robust. We have also started looking into AlexNet, but we might not need it since the fine-tuned pixel classifier is working very well.
• We’re making good progress, and are on track to finish the project by the due date!

Our team made good progress this week on various elements of our project. We now have two masks for rotten and good bananas respectively that we will use to calculate the percentage of rotten parts in a banana. We also have started researching our secondary ML solution and looked at AlexNet as of now. We had a meaningful discussion with Prof. Tamal and our TA Uzair who gave us the great idea of using live feed or burst pictures instead of periodically taking pictures which we talked about more as a team and decided would be better for our timing concerns on our conveyor belt. We also know that building a conveyor belt is an important part of our testing and so we have started building out the conveyor belt. We made good progress today in the woodshop. We managed to find free wood pieces that worked for us and we have cut all the parts necessary for our conveyor belt, including the wooden base, the wooden side stands, the PVC rollers. Next, we have to hook up the motor to the PVC rollers with woodwork and we have a few ideas on how to do that. So we are on schedule for our project as of now. Kush is going to also work more on the Nano early this coming week and Ishita K. and S. should have a good rottenness detector using masks developed as well as started on the ML plan. Our conveyor belt should be ready by mid-coming week.

 

The most significant risk we anticipate as of now is building the conveyor belt. None of us have mechanical experience so this would definitely be challenging. However, as a contingency plan for the same, we plan on using a treadmill since the conveyor belt isn’t a part of our product – our product is meant to be integrated into existing conveyor belt systems, so if our conveyor belt doesn’t work, it doesn’t jeopardise our product itself.

The design of the system was finalised and updated, as has been shown below:

We had to make this change to the design since the earlier design that was using pistons might not have been able to meet the requirements of the product. We were using the pistons to push an item off the conveyor belt. However, doing that would require a lot of force from the pistons, which may not be feasible. The pistons could possibly also not meed the speed requirements since they operate quite slowly. Thus, we shifted to this gate model. Corresponding to this change, we updated our model to be sorting fruit into only 2 categories – good v/s rotten, so the gate can rotate in one off 2 directions to push the fruit in the appropriate basket. Correspondingly, the updated block diagram is shown:

We updated the schedule to reflect a more feasible timeline given that we have midterms going on currently. This past week, Kush worked on understanding the Nano and Ishita Kumar and Ishita Sinha worked on improving results from their image segmentation and colour analysis. For the upcoming week, they’ll be integrating their code and working on building a good classifier. Besides this, the team will also be working on the Design Review Report.

• Finalized the mechanical design for the conveyer belt system (currently have rack and pinion as a contingency plan).
• Extracted color ranges from hsv space for certain images (experimented on a picture of a banana).
• Ordered the Nvidia Jetson Nano 2GB and the Raspi camera module V2. Started experimenting with them. Next step is to write a program to take a picture and save it on disk.
• Worked on the design presentation. Finalized some design points (apart from mechanical) that were previously unanswered.
• Performed risk analysis and risk mitigation. In particular, if we can’t assemble the conveyer belt ourselves, we will use the treadmills in the CUC gym to simulate the process.
• Started looking into algorithms for extracting features (e.g localized black spots in a banana) from the segmented images.

We worked on the project proposal and finalizing our mechanical design this week. We decided to not use the rotating plate for placing the fruit anymore as it would not support non-round fruits, like bananas, very well. We, instead, are planning to use a diagonally placed 2-camera system to take pictures of the fruit. We have also decided to use pistons to push the fruit in the right direction to physically sort it.

We researched pistons, specifically solenoid-type pistons, on Prof. Savvides’s suggestion, but we found them to be quite expensive. We are a little worried about using pistons as none of us have much of a mechanical background so we will discuss our mechanical design again tomorrow.

As of now, we are slightly behind schedule as we had wanted to order the parts this week but we think it is better to talk and share our parts list and reasonings to the professor and our TA on Monday before buying things. So, this is not too concerning as it is better to double-check before spending too much money. Ishita Sinha and Ishita Kumar have started looking at color analysis and image segmentation and should code up a simple working sample by next week. Kush will meanwhile take pictures for training data. If we get all this done next week, we will be very well on track.

For now, it seems like the most significant risk we’re undertaking is visualizing the design to the best of our abilities, but not actually being able to see it physically before us. We believe our design is fairly complicated, so there are many aspects of it that could go wrong or force us to rethink our entire design if it doesn’t work out as we expect it to. As a contingency plan, for now, we plan on rotating the stand holding the fruit manually, and/or clicking pictures manually, in order to at least be able to make progress while rethinking our design.

We modified the product to involve holding the fruit up and rotating it using a rotating disk of sorts. We did this because we plan on using a banana for our MVP, so a banana is a fruit we’d like to hold up in order to capture pictures from all sides.

We have a preliminary schedule planned for now, which we will be putting across in our proposal presentation.