Month: March 2022

During last week’s status report, I mentioned how we needed to find a dataset which exposed the network to a wider variety of fruit.

Samuel’s Status Report – 19 Mar 22

This week, was a relatively productive one: I managed to train the network on the “silvertray” dataset (https://www.kaggle.com/datasets/chrisfilo/fruit-recognition) which produced relatively robust results on test data that the network had never seen before (a green label indicates accurate detection; here we had 100% accuracy on test data).

Of course, the test data also involved the same silver trays that the algorithm trained on, so a high accuracy is expected.

I then moved on to making it perform detection using real-world data, on our C++ application with our webcam, and these are results!

As visible in the images above, the NN is able to detect the fruits accurately on in a real-world situation (including a noisy non-white background WITHOUT segmentation applied). That being said, there are some inaccuracies/misdetections such as with the orange above despite the frames being very similar. I describe needed improvements later below.

With this, we are currently on track towards a working prototype, although we could probably train the network to handle more classes with self-collected or web-procured data.

Next week, we will begin integration of the various components together, and I will work on several improvements to the current CV algorithm/setup:

1. Include a “change detection” algorithm that will detect when significant changes appear; this will allow us to tell if a fruit is needing to be scanned.
2. Normalization of the image before processing; this will help reduce issues with random lighting changes, but might require that the network be retrained
3. Build the actual rig with a white background and test the algorithm on that
4. If necessary, change to using a silver tray or silver-colored background similar to the network’s training set, and/or collect our own datasets.

This week, I focused heavily on getting the neural network to work properly. In the beginning of the week, I successfully trained the neural network on the new ResNet18 architecture (as opposed on the old one that did not work). After I realized that it didn’t work as well as expected on real data, I swapped to a more advanced ResNet50 architecture, but that did not seem to help it either.

It was then that I began to suspect something else was wrong besides the the network itself, because the networks kept reporting a 90+% validation accuracy, but whenever I tested the code, even on training images. This hinted at a problem with my testing code/script. Eventually, I realized that during the network training process, we were passing in normalized images, and the network was training on that; once I changed my test/evaluation script to feed normalized images into the network, and everything worked very well!

However, as I began testing the network on various images, we realized that the network was not very robust on external data:

After scrutinizing the dataset, we realized that the dataset was not good enough, and was subject to some major flaws that made it susceptible to overfitting. Firstly, it was a 360 degree shot of a single fruit per category, so even though there were many images of fruit, the network was fed only one example of something from that fruit category, thus making it hard for the network to generalize based on colour, shape etc.

To resolve this problem, I would need to search for more datasets, parse them, and train our network on them. This will be my focus for next week. Currently, I have found several datasets; however, they each have their own issues. The most promising one I have found so far is very similar to our use-case, with images of fruits taken from a top-down view, but has a reflective silver tray background which is very hard to segment away. Some pictures also have groups of fruit:

I will first try training the network on center-cropped and resized images and if that does not work, I will try algorithms like Otsu thresholding on saturation value, or GrabCut, to segment away the background.

This week, we spent most of our time writing up our design review report. Although a lot of the main content had been covered in our design review presentation, the devil was in the details for this report. In particular, we needed to make our block diagrams a lot more detailed since the ones we used in the slides were merely summaries.

We were very thankful that the deadline for the report got extended as that reduced the amount of stress we had, and allowed us to write a more polished report.

On the implementation side, I am slightly “behind schedule” in the sense that I was not able to get as much work done on the new neural network implementation as I had hoped to, because I was focusing on the report instead. However, we are still ahead of schedule since we already have an implementation going.

Next week, I will focus on implementing and training the ResNet18 network, and then testing the accuracy on a self-collected dataset of various fruits.