Category: status report (individual)

During last week’s status report, I mentioned how we needed to find a dataset which exposed the network to a wider variety of fruit.

Samuel’s Status Report – 19 Mar 22

This week, was a relatively productive one: I managed to train the network on the “silvertray” dataset (https://www.kaggle.com/datasets/chrisfilo/fruit-recognition) which produced relatively robust results on test data that the network had never seen before (a green label indicates accurate detection; here we had 100% accuracy on test data).

Of course, the test data also involved the same silver trays that the algorithm trained on, so a high accuracy is expected.

I then moved on to making it perform detection using real-world data, on our C++ application with our webcam, and these are results!

As visible in the images above, the NN is able to detect the fruits accurately on in a real-world situation (including a noisy non-white background WITHOUT segmentation applied). That being said, there are some inaccuracies/misdetections such as with the orange above despite the frames being very similar. I describe needed improvements later below.

With this, we are currently on track towards a working prototype, although we could probably train the network to handle more classes with self-collected or web-procured data.

Next week, we will begin integration of the various components together, and I will work on several improvements to the current CV algorithm/setup:

1. Include a “change detection” algorithm that will detect when significant changes appear; this will allow us to tell if a fruit is needing to be scanned.
2. Normalization of the image before processing; this will help reduce issues with random lighting changes, but might require that the network be retrained
3. Build the actual rig with a white background and test the algorithm on that
4. If necessary, change to using a silver tray or silver-colored background similar to the network’s training set, and/or collect our own datasets.

This week, I focused heavily on getting the neural network to work properly. In the beginning of the week, I successfully trained the neural network on the new ResNet18 architecture (as opposed on the old one that did not work). After I realized that it didn’t work as well as expected on real data, I swapped to a more advanced ResNet50 architecture, but that did not seem to help it either.

It was then that I began to suspect something else was wrong besides the the network itself, because the networks kept reporting a 90+% validation accuracy, but whenever I tested the code, even on training images. This hinted at a problem with my testing code/script. Eventually, I realized that during the network training process, we were passing in normalized images, and the network was training on that; once I changed my test/evaluation script to feed normalized images into the network, and everything worked very well!

However, as I began testing the network on various images, we realized that the network was not very robust on external data:

After scrutinizing the dataset, we realized that the dataset was not good enough, and was subject to some major flaws that made it susceptible to overfitting. Firstly, it was a 360 degree shot of a single fruit per category, so even though there were many images of fruit, the network was fed only one example of something from that fruit category, thus making it hard for the network to generalize based on colour, shape etc.

To resolve this problem, I would need to search for more datasets, parse them, and train our network on them. This will be my focus for next week. Currently, I have found several datasets; however, they each have their own issues. The most promising one I have found so far is very similar to our use-case, with images of fruits taken from a top-down view, but has a reflective silver tray background which is very hard to segment away. Some pictures also have groups of fruit:

I will first try training the network on center-cropped and resized images and if that does not work, I will try algorithms like Otsu thresholding on saturation value, or GrabCut, to segment away the background.

This week, we spent most of our time writing up our design review report. Although a lot of the main content had been covered in our design review presentation, the devil was in the details for this report. In particular, we needed to make our block diagrams a lot more detailed since the ones we used in the slides were merely summaries.

We were very thankful that the deadline for the report got extended as that reduced the amount of stress we had, and allowed us to write a more polished report.

On the implementation side, I am slightly “behind schedule” in the sense that I was not able to get as much work done on the new neural network implementation as I had hoped to, because I was focusing on the report instead. However, we are still ahead of schedule since we already have an implementation going.

Next week, I will focus on implementing and training the ResNet18 network, and then testing the accuracy on a self-collected dataset of various fruits.

The highlight of this week for me was delivering the Design Review presentation. I think I did quite well for the presentation, with my teammates and Professor commenting that the presentation was polished, with good graphics and content. We are currently working on the design report, and will use the content of the design presentation to write it.

On the technical side, I was able to make significant progress on the PyTorch port for the CV application, which was coded in C++ so that it can run optimized code on the Jetson, for maximum speed. The C++ application is now able to run the trained PyTorch model and spit out a set of predictions for the top 5 highest probability classes.

There were several challenges with the porting process, including the lack of good PyTorch documentation, which resulted in it being very difficult to figure out how to properly convert between C++ formats (cv2 to torch Tensor for example) and also important considerations in ensuring that the model can be serialized properly for running in C++ (in particular, no nested functions etc.). This is a lesson learnt on the importance of good documentation, and the pain/need of having to pour through various forums and articles as a result.

However, after training and testing the network, I began to realize big problems with the trained model. Most notably, the model failed to produce correct predictions. After consulting with Prof Mario’s PhD student, we realized that we were using a highly customized model that was not designed properly, and was not even a proper “ResNet” (lacking fully residual layers). To this end, he advised us to use other preexisting models like ResNet18 or AlexNet. This is a lesson learnt as to not blindly copy code over intern

Next week, I will focus on trying to train our data on either ResNet18 or AlexNet, as well as test it in the new C++ classifier. (There is also a Python one for quick testing, in case the C++ one still has bugs). Hopefully I will be able to train up a network that will achieve our desired accuracy of 85% (the network itself should reach about 95% validation accuracy).

Fortunately, despite this setback, we are currently still on schedule, because we were previously ahead of schedule with the successful porting of the C++ application.