Team A1: Fridge Genie

For this week, I focused on data collection and annotations, as well as getting started on the prototype of our model.

I’ve set up our PyTorch environment and completed an initial YOLOv5 prototype. I’ve developed a basic pipeline, where images are pre-processed with OpenCV and then fed to the YOLOv5 model, and conducted some preliminary training and testing using small annotated datasets of grocery items. Furthermore, I have been sourcing and noting down relevant datasets on Kaggle/ Roboflow.

Currently, I am working on integrating the online datasets with the YOLOv5 model, and conducting some initial tests on accuracy as well as inference speed. I aim to test the inference speed locally on Raspberry Pi, as well as on cloud, to get a measure of latency of either set-up.   I will also experiment with image processing methods using images taken from our fridge, in order to try and improve detection accuracy.

Looking to the future, I will have to obtain the necessary hardware(Raspberry Pi) in order to test the effectiveness of our model when run locally. I will also have to work on integrating the model with our peripheral device as well as the mobile application.

For this week, I focused on researching the relevant CV libraries and frameworks that we can used to build our model. I explored multiple libraries(OpenCV, PyTorch) for handling image preprocessing and deep learning, and compared various object detection frameworks to determine which one best fits our requirements.

We decided to employ OpenCV in our computer vision pipeline, due to it being the industry-standard library with a large active user community and extensive documentation. OpenCV offers a wide variety of image transformations and integrates extremely well with deep learning frameworks such as PyTorch, allowing it to be easily implemented within our pipeline. The use of OpenCV for data processing will be essential in integrating our CV model with our overall system. We have also decided to use a YOLOv5 model for object detection, due to its extensive documentation and active community, and its suitability for real-time applications such as ours. Based on proven real-world performance, we are confident that the YOLOv5 model can achieve our accuracy goal of 90%.

The following is an initial outline of how the CV pipeline will operate:

1. Image Capture and Basic Preprocessing (resizing, normalization with OpenCV)
2. Inference using a YOLOv5 model
3. Post-Processing to handle classifications/ quantity counts

In terms of my next steps, I will gather datasets of representative fridge ingredients, including variety in lighting, item arrangement and packaging. A diverse dataset of relevant images is necessary in order to achieve our accuracy goals. I will also set up a basic prototype using YOLOv5 on a small subset of test images, to evaluate initial detection accuracy, speed and resource usage. I aim to run initial benchmarks on local vs cloud inference, in order to confirm latency and throughput, as well as identify optimization techniques if needed to meet time constraints.

Overall, I am on track with researching the relevant CV libraries, laying out model architecture, and investigating how to integrate the detection pipeline within our system. For the next week, I will focus on data gathering and prototyping of the CV model.