Posted on

Team Status Report for 3/30

Risk Mitigation 

A notable risk for our project is being able to have testing data. We are combating this risk by using RPis in order to take camera footage and store chunks of it in order to use for testing. Since we have gotten approval to test our project at Salem’s Market, we will fortunately be able to accomplish this. 

 

Design Changes

We are now for each checkout line adding a RPi in order to take the camera footage and store it for usage in testing. 

 

Schedule Changes 

Nothing has changed from last week in terms of our schedule, our Gantt Chart remains unchanged



Posted on

Brian’s Status Report for 3/30

Accomplishments 

My model finished training, and this last Sunday Simon and I went to Giant Eagle to film some videos for testing purposes. However, running my throughput code on the video made me realize that the model performs very poorly on live video, not even recognizing a single item. I then trained another dataset for 100 epochs, and the results are below: 

  

Even after training the second dataset, the accuracy on live video is quite poor. Giant Eagle also has advertisements and a design on their conveyor belts, which could’ve caused the accuracy to drop. Below there is an image that clearly just shows only one thing being recognized as an object, which is clearly very inaccurate for what we want to do.

Therefore, I started working on edge detection to see if there was a way to calculate throughput and count items that wasn’t so inaccurate. Below are my results from trying to get rid of noise using Gaussian blur and canny. Words are still being shown which makes it seem as though it will be very difficult to continue with this approach as well. 

In terms of interim demo progress, I will pick up a RPi and cameras as they are available and try to work on getting the RPi working with two cameras.

Progress

I am still a bit behind schedule. I will try tomorrow to get the integration on the software side done so that we can have something to demo. As for throughput, if I can’t make significant progress training for more epochs or with edge detection, I will need to pivot to adding pictures of items specifically on conveyor belts from Salem’s Market to the dataset for training.



Posted on

Simon’s Status Report for 03/30/24

Accomplishments

At the start of the week, Brian and I took videos at Giant Eagle to test our models, and my model for the shopping cart detection works reasonably well on the footage that we have. However, I realized that the accuracy would probably be improved by putting the camera higher up than originally planned.

To improve the accuracy, I also tried to split the larger video into two 640 x 640 videos because the model is trained on image sizes of 640×640, but I couldn’t get the model to predict on two videos without some kind of error despite running the process in separate threads. I don’t think it will be necessary for the interim demo, and there might be better ways to improve accuracy (such as the higher camera angle and simply collecting more data from Salem’s Market, where we plan to implement our system), so I will just put this aside for now and just run inference on the original large video instead.

Lastly, I changed the method for counting carts from a static bounding box that counts when carts enter/leave. Instead, I use a static bounding box to find the first cart, and then look for the next cart in a small region after the first cart, and so on and so forth until the last cart, which should minimize errors due to carts that are passing by the line but not entering.

Progress

I plan to collect some real data from Salem’s Market this week and retraining my model for higher accuracy. If there is any component that’s missing for us to have a complete end-to-end system, I will also work on that (such as if our throughput calculations aren’t quite done).

Posted on

Shubhi’s Status Report for 3/23

Achievements

We got confirmation from Salem’s Market to test the system – they have 6 checkout lanes but we will only be testing out 3. Currently in the process of getting CMU approval to test at Scotty’s but no answer yet, due to legality concerns. Implemented relative fullness detection, but need to still test it out.

Progress

Since we don’t have Scotty’s approval but we have Salem’s Market approval, we will be testing at Salem’s in strip district, but a little far so not the most optimal, and will take more time. We need to still integrate all components of the system, but that should only take a few days and then we can use the rest of the week to test out the system, in time for the the demo.

Posted on

Simon’s Status Report for 3/23/2024

Accomplishments

Realized I had to retrain model because I accidentally set the image size to 64×64 in the original, which made it so that the model would fail to make accurate predictions on images with higher resolution. Scaling down images/video to 64×64 worked, but with lower confidence on predictions than I would have liked. With a new model trained with image size of 640×640, the model works much better and with higher resolution images and predicts correctly with higher confidence than the scaled down images for previous models.

With the new model, I used YOLOv8’s object counting to count the number of shopping carts in a certain region of an image. I’ve tested it with stock footage to verify that it works on an extremely simple case, and Brian and I will go to grocery stores tomorrow to take video footage from our desired camera angle so that we can see if it works correctly.

Progress

I’m hoping I’m close to being done with shopping cart detection (need to verify that it works and make changes accordingly), but if all goes well then I should be somewhat caught up. For next week, I will try to make sure the shopping cart detection is working and once I’m confident that it is, I can help Shubhi on relative fullness detection.

 

Posted on

Team Status Report for 3/23

Risk Mitigation

One of the biggest risks for our project is testing our software modules and making sure they individually work prior to integration. To combat this, we will be going to grocery stores and taking video footage in order to test our modules. OpenCV makes it very easy to read video files and so this allows us to test our individual modules without having to set things up live. For throughput calculations, we plan on taking video footage of store items moving on the checkout line conveyor belts and processing the footage via our code to check if the calculated throughput makes sense. Similarly, we will take videos of carts moving in the grocery store in order to test our other modules when we finish implementing them. 

 

Design Changes

Nothing has changed about our design as of this week, however, we are considering changing what the throughput calculating module inputs into the database: instead of just storing a calculated throughput, we will store item count and time elapsed in order to more easily keep count of a running average. 

 

Schedule Changes 

Nothing has changed about our schedule, our Gantt chart remains the same as before. 



Posted on

Brian’s Status Report for 3/23

Accomplishments 

This week, I managed to finish training my model after some delay (5000 images in a dataset quickly led to me being booted off Google Colab for hitting GPU usage limits). I trained it for 100 epochs total, but I had to split it into 50 epoch sessions because of GPU usage limits, so I trained for 50 epochs and then trained the resulting model for another 50 epochs. The resulting model seems to have serviceable precision, reaching about 75%-80%, and the validation batches seem to detect grocery items relatively well. 

Now the implementation for the throughput calculation module is complete with the updated module. Simon and I will go to a grocery store tomorrow (Giant Eagle/Aldis), take a video of items moving on a conveyor belt, and test the throughput module. 

Progress

I am still behind schedule, but I am currently working with Simon to fully flesh out the line detection module. This next week, we will try to implement this fully and test it for the interim demo, and start helping out with the relative fullness module since that is very important. 



Posted on

Team Status Report for 3/16

Risk Mitigation

Now that we are training several models to go through with our CV algorithms, we realized that due to time constraints, it is imperative that we get this done as soon as possible. To do this, we are using YOLOv8, which is very user friendly, and we are training models from datasets that are publicly available whenever possible. For example, we are training models from datasets of shopping carts that are publicly available and of datasets of grocery store items that are publicly available in order to save time. 

 

Design Changes

Nothing has changed about our design. We are still implementing throughput calculations for cashiers, relative fullness calculations for carts, and detecting how many people are lined up in the same manner as was determined before spring break in our design report. 

 

Schedule Changes

Our Gantt chart remains unchanged from before. 



Posted on

Brian’s Status Report for 3/16

Accomplishments

This week, I didn’t complete as much as I had hoped. I wanted to finish my implementation of the throughput calculating module and testing but I spent a lot of time this week trying to use YOLOv3 to accomplish it. Much of my time was spent trying to figure out how to train a model from the Freiburg grocery store dataset available online (http://aisdatasets.informatik.uni-freiburg.de/freiburg_groceries_dataset/), which has 25 different classes of objects typically found in grocery stores. However, a significant issue I ran into that I dealt with up until Thursday/Friday was that on Google Colab, I was getting many errors trying to run Darknet (the backend for YOLOv3), which I figured out was due to CUDA and OpenCV versions being different on Google Colab than what they should be. These errors were mainly because of YOLOv3’s age as a more legacy version of the framework, and so my attempts to fix them cost a large amount of time. I finally decided to pivot to YOLOv8, which was much less time consuming and allowed me to make some progress. Currently, I have written a rudimentary algorithm for determining the throughput of a cashier: essentially the software module takes in a frame every ⅕ seconds, checks how many new items appear on the left side of the frame (where the bagging area is), and then counts those processed items into a running total (that is divided by the time elapsed). Pictures of my current code are taken below. Since the model is pretrained, it doesn’t work with objects from grocery stores and so I tested to see if the code would even calculate throughput from a video stream using generic objects (cell phone, wallet, computer mice, chapstick): 

Throughput is being calculated and updated 5 times a second, but I will need to experiment with figuring out if this is an optimal number of updates per second. A huge benefit that I noticed about pivoting to YOLOv8 is that my code is much more readable and straightforward. My Google Colab GPU access was restricted for some of the week due to overuse when trying to figure out YOLOv3 issues, so I am currently training the dataset as of tonight (3/16), I will check back on it every few hours to see when it is finished. 

Progress

I am still behind schedule, but tomorrow and Monday I will be working on testing the code after the model is done training, and after that I’ll be able to help Shubhi and Simon with their tasks, as relative fullness and line detection are the most important parts of our project. 

 

YOLOv8 code: 

Outdated YOLOv3 snippet for reference: 

 

Posted on

Simon’s Status Report for 03/16/2024

Accomplishments

For this week, I found a few datasets to train my model on

The first one can be found here: https://universe.roboflow.com/furkan-bakkal/shopping-cart-1r48s

I used YOLOv8 to train a model and the results were quite good at first glance, with the best model having roughly 90% accuracy.

However, after looking through some of the validation data, it doesn’t look promising. Here’s one of the validation batches:

The image is pretty blurry, but the important thing to note is that the model performs poorly with realistic backgrounds (as opposed to plain white). I think it’s inaccurate enough to where tuning the hyperparameters probably won’t increase the accuracy significantly enough.

I used another dataset to train a model after deciding that the first one probably wouldn’t be good enough. Link here: https://universe.roboflow.com/roboflow-demo-projects/retail-nxvjw

The results are similar at first glance:

Looking through the validation batches, however, it seems to perform much better than the first model. Because the dataset includes much fewer images with white backgrounds, the new model looks to be significantly better than the old one at detecting shopping carts despite noisy backgrounds. I think I’ll try to tune some of the hyperparameters (such as number of epochs trained and image size) to see if the accuracy can be improved further.

I found one more much larger dataset that I’m currently training, but it isn’t done yet at the time of this status report, so I’ll check the model in a few hours.

Lastly, I learned how to use OpenCV to apply Gaussian blur, perform edge detection, and find contours, which is hopefully a good starting point for detecting how full shopping carts are, since we plan on NOT using a CV model for this part of the system. This is what an example currently looks like:

As it stands right now, this could be much improved. I will be looking into how I can use OpenCV to perform image alignment this upcoming week.

Progress

I think I’m still quite behind where I need to be at this point. For next week, I hope to refine the model I currently have a little further by tuning some hyperparameters and training again. Then, I need to see how it performs by actually feeding it camera footage of shopping carts from one of our cameras at the angle that we plan on using. As for the object tracking of the carts that I mentioned last status report, I no longer think it will be necessary, since we should only need to know how many carts are in a line at a given time, but I will check with my teammates to confirm. Lastly, I hope to figure out how to correctly implement image alignment so that the shopping carts will be consistently aligned to increase accuracy when we estimate the fullness of the carts.