Posted on

Brian’s Status Report for 3/16

Accomplishments

This week, I didn’t complete as much as I had hoped. I wanted to finish my implementation of the throughput calculating module and testing but I spent a lot of time this week trying to use YOLOv3 to accomplish it. Much of my time was spent trying to figure out how to train a model from the Freiburg grocery store dataset available online (http://aisdatasets.informatik.uni-freiburg.de/freiburg_groceries_dataset/), which has 25 different classes of objects typically found in grocery stores. However, a significant issue I ran into that I dealt with up until Thursday/Friday was that on Google Colab, I was getting many errors trying to run Darknet (the backend for YOLOv3), which I figured out was due to CUDA and OpenCV versions being different on Google Colab than what they should be. These errors were mainly because of YOLOv3’s age as a more legacy version of the framework, and so my attempts to fix them cost a large amount of time. I finally decided to pivot to YOLOv8, which was much less time consuming and allowed me to make some progress. Currently, I have written a rudimentary algorithm for determining the throughput of a cashier: essentially the software module takes in a frame every ⅕ seconds, checks how many new items appear on the left side of the frame (where the bagging area is), and then counts those processed items into a running total (that is divided by the time elapsed). Pictures of my current code are taken below. Since the model is pretrained, it doesn’t work with objects from grocery stores and so I tested to see if the code would even calculate throughput from a video stream using generic objects (cell phone, wallet, computer mice, chapstick): 

Throughput is being calculated and updated 5 times a second, but I will need to experiment with figuring out if this is an optimal number of updates per second. A huge benefit that I noticed about pivoting to YOLOv8 is that my code is much more readable and straightforward. My Google Colab GPU access was restricted for some of the week due to overuse when trying to figure out YOLOv3 issues, so I am currently training the dataset as of tonight (3/16), I will check back on it every few hours to see when it is finished. 

Progress

I am still behind schedule, but tomorrow and Monday I will be working on testing the code after the model is done training, and after that I’ll be able to help Shubhi and Simon with their tasks, as relative fullness and line detection are the most important parts of our project. 

 

YOLOv8 code: 

Outdated YOLOv3 snippet for reference: 

 

Posted on

Simon’s Status Report for 03/16/2024

Accomplishments

For this week, I found a few datasets to train my model on

The first one can be found here: https://universe.roboflow.com/furkan-bakkal/shopping-cart-1r48s

I used YOLOv8 to train a model and the results were quite good at first glance, with the best model having roughly 90% accuracy.

However, after looking through some of the validation data, it doesn’t look promising. Here’s one of the validation batches:

The image is pretty blurry, but the important thing to note is that the model performs poorly with realistic backgrounds (as opposed to plain white). I think it’s inaccurate enough to where tuning the hyperparameters probably won’t increase the accuracy significantly enough.

I used another dataset to train a model after deciding that the first one probably wouldn’t be good enough. Link here: https://universe.roboflow.com/roboflow-demo-projects/retail-nxvjw

The results are similar at first glance:

Looking through the validation batches, however, it seems to perform much better than the first model. Because the dataset includes much fewer images with white backgrounds, the new model looks to be significantly better than the old one at detecting shopping carts despite noisy backgrounds. I think I’ll try to tune some of the hyperparameters (such as number of epochs trained and image size) to see if the accuracy can be improved further.

I found one more much larger dataset that I’m currently training, but it isn’t done yet at the time of this status report, so I’ll check the model in a few hours.

Lastly, I learned how to use OpenCV to apply Gaussian blur, perform edge detection, and find contours, which is hopefully a good starting point for detecting how full shopping carts are, since we plan on NOT using a CV model for this part of the system. This is what an example currently looks like:

As it stands right now, this could be much improved. I will be looking into how I can use OpenCV to perform image alignment this upcoming week.

Progress

I think I’m still quite behind where I need to be at this point. For next week, I hope to refine the model I currently have a little further by tuning some hyperparameters and training again. Then, I need to see how it performs by actually feeding it camera footage of shopping carts from one of our cameras at the angle that we plan on using. As for the object tracking of the carts that I mentioned last status report, I no longer think it will be necessary, since we should only need to know how many carts are in a line at a given time, but I will check with my teammates to confirm. Lastly, I hope to figure out how to correctly implement image alignment so that the shopping carts will be consistently aligned to increase accuracy when we estimate the fullness of the carts.

Posted on

Shubhi’s Status Report for 3/16

Achievements

This week I worked on catching up and making progress on the project. I was able to wrap up implementing the database that all our modules will communicate with. Additionally, I went to Scotty’s Market to take pictures of carts so I could train a custom model that I could use in my implementation of identifying carts to be able to observe the relative fullness, which I also finished up. I also wrote the pseudo code for implementing the relative fullness. I reached out to Scotty’s Market, and followed up with multiple contacts, but no one has gotten back to me, so it is looking like we are going to have to makeshift a grocery store to test our system.

Progress

Since I have the module flushed out, I just need to wrap up implementation of identifying the relative fullness of the carts so that we can start testing the full system. I plan to get it done this week so that we have a week to test the system. I think we are catching up and making good progress on our system, and we should be done with implementation soon. 

Posted on

Shubhi’s Status Report for 3/9

Accomplishments

This week I spent a majority of time on the design report, especially since we had to make many changes to our design as I was working on object detection and realized that we can’t realistically count the number of items in each cart via the camera pipeline. Since it would be too inaccurate for our use case, we realized the better way is to look at the relative fullness of the cart and use edge detection to estimate the 2D space that is taken up by the items in the cart, and assign a quantitative number to that cart as a metric for relative fullness. Due to this design change, I went a little bit backwards in my research, as I had to switch gears and understand more of how to use edge detection to accomplish this. I spent some more time also doing research to account for this, which has set me back in progress a little bit. Additionally, since we are now not working on integrating an FPGA to our system, we have now decided to convert the repo to be on a python stack to allow for more simple OpenCV usage, which is something I reset this week. I also reached out to the director of dining services at CMU to talk about working with Scotty’s Market for our user testing, but I have not received anything back from them. There may be a chance that we have to create our own testing scenarios.

Progress

Currently, I am a little bit behind on the implementation according to the Gantt chart, but I plan on working extra this week to make up for it, and will hopefully have majority of the implementation done and preliminary tested with the camera that we ordered. To do this, I will be meeting with my group extra as well this week so we can all help each other get back on track.

Posted on

Simon’s Status Report for March 9, 2024

Accomplishments

I don’t have much to report this time. The week before spring break, I spent the majority of my time working on the design report. Since we have shifted responsibilities around and I am now working on the checkout line detection (and I have no previous experience in OpenCV), I spent some time over spring break working through OpenCV’s bootcamp and learning how to do object tracking and detection, so I think I’m ready to start my implementation of the checkout line detection this upcoming week.

Progress

Since we have decided to no longer use an FPGA, my previous progress (which was not much) is now irrelevant. As such, I’m pretty far behind where I’d like to be in terms of my new responsibilities, but I plan to spend extra time this week to implement the line detection algorithm. Hopefully, I will be able to start testing this week as well. I plan to go to a grocery store and ask to take a few pictures from our proposed camera angle for some initial testing data since I couldn’t find any suitable images online.

Posted on

Team Status Report for 3/9

Risk Mitigation

One vital part of our product solution is determining the relative fullness of a shopping cart using an edge detection algorithm. Edge detection can be potentially inaccurate, but it is the most important part of our calculation for figuring out which checkout line is the fastest, and therefore we need to minimize the amount of error in this case. Therefore, if our preliminary edge detection algorithm implementation has a larger error rate than wanted for figuring out the relative fullness of a cart, we will purchase more cameras (3 more, 1 extra camera per checkout line). We have enough budget to do this, and so it should allow us to achieve a higher accuracy rate with our edge detection algorithm given the extra camera angle to gather data. 

Design Changes

We made many changes this week due to fleshing out our design. The most important change we made was eliminating the use of an FPGA in our project. We realized that speed isn’t a stringent requirement for our needs (we care about accuracy the most), and therefore we decided to just run the backend entirely on a laptop. 

Furthermore, instead of detecting the number of items in a cart, which is error prone and very inaccurate, we are planning to use edge detection to instead estimate the relative fullness of a shopping cart and then use a metric of sorts to determine the approximate amount of time it would take to check out a cart that is x% full. 

We are planning on changing our programming language of choice to Python from C++ because of our choice to leave out FPGAs from our design. Python will be much simpler to understand and code reviews will be more quick with this change. Looking into OpenCV, we observed that writing in Python does not generate much overhead compared to a C++ implementation of the same function, and so our change should not incur any unwanted performance issues. 

Schedule 

We’ve changed our schedule because of our decision to change our design to no longer incorporate an FPGA. Now Simon is working on software modules. Here is our Gantt chart: 

Part A: Grocery stores are all over the world and an essential part of people’s lives. With our product solution, people will have a more streamlined and efficient shopping experience, waiting for a shorter period of time to check out of a store. Waiting much less time will help people spend less time at the grocery store and allocate more time for their other priorities. 

 

Part B: Our product solution emphasizes the need for a timely and more efficient grocery shopping experience. Today’s society is greatly motivated by fast paced lifestyles and convenient solutions, which our system design caters to. With our system producing a timely result to users, we expedite the shopping process for customers who are on a time crunch and have other priorities that they want to spend their time on. Our system uses algorithms that analyze the queue length and cashier speed to cater to the cultural value of time optimization. We can see across many cultures that time optimization is important and multiple cultures have even put efforts towards creating tools and innovations that revolutionize the efficiency of daily operations. Additionally, our system is inclusive for different cultures where people might not necessarily understand the same language, but since we only deal with numbers, we are able to cater to multiple groups of people.

 

Part C: We don’t believe our product solution will have any significant impact on the environment. The only related aspect is the electricity the cameras and laptop will use, but compared to the cost of lighting and climate control for a grocery store, this will be negligible. Grocery stores use roughly 50 kWh of electricity per square foot per year and the average grocery store is 40k square feet, while a camera uses around 10 kWh per year, a standard laptop uses 450 kWh per year, and a small monitor uses around 500 kWh per year. Assuming a reasonable number of cameras (<100), the increase in electricity consumption will be less than 0.1% by my estimates. 

 

Part A was written by Brian, Part B was written by Shubhi, Part C was written by Simon 



Posted on

Brian’s Status Report for 3/9

Accomplishments

The week before spring break, I mainly focused on the design report because we ended up making some major changes to our design. The lack of an FPGA mainly changed many things with our proposed solution and so I changed our Gantt Chart to reflect the new tasks. The week of spring break, I didn’t get much progress done with getting a working implementation of the throughput algorithm, but we decided to change our programming language of choice from C++ to Python because of the lack of significant overhead when using the latter, and more simple syntax. The main issue when trying to finish my C++ implementation that I thought of was how to figure out when to start and end the throughput calculations: if counting the number of items in the bagging area section and seeing how many go there per second and averaging it, it is difficult to accurately determine when to start and when to finish calculating the throughput. I have yet to figure this out. 

 

Progress

I am still a bit behind schedule as I have not fully implemented a working algorithm for detecting when someone picks up and puts down an item yet. I am currently in the process of converting my current (not working) algorithm to Python so I can test it more easily. This week I hope to work with Simon if necessary to get this algorithm working and simultaneously work with Shubhi for her edge detection algorithm as it is the most important part of our proposed solution. 



Posted on

Shubhi’s Status Report for 2/24/2024

Accomplishments

I worked on the software implementation for the architecture, ensuring that the directory is organized well and that the design of the software covers all the functions needed. Every module has been fleshed out as to what its purpose is to the overall goal, as well as how the modules will communicate with each other. Also, I was working on integrating openCV with the directory, researching on using OpenCV with C++ since I had only used it with Python before, and am also working on integrating YOLO into the project. There are some articles on how to use YOLO with C++, so I read them and feel more confident with using it in the project. I also placed an order for a camera, so that I can do live testing with the detection implementation.

Progress

Currently I am still building up the software, but hopefully by next week I will have some sort of item counting detection running to test. I underestimated how much time it would take for me to research and learn about implementing YOLO with OpenCV, but I also have now figured that I am going to need to use more edge detection related algorithms specifically for the item counting, in which case I am going to have to do a little bit more research there. I am not super confident in the item detection working, but I have a good idea on how to implement it, so we will see by the end of the week.

Posted on

Simon’s Status Report for 02/24/2024

Achievements

I made little progress this week. I didn’t manage to finish the HPS guide I mentioned last week due to not having an ethernet router or an ethernet port on my laptop, both of which are needed. As such, I am unable to use the UART to USB communication that I was originally planning on until I get the ethernet components. I have not yet figured out an alternative for communication without using the HPS. Also, I didn’t get to experiment with OpenCL this week because I didn’t have the required microSD card and the adapter, so I am waiting on those to be delivered this coming Monday (note to self: In the future, I should read all the required materials first thing). As a result, much of my time this week was spent on planning the OpenCL implementation since I was unable to make progress on the other deliverables.

Progress:

I am significantly behind schedule, since I am supposed to start implementing the CV algorithm using OpenCL next week. To resolve this, I plan to spend some extra time over spring break (which was originally scheduled to not have any work) to catch back up to schedule, since I doubt I will be back on schedule by the end of this week.

Posted on

Team Status Report for 2/24

Risk Mitigation

While we don’t have too much currently planned for our budget, it is imperative that we use it wisely when ordering cameras because of our design. Our design uses two cameras per checkout line, which gets very costly as we order more cameras to complete our system integration later in the semester. Therefore, we are currently just ordering one camera for testing and then we can change our camera if hardware requirements aren’t met, and not spend too much budget on this.

 

Design 

There were no design changes made to our system as of now. However, we are discussing what we need to do to handle choosing the correct frames in the real-time video footage we are receiving in order to not process redundant data, and how to handle the line decreasing in size.

 

Schedule

Our Gantt chart has not changed overall as of this week.