Shubhi’s Status Report for 4/27

Achievements

This week I spent time on the fullness detection model, playing around with different implementations for detecting the fullness to see what would increase the accuracy. Right now the carts all seem to be detecting to be similar fullnesses, despite having different fullness. The current approach was using edge detection to count the contours, but since our system is capturing the carts at a relatively slow fps, the carts are also a little bit blurry, rendering the edge detection a little inaccurate. Due to this, another approach I was playing around with was observing colors, and the number of black pixels there are in the frame. This also didn’t seem to the have the best accuracy, so I am still in the process of figuring out what could be the best approach.

Progress

Since I am not really happy with the current accuracy of the fullness detection, so I will be spending the next two days to increase it before the final poster is due, after which we will be working on the final video and report. We also need to work on the final demo because all of our testing has been done at Salem’s, but once we have a 1-hour long video to display on a monitor for the demo, we also need to make sure that our system will work for our makeshift in house checkout system that we will be demoing in person in Wiegand.

Shubhi’s Status Report for 4/20

Achievements

I worked on increasing accuracy on cashier throughput, worked through different approaches and implemented them and test them out for the most accurate approach for our use case. After that, I worked on a module that determined when a person was done with the line, for which I also did a little bit of testing. We also went to Salem’s and I created structures to attach the cameras to so that cameras could be in position for our system around the checkout counters. At Salem’s I also gathered more data to individually test the fullness acquisition module for accuracy.

Progress

Each individual component is working, and our system is also running end to end, but we are running into issues with testing at Salem’s due to concerns employees have been having with the cameras we set up. We are in the process of working something out with Salem’s but it is limiting the amount of testing we can do right now. I think the next big thing is to work on demoing and testing in a self constructed grocery checkout lane setup.

New Experiences and Learning Strategies

During this project I have gained a lot of new skills. One of the biggest ones is learning to train a model for object detection, which I learned from examples and articles online. Another thing that I learned to do was use a SQL database. I never took a class on it, so I relied on documentation and sites like stack overflow to help me achieve the goals I had for the database in the project. A lot of times for these new skills I relied on researching what I wanted to achieve to see if anyone had ever done something similar, and adapt from that to accomplish what I wanted to.

Shubhi’s Status Report for 4/6

Achievements

We went to Scotty’s Market and gathered more footage data for developing and testing purposes at Salem’s Market. We also finished integrating all the existing modules to be able to provide an output based on some footage we currently have. I also finished redoing the fullness acquisition module to use edge detection and improve accuracy for cart fullness. I used edge detection and contours to figure out the area occupied by the items within in the cart, and then after playing with the numbers I worked on increasing its accuracy. I also have been talking to Salem’s about getting permission to install our system in the market, and he brought up concerns about it impacting his business and some safety concerns, for which he wants to meet in person to talk about. I also got permission from him to borrow a few carts from Scotty’s Market for our in person demo, so we will be able to set up a realistic system for our demo in person as well.

Progress

After talking to Salem’s market, I need to schedule a meeting with the owner in person, and he requested that our advisor come as well, due to his concerns about the way this system will impact his business. His biggest concern is how cameras make people of minority groups feel, as well as if he were to give access to his pre-installed cameras, whether the security would be compromised. I hope to resolve this issue in the following week so that we can move on with more testing, but in the meantime, I am working on solidifying the modules to be more accurate, and I think based on our new daily plan we are on track.

Shubhi’s Status Report for 3/23

Achievements

We got confirmation from Salem’s Market to test the system – they have 6 checkout lanes but we will only be testing out 3. Currently in the process of getting CMU approval to test at Scotty’s but no answer yet, due to legality concerns. Implemented relative fullness detection, but need to still test it out.

Progress

Since we don’t have Scotty’s approval but we have Salem’s Market approval, we will be testing at Salem’s in strip district, but a little far so not the most optimal, and will take more time. We need to still integrate all components of the system, but that should only take a few days and then we can use the rest of the week to test out the system, in time for the the demo.

Shubhi’s Status Report for 3/16

Achievements

This week I worked on catching up and making progress on the project. I was able to wrap up implementing the database that all our modules will communicate with. Additionally, I went to Scotty’s Market to take pictures of carts so I could train a custom model that I could use in my implementation of identifying carts to be able to observe the relative fullness, which I also finished up. I also wrote the pseudo code for implementing the relative fullness. I reached out to Scotty’s Market, and followed up with multiple contacts, but no one has gotten back to me, so it is looking like we are going to have to makeshift a grocery store to test our system.

Progress

Since I have the module flushed out, I just need to wrap up implementation of identifying the relative fullness of the carts so that we can start testing the full system. I plan to get it done this week so that we have a week to test the system. I think we are catching up and making good progress on our system, and we should be done with implementation soon. 

Shubhi’s Status Report for 3/9

Accomplishments

This week I spent a majority of time on the design report, especially since we had to make many changes to our design as I was working on object detection and realized that we can’t realistically count the number of items in each cart via the camera pipeline. Since it would be too inaccurate for our use case, we realized the better way is to look at the relative fullness of the cart and use edge detection to estimate the 2D space that is taken up by the items in the cart, and assign a quantitative number to that cart as a metric for relative fullness. Due to this design change, I went a little bit backwards in my research, as I had to switch gears and understand more of how to use edge detection to accomplish this. I spent some more time also doing research to account for this, which has set me back in progress a little bit. Additionally, since we are now not working on integrating an FPGA to our system, we have now decided to convert the repo to be on a python stack to allow for more simple OpenCV usage, which is something I reset this week. I also reached out to the director of dining services at CMU to talk about working with Scotty’s Market for our user testing, but I have not received anything back from them. There may be a chance that we have to create our own testing scenarios.

Progress

Currently, I am a little bit behind on the implementation according to the Gantt chart, but I plan on working extra this week to make up for it, and will hopefully have majority of the implementation done and preliminary tested with the camera that we ordered. To do this, I will be meeting with my group extra as well this week so we can all help each other get back on track.

Shubhi’s Status Report for 2/24/2024

Accomplishments

I worked on the software implementation for the architecture, ensuring that the directory is organized well and that the design of the software covers all the functions needed. Every module has been fleshed out as to what its purpose is to the overall goal, as well as how the modules will communicate with each other. Also, I was working on integrating openCV with the directory, researching on using OpenCV with C++ since I had only used it with Python before, and am also working on integrating YOLO into the project. There are some articles on how to use YOLO with C++, so I read them and feel more confident with using it in the project. I also placed an order for a camera, so that I can do live testing with the detection implementation.

Progress

Currently I am still building up the software, but hopefully by next week I will have some sort of item counting detection running to test. I underestimated how much time it would take for me to research and learn about implementing YOLO with OpenCV, but I also have now figured that I am going to need to use more edge detection related algorithms specifically for the item counting, in which case I am going to have to do a little bit more research there. I am not super confident in the item detection working, but I have a good idea on how to implement it, so we will see by the end of the week.