Posted on

Shubhi’s Status Report for 4/27

Achievements

This week I spent time on the fullness detection model, playing around with different implementations for detecting the fullness to see what would increase the accuracy. Right now the carts all seem to be detecting to be similar fullnesses, despite having different fullness. The current approach was using edge detection to count the contours, but since our system is capturing the carts at a relatively slow fps, the carts are also a little bit blurry, rendering the edge detection a little inaccurate. Due to this, another approach I was playing around with was observing colors, and the number of black pixels there are in the frame. This also didn’t seem to the have the best accuracy, so I am still in the process of figuring out what could be the best approach.

Progress

Since I am not really happy with the current accuracy of the fullness detection, so I will be spending the next two days to increase it before the final poster is due, after which we will be working on the final video and report. We also need to work on the final demo because all of our testing has been done at Salem’s, but once we have a 1-hour long video to display on a monitor for the demo, we also need to make sure that our system will work for our makeshift in house checkout system that we will be demoing in person in Wiegand.

Posted on

Simon’s Status Report for 4/27

Accomplishments

I presented this week. I also went to Salem’s three separate times to try and get footage to test with, but was only successful the 3rd time (see Brian’s status report for more details). I did discuss alternative camera angles for our cashier throughput camera with the manager and the cashiers on my first visit this week (so I didn’t go for nothing), so I realized that I would need to train another model for the new angle (this time, a generalized one using the EgoHands dataset). The model seems to perform rather well, but we need to confirm tomorrow with further testing at Salem’s. On the 3rd visit, I got some data to replace the lost footage from last week, and realized that my shopping cart detection model was also failing to perform on our new camera angle for the small shopping carts (which was surprising, because it worked both from overhead and from a side view, but I guess there was insufficient training data for in between the two angles?). I annotated and added our new images of carts into the old dataset and retrained the model, which is now somewhat improved in terms of performance. However, I’m not confident that it predicts with high enough confidence on carts (it’s around .5 confidence), so we will need to test this as well to see if it ends up being reliable enough. If not, I will probably try to get more footage and just add more data, at the risk of overtraining the model to Salem’s.

Progress

Obviously, we are still very behind. We need to go to Salem’s tomorrow and finalize testing by Monday so that we can prepare for the final demo. As such, I haven’t bothered with the pose estimation for line detection that I mentioned last week, since failure to detect shopping carts/throughput is a much more pressing issue. Also, we need to configure the RPi’s to CMU wifi for the final demo (which is weird with CMU-DEVICE), but we can just use a phone hotspot instead, so this isn’t much of a concern.

Posted on

Team Status Report for 4/27

Risk Mitigation 

Our biggest risk this week is being able to test enough and fine tune our system for accuracy. We want to collect a variety of data in order to make sure our system is robust and we are able to include the relevant information in our final poster and report. We plan on going to test at Salem’s almost every day this week, going at different times (sometimes in the morning to get data for when there are fewer people around, and sometimes at night from around 5-8 PM, where traffic is greater). This way we will have a variety of different tests to show for when it is time to write about testing in the final report. 

Design Changes 

We are exploring another way to measure throughput, instead of using our original camera angle which some cashiers were uncomfortable with, we are looking into a different camera angle (instead of overhead, we are looking into an angle that is on/near the conveyor belt). However, this change is not final yet. 

Schedule Changes 

We have no schedule changes as of this week. 

Tests 

Below are some results from testing the upload and download time when recording different length videos. Interestingly, there is a spike in upload time from 2 to 3 seconds, but the internet that this was tested on (Brian’s) is spotty, so that might have to do with the abrupt increase. Also, upload and download times remain quite stagnant from 5 second to 6 second videos. Based on these results, we figured it would be best to use 2 second videos, as the upload and download time seem to work best for meeting our latency constraints (< 5 seconds to compute a result after initially processing the video feed). It would be possible to go up to 4 second videos, but the spike from 2 to 3 second videos is a deterrent. 

Duration Upload Time (Averaged over 10 trials)  Download Time (Averaged over 10 trials)
1 second 0.61s 0.32 s
2 seconds 0.92 s 0.47 s
3 seconds 1.35 s 0.53 s 
4 seconds 1.52 s 0.55 s 
5 seconds 1.61 s  0.57 s
6 seconds 1.65 s 0.59 s
7 seconds 1.80 s 0.68 s 
8 seconds 1.92 s 0.72 s
9 seconds 2.21 s 0.76 s
10 seconds 2.3 s 0.88 s 

In terms of testing relative fullness, we have tested with various different pictures and footage from Salem’s in order to check the accuracy of the module. From these results, we determined that fullness calculations need a bit of tweaking in order to increase the accuracy. The module performs especially poorly with a very full cart as shown below. We have tried to decrease the Gaussian blur, which has increased the accuracy for these very full carts to a degree, but the calculations are still a bit unreliable, so we will need to adjust accordingly.



Posted on

Brian’s Status Report for 4/27

Accomplishments 

This week, I went to Salem’s with Simon and Shubhi early on in order to get approval to test our full system, but we weren’t able to do so then because of availability issues with the IT lead. Simon and I then went later on in the week, and we were able to get approval to test, and we got some data to use for testing our fullness calculating module and line counting module. I also went to Giant Eagle this week and took pictures of a few different cart configurations (mostly very full carts), like so:

Because there is noise (in the surroundings), I cropped out the images according to the bounding boxes displayed when running the shopping cart model alone and tested relative fullness on them, but found that extremely full carts are for some reason hard to detect with the current implementation. I will need to refine the implementation in the next few days in order to hopefully improve accuracy, but the module seems to work fine with carts that aren’t as full. Testing the relative fullness module on the test footage we got from Salem’s, results are very similar. The carts with much less seem to be much more accurately counted for by the relative fullness module than the carts that are nearly full. I have tried decreasing the amount of Gaussian blur in order to increase accuracy, and while the returned fullness values are somewhat more corrected, they are still quite inaccurate (returning 65% fullness for the above picture on the left), which is concerning. I have also worked on testing the upload and download speed of different length videos, and I’ve created graphs of the relationship between video length and upload/download time. My internet is a bit spotty, so I am not sure if this graph is a completely accurate depiction of the relationship between the video time and the upload/download times. Here is my code and the resulting graphs: 

Duration Upload Time (Averaged over 10 trials)  Download Time (Averaged over 10 trials)
1 second 0.61s 0.32 s
2 seconds 0.92 s 0.47 s
3 seconds 1.35 s 0.53 s 
4 seconds 1.52 s 0.55 s 
5 seconds 1.61 s  0.57 s
6 seconds 1.65 s 0.59 s
7 seconds 1.80 s 0.68 s 
8 seconds 1.92 s 0.72 s
9 seconds 2.21 s 0.76 s
10 seconds 2.3 s 0.88 s 

Progress 

In terms of progress, we are still behind schedule by quite a bit. We should be testing the fully integrated system by now, but we have run into a few roadblocks with testing at Salem’s due to discomfort from the cameras by the employees. However, since we figured out workarounds to these issues, we should hopefully be able to get testing done by tomorrow or Monday in the worst case. I will work on trying to increase fullness calculation accuracy for the next day or two while we continue to gather testing data. 



Posted on

Shubhi’s Status Report for 4/20

Achievements

I worked on increasing accuracy on cashier throughput, worked through different approaches and implemented them and test them out for the most accurate approach for our use case. After that, I worked on a module that determined when a person was done with the line, for which I also did a little bit of testing. We also went to Salem’s and I created structures to attach the cameras to so that cameras could be in position for our system around the checkout counters. At Salem’s I also gathered more data to individually test the fullness acquisition module for accuracy.

Progress

Each individual component is working, and our system is also running end to end, but we are running into issues with testing at Salem’s due to concerns employees have been having with the cameras we set up. We are in the process of working something out with Salem’s but it is limiting the amount of testing we can do right now. I think the next big thing is to work on demoing and testing in a self constructed grocery checkout lane setup.

New Experiences and Learning Strategies

During this project I have gained a lot of new skills. One of the biggest ones is learning to train a model for object detection, which I learned from examples and articles online. Another thing that I learned to do was use a SQL database. I never took a class on it, so I relied on documentation and sites like stack overflow to help me achieve the goals I had for the database in the project. A lot of times for these new skills I relied on researching what I wanted to achieve to see if anyone had ever done something similar, and adapt from that to accomplish what I wanted to.

Posted on

Simon’s Status Report for 4/20

Accomplishments

In the past 2 weeks, Brian and I collaborated together for most of the work. To start, Brian and I managed to get uploading from our Raspberry Pi’s to S3 working, and then we were able to download as well. After that, we came up with a really simple synchronization method, which was just uploading a .txt file to indicate an upload was complete and then downloading the .txt file frequently to check if an upload had occurred (since the file is very small, there’s negligible cost to doing this).

Brian and I then worked on line detection, in which we used YOLOv8’s pretrained model for detecting people, using the bounding boxes to keep track of their location, and then determined whether people should be considered in line by distance and angle between them. To make it more robust, we tried to add functionality to only count people as in the line if they’ve stood still for a few seconds, which should eliminate people just passing by. However, we weren’t able to get this to work reliably. I was thinking of implementing pose detection instead, so we could eliminate counting passersby by checking whether they are facing forward (in line) or facing to the side (passing by), but we will do this next week if we have the chance.

Lastly, we spent some time testing in Salem’s this week. We spent Monday trying to set up and resolve Wi-Fi issues between our RPi’s and Salem’s. We went back on Thursday and managed to get some good footage using an actual checkout lane to test our individual components. We also went earlier today, but were unfortunately not allowed to test this time.

Progress

The main issue is that we didn’t get as much testing done as I would have liked (and also not enough to create much of a presentation). I’m planning to go back to Salem’s tomorrow and try again at a better time for them. Less importantly, I was going to try and implement pose detection to improve our line detection.

New Knowledge

Since I haven’t done anything with computer vision previously, I had to learn how to use OpenCV, which I accomplished by completing a short boot camp from their website. Afterwards, I mostly used YOLOv8’s documentation along with a few articles I found online to learn how to train a model and run inference with it. Afterwards, I used some more online articles to figure out how to set up a headless RPi. For accessing the camera module, I used the picamera2 documentation and some of the examples that I found inside. Lastly, I had to use forums quite frequently when I ran into issues setting up (stuff like how to download packages and change the WiFi settings).

Posted on

Team Status Report for 4/20

Risk Mitigation 

A risk for us with gathering test data is making sure that we can go at times where there is a decent amount of traffic at Salem’s and we can get a sufficient amount of testing data. We have already went to Salem’s a few times where it wasn’t busy enough to set up three checkout lanes. The way we are mitigating this is by going to test our system near peak hours instead of too early in the morning on a weekday where only 1 or 2 checkout lanes are available. Furthermore, if we do go at a lower traffic time, we want it to be a time where there are at least 2 checkout lanes running because we want to at least have enough traffic to have our system compute the speeds of two different lanes and compare them properly. 

Design Changes 

We haven’t made any changes to our design. 

Schedule Changes 

We haven’t made changes to our schedule as of this week, since we already have a day-to-day schedule. 

Posted on

Brian’s Status Report for 4/20

Accomplishments

This week, I went to Salem’s multiple times in order to try and get testing data. On Monday, Simon and I went twice (second time with Shubhi as well) to get WiFi on the RPis up and running and debug an error with S3 bucket downloading/uploading. We ran into an issue in the morning with the public WiFi having an “OK” prompt, which effectively blocks connections on a Raspberry Pi. However, Salem’s was kind enough to give us their private network, and so we were able to set up WiFi on all of the Raspberry Pi 4s. In the downtime between our trips to Salem’s, I worked with Simon to make detecting the number of people in a line more robust and to at least work somewhat. All three of us then went on Thursday to actually set up our full system and let it run, but we ran into issues debugging (since we needed to copy code to all of the RPis and then change minute aspects in order to upload different data), and fixing the camera angles, since the RPi camera modules have a very short connector the the RPi itself. Today we went again, but weren’t able to collect data from the full system running because a few of the employees there today were uncomfortable with being recorded. Therefore, we took pictures of people’s shopping carts, and used an empty checkout lane to record videos for testing throughput calculations. I also started working on the final presentation slides, since that’s coming up. 

Progress

In terms of progress, I am now behind the day-to-day schedule that I created by a few days, mostly because setting up the integrated system at Salem’s had so many setbacks preventing us from running the full system for data collection. In terms of what I will be doing this upcoming week to catch up, we need to test our integrated system completely. Therefore, I’ll try to help for finalizing individual component testing, and after that we can try to go on a day where the employees at Salem’s won’t be uncomfortable with their hands being recorded for throughput. 

New Tools

In terms of new tools and technologies that I managed to learn over the course of this semester, I was quite unfamiliar with most of the technologies used in our project. I didn’t know how to use YOLO, I never used AWS S3 buckets, and I didn’t do much programming at all with Raspberry Pis prior to this semester. My method of learning how to use these tools and technologies was to mainly browse the documentation available for most of my needs, and checking various forum posts to debug when needed were vital to learning how to implement various things for this project. Sometimes reading the forum posts led to pitfalls (i.e with the raspberry pis, one of the features for getting addition WiFi connections was only supported on a legacy OS, and so I used up a lot of time trying to implement the wrong solution).

Posted on

Simon’s Status Report for 4/6

Accomplishments

On Wednesday, I flashed the Raspberry Pi OS onto a microSD card and set up it up as a headless RPi. On Thursday, I attached the RPi camera module and made sure that I could record and save a video using the picamera2 module (took me unnecessarily long to realize that I needed picamera2 and not picamera). On Friday, I also made sure that I could record and save a video from the USB camera. On Saturday, I got the camera module and the USB camera to capture and save videos simultaneously, so Brian and I can put our work together tomorrow and see if we can record and then upload/download from S3 successfully. I also spent some time trying to stream from my RPi to a website, but I couldn’t figure out why I wasn’t able to connect, so I think I’ll just scrap that approach (Plus, it’s probably a bad idea since anyone who can view the website can view all the footage).

Here’s the code with the camera module and USB camera recording simultaneously:

Progress

I’m keeping up with the new day-by-day schedule that Brian created, but I was hoping to be further ahead of schedule by now because the RPi stuff should not have taken as long as it did. For next week, it seems like it should be easy enough to put Brian and I’s work together tomorrow, and from there I will just continue following the schedule outlined in Brian’s status report.

Testing

For what I’ve worked on/plan to finish, which are shopping cart detection and line detection, I could test by going to Salem’s and setting up a camera in the checkout lines for 10 minutes each (since there are 6 checkout lines, this will total an hour or so) and just mark down what percentage of the shopping carts are correctly detected (correctly meaning detected by the time they enter the bounding box). For line detection, I will look at how many people are in the line from the camera footage and see if the correct number of people are detected as being in the line and keep track of how many people we are off by.

Posted on

Team Status Report for 4/6

Risk Mitigation 

A risk in our project that we are trying to mitigate at the moment is whether or not our intended platform for storing data will be sufficient in terms of upload/download speed. Since we are planning on uploading camera footage from Salem’s and immediately using it and discarding it in real time, it is necessary to make sure that the platform we choose can be fast enough to allow for this data transfer from RPi to laptop to work properly. Therefore, we will test with uploading data from our RPis to cloud with AWS S3 buckets, and if those do not meet our requirements, then we can pivot to Dropbox or other distributed storage alternatives. 

 

Design Changes

Since we are using RPis to use real-time data from Salem’s, we are using AWS as a middleman in the content delivery process from the store to our remote laptops, which can then use the data for testing purposes on our system. 

Schedule Changes 

We have created a rough draft schedule for the last few weeks of the semester to stay on track: 

 

Brian Shubhi Simon
April 3rd, 2024, Wed Fix environment for cv2.imshow, look into options for RPi data transfer Purchase requests for cameras/displays/RPis RPi setup 
April 4th, 2024, Thursday Fix environment for cv2.imshow/RPi setup Talk to Salem’s about using their Cameras, Wifi, installing our own cameras, reimplement fullness acquisition RPi setup 
April 5th, 2024, Friday RPi setup, set up server/method for data uploading from RPi Reimplement fullness acquisition RPi access camera module and USB camera simultaneously 
April 6th, 2024, Saturday Set up server/method for data uploading from RPi Reimplement fullness acquisition RPi access camera module and USB camera simultaneously 
April 7th, 2024, Sunday Set up server/method for data uploading from RPi, test uploading multiple cameras data per raspberry pi, implement fetching Improve accuracy of fullness acquisition RPi access camera module and USB camera simultaneously, test uploading multiple cameras data per raspberry pi, implement fetching
April 8th, 2024, Monday Go to Salem’s, add pictures to dataset to train for throughput, implement line detection Go to Salem’s, improve accuracy of fullness acquisition Go to Salem’s, implement line detection, add pictures to dataset to train for throughput acquisition
April 9th, 2024, Tuesday Implement line detection, retrain throughput model Improve accuracy of fullness acquisition Implement line detection
April 10th, 2024, Wednesday Implement line detection, retrain throughput model Improve accuracy of fullness acquisition Implement line detection
April 11th, 2024, Thursday Increase accuracy of throughput acquisition, go to Salem’s and install cameras Increase accuracy of throughput acquisition, go to Salem’s and install cameras Increase accuracy of throughput acquisition, go to Salem’s and install cameras
April 12th, 2024, Friday Test throughput, line, fullness acquisition with footage from Salem’s Test throughput, line, fullness acquisition with footage from Salem’s Test throughput, line, fullness acquisition with footage from Salem’s
April 13th, 2024, Saturday Test throughput, line, fullness acquisition with footage from Salem’s Test throughput, line, fullness acquisition with footage from Salem’s Test throughput, line, fullness acquisition with footage from Salem’s
April 14th, 2024, Sunday Test throughput, line acquisition with footage from Salem’s, integrate line detection Test throughput, line acquisition with footage from Salem’s, integrate line detection Test throughput, line acquisition with footage from Salem’s, integrate line detection
April 15th, 2024, Monday Integrate line detection, purchase toy carts, prop items, test integrated system Integrate line detection, test integrated system Integrate line detection, test integrated system
April 16th, 2024, Tuesday Test integrated system Test integrated system Test integrated system
April 17th, 2024, Wednesday Test integrated system Test integrated system Test integrated system
April 18th, 2024, Thursday Figure out test configuration for final demos, dry runs with test configuration  Figure out test configuration for final demos, dry runs with test configuration  Figure out test configuration for final demos, dry runs with test configuration 
April 19th, 2024, Friday Slack Slack Slack
April 20th, 2024, Saturday Slack Slack Slack
April 21st, 2024, Sunday Slack Slack Slack
April 22nd, 2024, Monday, FINAL PRESENTATION
April 23rd, 2024, Tuesday
April 24th, 2024, FINAL PRESENTATION

 

Testing 

Using data from Salem’s market, we plan on validating our system by setting up our system there, letting it run, and observing how frequently it is the case that an individual that uses the system before the next one gets out of the store or to the checkout line faster. We will then tally the total number of users that are able to get to the front of a checkout line or out of the store faster and divide it by the total number of users in order to measure accuracy. Furthermore, for testing the speed of the system, we can programmatically check how fast the system is from start of computation to displaying the result, and check if it is under our design requirements.