Weekly Status Reports – Page 3 – Team B0: Where's the Milk?

March 26, 2022March 26, 2022

Team Status Report for March 26, 2022

An overview of what we did:

We each attended the ethics seminar in person and that lead us to some design choice thoughts and considerations we had not gotten too on our own
- The main one was considerations on users customizing which items are available by our system to recognize
  - Different approaches had different security concerns and user requirement concerns i.e. users updating the global data set
We had a couple meetings and conversation to go over our design changes and considerations
- We decided to have a global data set of images we have prescreened and tested for security / ethical issues, and have tester to meet user requirements
  - Then the user’s can select from our global data set
  - In a real world environment, we would institute a system for user’s to request additional items for us to screen and add to the global data set

In terms of what we will do this upcoming week, we hope to do the following:

Meet to iron out the specific implementation of our integration components
- Get the raspberry pi’s running
- Finish sub systems so we can move to testing

The most significant risks that we know of thus far are:

Interconnection between our subsystems
- We have a more high level idea of how the whole system works, but now we have to programmatically begin to implement them in our code

Changes To our schedule:

There are no changes to our integration schedule of late, we are proceeding as planned
Takshsheel is changing his schedule where he initially wanted the counter completed by this week which is moving to next week.

March 26, 2022

Lucky’s Status Report for March 26, 2022

This week I did the following tasks:

I continued to work on the detection script
- I had to make reconsiderations and adjustments to the code I have written / been working on given the concerns explained in the next following section titled “Things I ran into worth considering”
Raised my concerns and spoke to my team about the design choices

Things I ran into worth considering:

User item customization
- Cons:
  - Can reduce accuracy of system
  - Can institute ethical issues
- Pros:
  - Enables users to customize to their store needs
  - Faster system that does not go through a large set of products that aren’t even available in store
  - A system that has all of the store’s available products
- Possible solutions:
  - Allow store owners to capture new images from their live feed i.e. add new items to dataset from live feed
    - Cons:
      - Can add ethical issues
        
        Misuse: unsolicited images taken of customers i.e. pervasive / private images
        
        If images are added globally
        
        Need additional infrastructure to protect against things such as predatory images that intentionally mismatch, inappropriate images, etc.
      - Requires a separate local data storage abstraction / subsystem if images are not uploaded globally
      - Will most likely add complexity that could be avoided with different design choice
      - Can reduce accuracy of system
  - Have a universal set of items we have tested and pre uploaded data on our side controlled by us, then allow local systems to select from those pre-validated items
    - Cons:
      - We may not have all items that a store needs
      - We will have a large data need that may have more items then users even use i.e. what if our customer base is mostly produce, but we have numerous items in home improvement => inefficient data storage
      - We have to add the necessary back end infrastructure and integration with the detection system to handle that => may be something to consider after mvp
    - Pros:
      - No
      - No security concerns of uploaded items / images from users
      - No accuracy concerns of uploaded items / images from users
      - No ethical concerns of uploaded items / images from users

In terms of schedule, depending on how testing goes upon completion of this second phase of object detection, I think I should be good on schedule. If testing goes unfavorably, I will have to step up some on my pacing.

What’s Next:

Now that we have solidified an approach for the item customization, I am going to move forward with those design choices in mind and finish the object detection version that I am working on

March 26, 2022

Allen’s Status Report for March 26, 2022

Discussed ethical concerns/precautions that we should consider for our project amongst our team and fellow classmates during our ethics lecture
Researched various ways to set up our wireless system – I have concretely decided on what to do and have the resources ready to start up the system besides one part that was ordered tonight (last piece of the system needed to be purchased)
Discussed in depth the details of integration along with making our system more accessible to more stores in general. In addition to other things, one major thing we decided is to add a global item data store for all stores and aisles and allow users to choose aisles/items to view on their UI.
In regards to the previous point, I mapped out how that would look on the web application in terms of code and design

I believe that our progress is a bit behind – we understand that the interim demo is on the horizon in less than 10 days and our final demo is in about a month. That being said, we want to make sure we have those deadlines in mind to finish as early and robustly as possible.

During this upcoming week, I plan to do the following:

Researching web sockets as that is a potential bridge that will allow our subsystems to connect with each other (for integration) (besides hosting the web application on the cloud)
Adding point #3 above for the web application
Finishing final parts of the web application (i.e. some UI and routing for the page that will show the data to the user)
When the last piece of the wireless system is here by mid-week, I’ll shift my focus to fleshing out the wireless system (the first half of the week will be web application dominant)
Figuring out how to set up our system and shelf/items for our interim demo
Assisting Lucky and Takshsheel with anything that they need

March 19, 2022

Lucky’s Status Report for March 19, 2022

This week I did the following tasks:

I finished producing a functional SIFT matching script that detected matches and produced results between some test images
I ported all of the object detection code I’ve been working on to a version control system to prep for collaboration and integration
I began a new python file/script to run continuously on a system in order to prepare something that can function in production
- I cleaned up the code, and modularized sections for readability and testing of subcomponents
  - This included modularizing the SIFt detection portion so we can more easily swap out the detection algorithm for other algorithms for benchmarking the best algorithm for our use case
  - I organized global variables for making the tweaking of threshold / customizable portions of the code easily accessible for benchmarking and optimization

In terms of schedule, it felt like I finally began catching up on the slack I had built up on weeks prior. I now have a clearer trajectory of what I want to get done soon

What Next

I want to finalize the transition to an almost production ready subsystem of the object detection algorithm for our MVP
- This entails finishing a functional main script that runs the detection and matching continuously with prompts from the terminal
  - The terminal prompts will service as a trigger mechanism to simulate the trigger from the motion detection subsystem
  - There is als a capture simulation to simulate capturing new images from the shelf camera for detection / matching

March 19, 2022

Team Status Report for March 19, 2022

An overview of what we did:

Wrote our Ethics assignments and discussed what we wrote amongst each other
Received feedback on the design report review and have noted those changes to be implemented for our final version of the report
Progress was made in the background subtraction for motion / people detection
Progress was made for the object detection as well as a potential solution to our SIFT / SURF issue
Ordered and received the remaining parts that we need for one complete wireless system (we were missing a SD card and a camera tripod)
Completed essentially 80% of the web application before spring break

In terms of what we will do this upcoming week, we hope to do the following:

Determine how our set-up will look for the interim demo (i.e. how are we going to set up the shelf?)
Set up the wireless system and attempt to send dummy data to a local machine (i.e. one of our laptops)
Wrapping up everything on the web application that doesn’t require the input data from the detection algorithms (i.e. designing the UI for the view that shows the data to the user, linking some more URL routes, etc.)
Continued progress on detection for both motion/people and objects

The most significant risks that we know of thus far are:

The most significant thus far is still the potential that we will have to work with 2 different camera systems considering the scarcity of raspberry pi’s right now
- We might have to use a laptop, webcam, and do a different control design
We are not sure how well the ARDUCAM lenses will work with images (i.e. we believe that it is a fisheye lens and we may need to order another type of lens if that doesn’t work out with object detection)
- We will determine how well the current fisheye lens works with object detection this week.
Another potential risk / consideration is integration since we have made progress in detection, but we need to really think through how are system will actually use the detection in an autonomous integrated manner
A risk we have in the background subtraction part of our openCV background subtraction part is that, there’s not been a resolution to keeping track of how many people are on the screen. Mitigation plan for this would probably involve getting help from our instructors and professors in CMU and research beyond for figuring out how to extract information from a grayscale image.

There have been no changes to our schedule, but in terms of the design, we realized through research in setting up our wireless system that we will need a SD card to run the RPI and the camera will need to be propped up with a tripod. As a result, we ordered both those parts this week and they have recently arrived. We will be setting up the system this week.

March 19, 2022

Allen’s Status Report for March 19, 2022

Worked on RPI design/research/ordering more parts and the Ethics assignment on Monday and Wednesday
1. ~5 hours
During the week before spring break, I spent roughly 18-20 hours setting up the web application (the login/register pages are complete with user authentication, the homepage is set up, and the UI is nearly finished) and writing up my portions of the design review document.

I believe that our progress is slightly behind – we got a lot of work done before spring break, but we’ve had some slack time since break. The necessary parts for the wireless transfer system arrived during the tail end of the week (as I research more, I realize how we need little things along the way, such as: SD card, tripod for camera, etc.), so I’ll be focusing on that this week.

During this upcoming week, I plan to do the following:

Setting up the wireless transfer system now that all of our parts for one system are here (our other RPI is hopefully coming during the first week of April. I say hopefully because it might get backordered, but the expected date of arrival is during the first week of April.)
Finishing final parts of the web application (i.e. some UI and routing for the page that will show the data to the user)
Figuring out how to set up our system and shelf/items for our interim demo
Assisting Lucky and Takshsheel with anything that they need

February 26, 2022February 26, 2022

Lucky’s Status Report for February 26, 2022

This week I did the following tasks:

I finished a BRISK algorithm version
- Last week I ran into an error using SURF and SIFT in which open cv kept giving runtime errors stating that they were patent protected / copyrighted, so I proceeded with BRISK
- Finished the BRISK implementation that extracted features from images of cereal and compared it to images containing cereal
  - I tried to tweak the filtering algorithm I used – Lowe’s ratio Test – and some other parameters to see if I could detect one cereal box from a group of cereal boxes, but the process did not seem to be productive
    - I plan to instead have the system detect if a cereal is present rather than for example honey nut cheerios vs regular cheerios for now, then branch to more detailed filtering in the future if possible since it is not part of our MVP
Attended the presentation, asked questions and took notes
- One important take away was that another group actually was able to use SIFT
  - We had previously pivoted from SIFT to BRISK because when I tried to use SIFT, open CV kept giving me an error stating that SIFT is a patented / copyrighted software so I could not use it, but the other team said they were able to use it
- I asked for their requirements file because I suspect that the patent issue can be avoided by using older versions of opencv and/or python
  - I began attempting to develop a similar detection algorithm with SIFT as I did with BRISK

In terms of schedule, it felt like I did more work than weeks prior, but that could be because I began shifting from research to implementation. As always, I have room to improve on my time management, but I do not feel any more behind than last week.

The main way I intend to catch up and do better this upcoming week is the following

Finish a comparable SIFT detection as I did with BRISK
Begin designing and implementing a process to actually use the detection for a pool of images to simulate the process of the algorithm looking for multiple items rather than fixed test items / images

February 26, 2022February 26, 2022

Allen’s Status Report for February 26, 2022

During this week, I worked on the following (in chronological order):

Worked on Design presentation with my team during Sunday night
1. ~2-3 hours
Reviewed/conducted peer reviews for the other teams during class time on Monday and Wednesday
1. ~3 hours
Went through a comprehensive Flask tutorial, researched and planned out how to upload/download from an AWS S3 bucket, researched on how to use an RPI with the ARDUCAM, and researched extra parts that we need to order (camera module and ribbon for RPI to ARDUCAM)
1. ~5 hours

In the past week, I’ve had two exams so my output this week was not as optimal as I would have liked it to be. This upcoming week should be much more free, allowing me to put in 12+ hours for sure.

I believe that our progress is behind, and I shoulder my fair share of responsibility. The key hindering factor for me (besides having midterms the past two weeks), is that I tend to mostly work on the tail ends of the week. This week, I will work for at least 3/5 of the weekdays for 2+ hours outside of class time. My goal output this upcoming week is 15+ hours.

During the next week, I plan to do the following:

Setting up the backend and frontend of our flask app
Sending dummy data from the RPI to a local machine (i.e. since we need to order a camera module, we cannot take pictures yet. Thus, I’ll send over dummy data to get used to using the RPI so that once our camera module comes in, we’ll be good to go)
Assisting Lucky and/or Takshsheel on object/human detection

February 26, 2022February 27, 2022

Takshsheel’s Status Report for February 26, 2022.

This week, I accomplished the following tasks:

Worked for a bit on the design review presentation delivery, since I generally struggle with presentations, so it took longer than I thought, but necessary in my opinion.
Made a working implementation of a background subtraction algorithm and a framework where we can test various background subtraction algorithms on a pre-recorded video. I tested a couple to see what they looked like myself. Images below are pictures of people moving across the camera. I tested 2 algorithms, MOG2 and KNN, documentation for which I found online.
Test for KNN. Multiple people in frame
Besides the algorithms above, I also tried to extract the data from the grayscale images to see if I could record how many people were in the frame, but I’ve not managed that yet. (Deliverable for next week)

In total I worked on capstone this week for about 11-12 hours, but for sake of overestimation of time I’ll claim 11 hours this week.

In terms of schedule, I feel like I’m finally on track because I’d said that I’d have a background subtraction program working and it is done as of now. For next week I plan on having an implementation where I can have a live video instead of a pre-recorded one, as well as extract information from the data that I get from openCV (eg: Number of people in frame)

February 26, 2022February 27, 2022

Team Status Report for February 26, 2022

An overview of what we did:

We presented our design review
We received feedback from the review and went over changes as a group
Progress was made in the background subtraction for motion / people detection
Progress was made for the object detection as well as a potential solution to our SIFT / SURF issue
Went through a comprehensive Flask tutorial, researched and planned out how to upload/download from an AWS S3 bucket, researched on how to use an RPI with the ARDUCAM, and researched extra parts that we need to order (camera module and ribbon for RPI to ARDUCAM)
We realized we need to order some additional inventory so we discussed the items and plan to submit an order this week

In terms of what we will do this upcoming week, we hope to do the following:

Setting up the backend and frontend of our flask app
Sending dummy data from the RPI to a local machine (i.e. since we need to order a camera module, we cannot take pictures yet. Thus, I’ll send over dummy data to get used to using the RPI so that once our camera module comes in, we’ll be good to go)
Continued progress on detection for both motion/people and objects

The most significant risks that we know of thus far are:

The most significant thus far is still the potential that we will have to work with 2 different camera systems considering the scarcity of raspberry pi’s right now
- We might have to use a laptop, webcam, and do a different control design
We are not sure how well the ARDUCAM lenses will work with images (i.e. we believe that it is a fisheye lens and we may need to order another type of lens if that doesn’t work out with object detection)
Another potential risk / consideration is integration since we have made progress in detection, but we need to really think through how are system will actually use the detection in an autonomous integrated manner
A risk we have in the background subtraction part of our openCV background subtraction part is that, there’s not been a resolution to keeping track of how many people are on the screen. Mitigation plan for this would probably involve getting help from our instructors and professors in CMU and research beyond for figuring out how to extract information from a grayscale image.

There have been no changes to our schedule, but in terms of the design, we noticed that we will most likely need to upload/download our processed images/data to the cloud (i.e. AWS S3 bucket). Originally, uploading information to the cloud was going to be a reach step, but in order for us to connect the processed data from the CV program to the Flask app, we are probably going to need an intermediate cloud storage space. Since we have used AWS before, we will most likely be moving forward with that.

We were able to get our background subtraction algorithm to detect the presence of motion in frame.