Weekly Status Reports

This week I did the following tasks:

• Integrated motion and object detection for live testing
  • Completed a function live setup
    • Used a webcam rather than raspberry pi uno to ensure functionality prior to dealing with increased complexity of raspberry pi
      • Use of single webcam to test out the pivot we recently decided which was to use a single camera for both motion and item detection
    • Physical parameters of setup
      • Camera placed 5 ft away from front of shelf and 3 ft above the found
      • Items Tested:
        • protein powder, quick oats, empty milk container, cereal, chia seeds, peanut butter, brownie mix, tuna, ziploc box, vegetable oil, paper towels
    • Recorded a video clip of the testing, and incorporated that into our final presentation
      • System was able to detect motion
      • Finite state control mechanism functioned properly to trigger the object detection
      • Object detection worked accurately – presence detection and bounding boxes were correct – for all of the items except the following:
        • empty milk, vegetable oil
• Continued working on the video script
• Worked on the project poster

In terms of schedule, I feel my biggest worry is setting up the wireless system with the raspberry pi’s and integrating control and file transfer with the website

What Next

• Establish working system with wireless raspberry pi camera setup
• Establish file transfer and control a.k.a. functional integration between raspberry pi and website

This week I did the following tasks:

• Worked towards integration
  • In order to make integration more seamless, I decided on the approach to include the object detection source code with the web app source code
    • This required transitioning the source code from a script that was to run on the raspberry pi camera subsystem to an internal library of functions that could be used by the web app using object oriented programming
      • This enables the web app to more easily supply the desired aisle and items, and receive the desired json object without too much hassle with transfer protocols
        • The web app will create a class instance for each aisle by calling on the object oriented library and supplying necessary information i.e. items expected in that aisle
        • Each class instance has its own set of functions for detection distributed detection for multiple aisles if needed
    • Also created a testing script for the library
•  Began planning for functionality of displaying an image of the detected shelf (the idea was a cool feature idea from Prof. Savvides that we will try after completing MVP)
  • The overall programmatic design I am working on is
    • Produce empty shelf image base layer
    • As the sliding window moves through during detection, store the presence of items along with the x and y coordinates
    • For each item that was detected as present, use the x and y coordinates to add an additional image layer in the location where it was detected
      • The images will be preprocessed stored images for our global item set
• Live testing and prep for demo
  • Finalized workable live capture code
  • Acquired groceries for testing and demo day
  • Began testing of live capture object detection
• Final Presentation
  • Put together tests and visualizations for the presentation
• Started working on the video script

In terms of schedule, I feel a bit pressured, basically hoping things go as planned, but preparing for when they do not, which is almost always the case for engineering projects. My biggest worry is the trigger mechanism between the background subtraction subcomponent and object detection subcomponent because it requires communication between multiple devices

What Next

• Test, test, and more test, because we need to produce quantitative data and documentation of our decisions that we have made thus far
• Finalize integrated system
• Work on setup for demo
  • Cameras and shelf locations / distances
  • Script / overview of our demonstration
• Work on video script
• Storyboard poster components

This week I did the following tasks:

• Deeper research into more complex segmentation approaches
  • Upon further research and designing of sliding window I found some disadvantages to the approach
    • Fixed window size or aspect ratio can result in not capturing the full objects in windows or expensive computation
  • I learned about modern techniques that were developed to tackle that issue i.e. R CNN and Fast R CNN, Faster R CNN, and more
  • Therefore, I compared K-means, Mean Shift, and Graph Based segmentation
    • K-Mean: predefined number of clusters, randomly set centroids (will be the mean(s) i.e. the average of the colors in a cluster) for each cluster, assign each point to the nearest mean thereby creating a cluster, then recompute the means and repeat until the means stop moving i.e. convergence
      • Pros:
        • Not as complex as the other two approaches
        • Very fast
        • Multiple initialization methods can be used
      • Cons:
        • Manually have to set number of clusters
        • Sensitive to initialization
        • Sensitive to outliers
    • Mean Shift: for each pixel – first compute mean of the pixels in a surrounding window, then move the pixel’s assigned centroid to the new mean location, repeat until the mean converges to the nearest mode which will be the local extrema i.e. the hill of a normalized density feature mapping. Each hill will be the centroid of the pixels around them thus separating the pixels into clusters
      • Pros:
        • Simple algorithm
        • No initialization required
        • Robust to outliers
      • Cons:
        • Computationally expensive
        • Sensitive to window size parameter
    • Graph Based Segmentation: each pixel is a vertex, an edge is produced between neighboring pixels, each edge is assigned a weight based on affinity / similarity between the pixels/vertices, then the graph is partitioned to subgraphs where pairs of pixels in the same subgraph have a high affinity, and pairs of different subgraphs have a low affinity i.e. a min-cut
      • Pros:
        • Complex segmentation capabilities
      • Cons:
        • There is a bias to cut into smaller segments
          • Requires normalizing cuts to favor larger subgraphs
            • No known polynomial time solution for minimizing the cost of a normalized cut, but there are approximations
• Upon comparing all of the solution approaches I used the following constraints:
  • Low computational expense and latency to meet user requirements
  • The camera is going to be fixed and the shelf is of a fixed size and distance from the camera
    • Results in images of fixed aspect ratio
    • Results in items having a constrained placement on shelf rows
  • We constrained the size of items for the scope of this project
• Therefore, I decided that given the constraints and scope of the project, an implementation of sliding window would be the feasible design solution
• Completed a functional sliding window matching algorithm
• Put together the shelf we bought

In terms of schedule, I think I am around where I expected to be. I would like to move faster this next week though to give some more leeway for hiccups as we get our project to something worthy of a decent final demo.

What Next

• Set up webcam and shelf with the following specifications to begin live testing such that the frame is composed of only the shelf for speed and accuracy since there would be minimal sliding window matching segments outside of shelf area
  • Distance from shelf to webcam:
  • Orientation of webcam:
    • Using SIFT matching which is supposed to be rotation invariant, but algorithm rotates test image for the sake of user visualization while testing
  • Height from floor of webcam:
    • Centered to shelf to allow for most forward facing camera angle of complete shelf
  • Figure out a set original captured Image aspect ratio and adjust sliding window accordingly