Posted on

Keaton’s Status Report (3/26)

I got very little done this week, outside of some minor admin stuff, like writing a rough draft for the ethics section of the final report. I apologize for this, and expect to make up for it this week.

Posted on

Team Status Report 3/26

While we’ve all been swamped with midterms and academic fatigue, work on the group project continues apace. Harry created a working version of OAuth2 (more on his blog). Jay is continuing to work on the wireframing, and will incorporate CSS shortly. Keaton is taking a much-needed respite from CV, and is looking into hardware design considerations.

We hope to see some progress on the database component of the website’s backend this week.

Posted on

Jay’s Status Report 3/26

Sorry for the late update, but there wasn’t much to provide, website is still a work in progress. I have the basic outline of the wireframe down, but now is the challenge of adding CSS to bring it to life. I’ll try to use bootstrap to make my life simpler.

Posted on

Team status report for 3/19

Made some progress with CV component:

  • Pixel diff is working sufficiently (may still need fixed light if we’re doing it in different lighting conditions)
  • Tested directly under the camera, and off to the side with anti-distortion of the image
    • Overall fine, a few standout items that were confused with each other, but nothing that can’t be handled on a case by case basis (specifically applesauce was a bother)
    • Have a confusion matrix based on the number of matches we saw for each situation (See Keaton’s report)

Harry got OAuth working for various services. Jay expanded bringing the wireframes to life, though he acknowledges progress is slower than he’d like.

Posted on

Keaton’s status report for 3/19

Mar 19th:

New, fully realized beautiful setup. Thank god for heavy weight bearing adhesive velcro straps and sticky putty.

Pixel diff is working, at least for all the situations I used it in excluding one (crushed tomatoes) (may still need fixed light if we’re doing it in different lighting conditions). No major issues to report there.

Tested two situations, one where the item was directly under the camera, and one where it was off to the side with anti-distortion of the image.

Overall results were fine, but definitely worse than the photos taken with the I-phone camera. also, the side shot was definitely worse then the center shots for most items Here is the confusion matrix based on the number of matches we saw for each situation. (the number shown is the number of (matches)/(number of matches for the best match for that particular query image))

Sideshot:

Applesauce Milk CrushedTomatos Shredded cheese Spaghetti BakingPowder Yogurt Beans Cereal Crackers
Applesauce 0 0 0.40678 0 0.147959 0.820513 0.869565 0.083333 0.7 0
Milk 0.462236 0.68254 0.20339 0.093023 0 0.846154 0.942029 0.375 0.766667 0.113861
CrushedTomatos 1 1 1 1 1 1 1 1 1 1
Shredded cheese 0.954683 0.936508 0.559322 0.937984 0.908163 0.974359 0.956522 0.166667 0.772222 0.935644
Spaghetti 0.960725 0.952381 0.254237 0.984496 0.80102 0.974359 0.913043 0.458333 0.844444 0.970297
BakingPowder 0.845921 0.809524 0.694915 0.728682 0.739796 0 0.884058 0.125 0.811111 0.79703
Yogurt 0.867069 0.857143 0 0.806202 0.816327 0.871795 0 0 0.75 0.866337
Beans 0.975831 0.968254 0.644068 0.984496 0.97449 0.948718 0.971014 0.125 0.811111 0.980198
Cereal 0.975831 0.968254 0.644068 0.968992 0.969388 0.871795 0.898551 0.208333 0 0.985149
Crackers 0.97281 0.936508 0.237288 0.945736 0.954082 0.948718 0.913043 0.208333 0.783333 0.712871

Center Shot:

Applesauce Milk CrushedTomatos Shredded cheese Spaghetti BakingPowder Yogurt Beans Cereal Crackers
Applesauce 0.151487 0.139073 0.0261 0.028105 0.082428 0.001076 0.002422 0.004927 0.041316 0.295754
Milk 0.081465 0.04415 0.035048 0.02549 0.096742 0.000923 0.001076 0.003359 0.032135 0.262079
CrushedTomatos 0 0 0 0 0 0 0 0 0 0
Shredded cheese 0.006865 0.00883 0.019389 0.001743 0.008885 0.000154 0.000807 0.004479 0.03137 0.019034
Spaghetti 0.00595 0.006623 0.032811 0.000436 0.01925 0.000154 0.001615 0.002912 0.021423 0.008785
BakingPowder 0.023341 0.02649 0.013423 0.007625 0.025173 0.005996 0.002153 0.004703 0.026014 0.060029
Yogurt 0.020137 0.019868 0.043997 0.005447 0.017769 0.000769 0.018568 0.005375 0.03443 0.039531
Beans 0.003661 0.004415 0.01566 0.000436 0.002468 0.000308 0.000538 0.004703 0.026014 0.005857
Cereal 0.003661 0.004415 0.01566 0.000871 0.002962 0.000769 0.001884 0.004255 0.13772 0.004392
Crackers 0.004119 0.00883 0.033557 0.001525 0.004442 0.000308 0.001615 0.004255 0.029839 0.084919

A few standout items that were confused with each other: Applesauce just generally had a larger amount of descriptors, so there were more matches. I think there was also an issue with crushed tomatoes where the pixel diff failed. Overall, I don’t think there’s anything that can’t be handled on a case by case basis. However, I’m somewhat concerned that our ability to recognize when we have an unsupported object will be more difficult.

Overall, I’m mostly happy with the state of the CV component, it’s at least minimally functional. I will likely move to help working on the other stuff to get the demo working. I may also try adding some checks for the stuff that was confused, if I can find something simple.

Posted on

Harry’s Status Report for 3/19

This week I have been looking into setting up the user registration system. Although I was blocked by the Facebook OAuth for two days, after looking into Google and Facebook’s API documentations and python-social-auth’s codebase, I was able to get the sign-in components working. By default, I don’t have access to a user’s email from the Facebook OAuth API (which I need to finish the user registration steps), and I don’t have access to the user’s profile picture from both services (which is something nice to have for the interface), so I looked into ways I could retrieve this information. After digging through many documentations and StackOverflow examples, I was able to figure out how to fetch it from Google and Facebook’s API endpoints. Specifically, I found the correct endpoints from both services so that I could send GET requests with an access token provided from OAuth to get the information I need. As shown below, while the resulting code only consists of a couple lines, it took a lot of documentation reading and trails and errors to find the correct parameters and API endpoints to make it work. Since the ways I can fetch the information are mostly the same for both services, the next step will be coming up with a suitable design pattern to achieve better reusability and scalability in case we need to add more OAuth providers in the future.

Next week I will finish up the user registration system and start working on the view methods for other pages. Since we have designed the class models for the database, the implementation should be trivial when compared to the OAuth component. Specifically, I will create the User model and a corresponding form to send to the user if they log in for the first time to complete their registration. I will also work with Jay to connect the backend logic with the user interface so that we can have a working prototype.

Posted on

Jay’s Status Report for 3/19

This week I had several revelations regarding the flow of the project related to scheduling and the work itself.

  1. Progress on the webapp is slower than I had anticipated. While we have the basics working for OAuth and various pages, I still haven’t integrated as much of the dynamic AJAX as I would’ve liked by this stage. I’m still struggling with the details of social-django as well, but I’m finding support documents as issues arise.
  2. I’m still trying to figure out how to use Bootstrap and perhaps ease the burden of CSS. I’m cognizant that this isn’t a majority component of the capstone, but given my inexperience with CSS I still would rather look into this.
  3. I’m realizing in our Gantt chart that we need to account for days when we would have less time to advance the project than we had planned. For example, days that the University gives us off like most recently spring break and the upcoming Carnival; or preparing documents and presentations for the Capstone. While we haven’t fully completed our revamp of the Gantt chart, we definitely need to shift some items and be realistic about our productive abilities on those days.

In the coming week, I hope to have a working version of the wireframes I presented earlier in the semester, perhaps not with live user data but with a set of dummy data.

Posted on

Keaton’s Status Report 3/12

During break, I managed to make some progress, but not as much as I ideally should have. I made progress in two area’s, the first being pixel diff localization, and the second being camera setup/image comparison.

Using just subtraction for the pixel diff didn’t really work, due to very minor changes in lighting conditions. To resolve this, I used a slightly more involved method that I got from stack overflow. I calculated the Structural Similarity Index, located contours, and got the bounds from them worked pretty well:

I initially figured that the largest contour region will likely always be the object we are looking for, but this proved to be false (the shadow was bigger both in the example with my hand and the example with the applesauce farther down), so we may need some overhead lighting to prevent that from happening. Overall, I’m pretty happy with the localization code itself, with the correct lighting, it’ll work fine.

The actual setup stuff was a mixed bag. I made a exceptionally high tech setup for the camera apparatus, shown below:

As you can see, it’s the camera module, sticky putty’d to an empty hand sanitizer bottle, with some velcro strips attached. There are equivalently spaced velcro strips on the ceiling of my cabinet, so I can attach/rotate the angle of the camera as needed. This worked fairly well for the camera itself, but I had to manually hold the RPI in place, which led to a bit of shifting/jiggling, which screwed the pixel diff. The RPI belongs to Harry, I didn’t want to attach the velcro strips (which are really hard to remove and leave a lot of sticky gunk behind) without his permission. I also wasn’t certain that the velcro strips would hold the weight of the RPI, and I didn’t want to break something that wasn’t mine. Despite this, I had one decent photo of the applesauce where the pixel diff worked pretty well (omitting some changes in the RPI/background outside of the cabinet, and the shadow)

I manually cropped the image, and did a quick pairwise comparison using only the applesauce image, and the results were REALLY bad. We got at 5 matches (on shredded cheese), out of several hundreds of possible features. So, it seems that we’ll likely need to enforce either facing the products so that the label is directly towards the camera, or we’ll have to move the camera to be on the door, and prevent the user from placing large objects in front of smaller ones.

To summarize, while I made a fair bit of progress, I think that I’m still behind where I probably should be at this point in time. Thankfully, I think that most of the blockers are just some design stuff about how to handle label visibility, the code that was used to do the initial algorithm comparison was fairly easy to extend now that I know what I’m doing.

Posted on

Team Status Report March 12

This week was fairly uneventful, as it was the week of Spring break, and the week prior we were primarily focused on finishing the design document. During the current week, some very minor prototyping/testing occurred with the fish eye lens camera, with the camera in bird’s eye position. The localization/pixel diffing worked pretty well (with one caveat) but the results when performing sift on the images were pretty bad, so will likely need to iterate as a group (see Keaton’s status update).