Posted on

Alex’s Status Report 4/16

Last week, we used the saws and equipment from carnival to cut up our wood into usable pieces. This week, we finally got the chance to actually assemble them into a folding platform to assist in our computer vision algorithm. This is necessary to create a nice, light background that is easy to mask out and better detect changes on. For example, when someone walks by, there won’t be significant change to the camera detection since it won’t see the foot, just small changes in lighting. The platform also helps to get more regular sizing of fruit. Users will know to hold it at a consistent height (next to the platform) which makes classification and segmentation much easier.

The folding feature is a small feature so that the fridge takes up less square footage when scanning is not in use.

This week I finished construction of the folding platform for the camera. using the pieces we had cut last week. We found various wood screws in tech spark to finish attaching the foldable hinges. The wood screws were slightly long, so I used hot glue to cover up anything dangerous that could catch.

We also purchased white spray paint for the platform to even further decrease any barriers of detection, but when Samuel ran the computer vision it appeared to work fine with just the light colored wood. I may still paint it next week for aesthetic purposes.

On the front end side, I have begun work on a nutrition information system in place of the recipe recommendation system. The recipe recommendation system was proving very difficult to integrate and test, since these timers are for very long periods of time and required extra frontend and backend integration. Instead, when clicking on items in the interface, it will display nutritional information about the item. For an apple, this will pop up: Apples - MSU Extension Nutrition | Montana State University

I’m still in the process of integration and will work to finish this up next week. Overall, we’re in very good shape. I just need to get the front end up and running on the tablet next week.

Posted on

Team Status Report – 16 Apr 22

This week we spent most of our time building the actual platform, bug testing and making minor incremental improvements on the system.

 

Figure 1: Example of Fully-Constructed Platform

Alex: Built the platform and installed camera onto it. Readying Jetson for integration. Started work on nutritional information

Sam: Helped Alex with building of platform. Tested scanning + collected data on actual platform. Working on various normalization algorithms to improve illumination robustness.

Oliver: Conducted more extensive bug testing and fixed a number of bugs in the backend. Made expiry date calculations generate more robust and user-friendly output (i.e. granularity on a daily level instead of on a microsecond level).

We are currently on schedule but need to speed up the integration process to leave more time for testing.

Next week, we will work on integrating the full system, by testing it with the Jetson and installing on a real fridge door. We are also in the midst of training with our self collected dataset and will try that next week.

Posted on

Samuel’s Status Report – 16 Apr 22

This week I mostly helped Alex to build the platform for the camera setup, and then attempted some basic tests on the setup. I also am currently training a model based on the data collected from previous weeks (and some from this week after getting the platform up)

 

Figure 1: Example of Fully-Constructed Platform

Figure 2: Example of self-collected data (here, of yellow squash) taken on new platform

 

More notably, I have been trying to improve the robustness of the change detector since I predict issues with the CV algorithm falsely detecting a change due to illumination changes from the opening of the door or a person walking around etc.

To this end, I tried using division normalization, where we divide the image by a very blurred version of the same image as per https://stackoverflow.com/a/64357685/3141253

While this allowed the algorithm to be very robust against illumination it also reduced the sensitivity significantly. One particularly problematic aspect occured because my skin color was similar to the birch wood color and so my hand was “merged” into the background. When displaying with a red apple, however, there was a better response, but not always enough to trigger changes. With this in mind, we have plans to color the platform white and hope that this might cause a larger differential between natural colors like that of skin and fruit, vs those of the artificially white background.

Another alternative is to play around with the saturation value of the HSV dimension. Or, since the main issue is going to be falsely detecting a change that moves the FSM into a background state falsely, we can potentially check for  a true extra object using a Harris corner detector.

Next week, in addition to trying the aforementioned items, I will also be helping to integrate the CV algorithm on the Jetson. We are currently on schedule but might need to speed up the integration component so we can spend more time on robustness testing.

Posted on

Samuel’s Status Report – 9 Apr 22

This week we successfully demoed an integrated, working system where a user could scan a fruit, and that fruit will be stored in a database and sent to the UI for modification and subsequent entry into the calendar.

I was very proud of the CV system’s robustness and smooth workflow as it was able to consistently scan different fruits under different environments and reasonable accuracy.

With basically an MVP already, we are definitely ahead of schedule but could add more features and improvements.

In particular, my next step is to collect more of our own data and train on it since the original dataset had no vegetables and obscure fruits. I spent most of the post-demo time this week collecting my own dataset. In fact, I collected over 800 images each for red peppers, yellow peppers and yellow squash, under various lighting conditions, positions, occlusions and shadows etc.

Examples of dataset images:

Next week, I will continue collecting dataset images on more vegetables like carrots, and begin work on making the hardware platform/stand for our integrated system. When the Jetson is ready, I will also begin work on that.

Posted on

Team Status Report – 9 Apr 22

This week, we were able to make our preliminary demo which went quite well. We were able to show the robustness of our CV algorithm and the full integration of our system (front and back end), through the successful scanning and adding of fruit to the database, and having the added fruit show up on the UI.

There were some slight hiccups during the actual presentation, in particular when we tried to show the remove feature that was added in a rush without much testing. In the actual demo/final presentation this will not happen, and there will probably be some sort of code freeze so that we will only demo robust and tested features.

Since we basically have an MVP complete already, we are slightly ahead of schedule, and have been and are currently working on making the chassis/hardware and adding on some key features.

Summary of some progress this week

  • Samuel: Collecting more data for various fruits and vegetables
  • Oliver: Fixed remove feature bugs
  • Alex: UI touchups, cut wood for chasis and platform

Next week,

  • Samuel, Alex: Work on chasis/platform, integrate Jetson
  • Oliver: Recipes and/or nutritiontal information, bug-test
Posted on

Oliver’s Status Report – 2 Apr 22

This week was an incredibly productive week for me because the primary blockers, such as the development environment and library issues have been tackled in the preceding weeks, leaving me free to implement the meat of the back-end swiftly. I delivered multiple API endpoints and worked closely with Alex and Samuel to integrate the API with their front-end and CV system respectively, producing a cohesive product very rapidly.

One thing I also achieved this week is that I actually refactored the database schema in order to better work with Samuel’s idea of sending a list of items the CV system has predicted, in descending order of probability, and presenting it to the user. The previous schema did consider such a use case but was not well designed for it, and did not store the CV’s predictions in the database at all. By swiftly refactoring (which was made much easier by the Prisma ORM and TypeScript!) I managed to deliver the results we wanted for demo week and beyond.

Going by my personal Gantt Chart, we are now exactly 1 day ahead of schedule. It’s a fantastic sign, and I believe that with this pace, we can work even faster. In the following weeks, I have identified several improvements my API could perform, such as coalescing multiple transactions within a single day, as well as fixing some security vulnerabilities.

Posted on

Samuel’s Status Report – 2 Apr 22

This week was an extremely productive week, and I was able to get a fully working CV system up and running, with various features such as motion detection, stability checks and sending the top 5 predictions to the backend via the API.

FSM – Motion Detection/Stability Check

Last week, I was able to get basic background subtraction/motion detection working that compares the current frame with the previous one to check for changes. This week, I created a simple FSM which would use this motion detection algorithm to check for changes that indicates a user wants to predict an image, wait for stability, and predict on that stable image. Then, when the user removes the fruit, the algorithm similarly detects that change, waits for stability, and returns to a “background state”.

This FSM allows for a stable and robust detection of an item, as opposed to the initial algorithm that performed predictions on a potentially-blurry “delta image”.

 

Fig 1: Prediction after stability

Fig 2: Background State after stability

Fig 3: Ready for next prediction

API Integration

After Oliver was able to get a simple API working, it was time to integrate the CV system with the backend. For this, I used the curlpp library to send a HTTP request in the form of a JSON array of item IDs to the server/backend after a successful prediction.

Normalization

I also attempted to experiment with different normalization techniques (and re-trained the network accordingly), including ImageNet’s original normalization statistics, to improve the robustness of the algorithm against different lighting conditions. However, for some reason, the normalized models always worked worse compared to the non-normalized ones, and therefore I am currently sticking to the unnormalized model that seems surprisingly robust. It has worked amazingly well on different backgrounds, lighting conditions and environments (I’ve tested using monitors and chairs as platforms, at home and on different parts of campus, and it works well in all cases with few problems).

Demo + Future Work

Moreover, we were able to get our algorithm interfaced and working with the front-end and back-end, in preparation for our demo next week. Now that we have a fully working and robust scanning system, we  are currently very much on schedule, and in fact slightly ahead. My next step is to work on

  1. Integrating with the Jetson hardware
  2. Collecting my own datasets
  3. Seeing if there are any bugs with my normalization algorithm that affected its accuracy (because normalization should fix problems not cause them)
Posted on

Team’s Status Report – 2 Apr 22

This week we made much progress as a group in preparation for demo day!

We were all individually able to get all our individual components working AND integrated together:

Individual Components

  • Samuel: Completed CV scanning system, allowing for accurate detection of scan/removal and prediction on stable image
  • Alex: Trained NN and got websockets coded and working/integrated with the API.
  • Oliver: Made a lot of progress this week. Almost fully integrated all available backend API endpoints with the front-end as well as the CV system. Product works end-to-end as a result. Also implemented “live-streaming” of CV predictions to all front-ends

Overall workflow

  1. User places fruit under camera in CV system
  2. CV system detects movement and waits for stable image
  3. CV system predicts on image and sends the top 5 predictions to backend via JSON API
  4. Back-end stores the CV predictions and their relative ranks, and emits a “newTransaction” event to client front-ends.
  5. Front-end receives event from the API and presents a prompt for the user to confirm the CV’s predictions and committing the transaction into the back-end database.

 

We are happy with our current progress / product right now, as everyone in the team worked hard on their individual parts, and were also able to integrate everything together smoothly.

With a working integrated MVP, our team is on track (in fact slightly ahead of schedule) to deliver all that was promised in earlier stages, and it is even possible that we can achieve some of our stretch goals, such as a door sensor to tell whether it is an item addition or removal.

The immediate tasks at hand (in order of priority) are:

  1. Item removal – System currently only supports item addition, but needs to have an interface to support removal. Should be completed quite fast.
  2. Hardware integration – system currently works off a laptop, but want it to work on our Jetson, which currently needs its motherboard fixed. Once the new one arrives, integration should be pretty fast.
  3. Chasis/Platform – Currently propping up the camera on white chairs or monitors facing a white table/table with piece of paper, but eventually need to have it mounted on a proper chasis and platform. However, the algorithm still works despite different environmental conditions, indicating its robustness!
  4. Data collection  – currently we have 14 classes of fruits that the algorithm can robustly detect (including apples, oranges, tomatoes, bananas, pears), but some of these fruits are not commonly bought (eg. pomegranate, kiwi, starfruit).  Would need to collect some data of our own for vegetables like carrots, bell peppers, broccoli, cauliflower etc
  5. Recipes – Was originally a stretch goal, but now something we can possibly do!
Posted on

Alex’s Status Report – 2 Apr 22

This past week, I worked on integrating scans with the API. We got websockets and a few extra We were able to complete the full pipeline of scan -> API -> popup on the frontend.

I also worked on getting the new neural network trained.

We are back on schedule and ready for the demo!

Posted on

Oliver’s Status Report – 26 Mar 2022

Indeed, as I’ve talked about in the team status report, we are now entering the phase of the project where the fun truly starts, and the product begins to take shape. This week, I kicked off the integration process between the front-end and back-end, ironing out kinks such as CORS header mismatches and API design issues, paving the road for full integration ahead. Right now, the front-end is now able to display real expiration dates fetched from the API.The back-end API had quite a bit of work done to it as well. Logic has been added for when items are replaced into the fridge, ensuring that the item count and expiry dates remain correct even when the same item is removed and placed back into the fridge. This is done by storing the state of the fridge – new items are only added to the tally when it is in “add” mode, i.e. when the user is loading new groceries from the store. All other additions in the “track” mode are considered replacements of previous removals.

We’re still healthy in terms of schedule, but with just a bit over a month left, time can get a little tight if we are not careful. Going forward, we will be integrating the back-end with the Jetson and CV system, and I will also continue working with Alex to bring other aspects of the front-end into fruition, such as editing previous transactions.