Posted on

Team Status Report – 16 Apr 22

This week we spent most of our time building the actual platform, bug testing and making minor incremental improvements on the system.

 

Figure 1: Example of Fully-Constructed Platform

Alex: Built the platform and installed camera onto it. Readying Jetson for integration. Started work on nutritional information

Sam: Helped Alex with building of platform. Tested scanning + collected data on actual platform. Working on various normalization algorithms to improve illumination robustness.

Oliver: Conducted more extensive bug testing and fixed a number of bugs in the backend. Made expiry date calculations generate more robust and user-friendly output (i.e. granularity on a daily level instead of on a microsecond level).

We are currently on schedule but need to speed up the integration process to leave more time for testing.

Next week, we will work on integrating the full system, by testing it with the Jetson and installing on a real fridge door. We are also in the midst of training with our self collected dataset and will try that next week.

Posted on

Samuel’s Status Report – 16 Apr 22

This week I mostly helped Alex to build the platform for the camera setup, and then attempted some basic tests on the setup. I also am currently training a model based on the data collected from previous weeks (and some from this week after getting the platform up)

 

Figure 1: Example of Fully-Constructed Platform

Figure 2: Example of self-collected data (here, of yellow squash) taken on new platform

 

More notably, I have been trying to improve the robustness of the change detector since I predict issues with the CV algorithm falsely detecting a change due to illumination changes from the opening of the door or a person walking around etc.

To this end, I tried using division normalization, where we divide the image by a very blurred version of the same image as per https://stackoverflow.com/a/64357685/3141253

While this allowed the algorithm to be very robust against illumination it also reduced the sensitivity significantly. One particularly problematic aspect occured because my skin color was similar to the birch wood color and so my hand was “merged” into the background. When displaying with a red apple, however, there was a better response, but not always enough to trigger changes. With this in mind, we have plans to color the platform white and hope that this might cause a larger differential between natural colors like that of skin and fruit, vs those of the artificially white background.

Another alternative is to play around with the saturation value of the HSV dimension. Or, since the main issue is going to be falsely detecting a change that moves the FSM into a background state falsely, we can potentially check for  a true extra object using a Harris corner detector.

Next week, in addition to trying the aforementioned items, I will also be helping to integrate the CV algorithm on the Jetson. We are currently on schedule but might need to speed up the integration component so we can spend more time on robustness testing.

Posted on

Samuel’s Status Report – 9 Apr 22

This week we successfully demoed an integrated, working system where a user could scan a fruit, and that fruit will be stored in a database and sent to the UI for modification and subsequent entry into the calendar.

I was very proud of the CV system’s robustness and smooth workflow as it was able to consistently scan different fruits under different environments and reasonable accuracy.

With basically an MVP already, we are definitely ahead of schedule but could add more features and improvements.

In particular, my next step is to collect more of our own data and train on it since the original dataset had no vegetables and obscure fruits. I spent most of the post-demo time this week collecting my own dataset. In fact, I collected over 800 images each for red peppers, yellow peppers and yellow squash, under various lighting conditions, positions, occlusions and shadows etc.

Examples of dataset images:

Next week, I will continue collecting dataset images on more vegetables like carrots, and begin work on making the hardware platform/stand for our integrated system. When the Jetson is ready, I will also begin work on that.

Posted on

Team Status Report – 9 Apr 22

This week, we were able to make our preliminary demo which went quite well. We were able to show the robustness of our CV algorithm and the full integration of our system (front and back end), through the successful scanning and adding of fruit to the database, and having the added fruit show up on the UI.

There were some slight hiccups during the actual presentation, in particular when we tried to show the remove feature that was added in a rush without much testing. In the actual demo/final presentation this will not happen, and there will probably be some sort of code freeze so that we will only demo robust and tested features.

Since we basically have an MVP complete already, we are slightly ahead of schedule, and have been and are currently working on making the chassis/hardware and adding on some key features.

Summary of some progress this week

  • Samuel: Collecting more data for various fruits and vegetables
  • Oliver: Fixed remove feature bugs
  • Alex: UI touchups, cut wood for chasis and platform

Next week,

  • Samuel, Alex: Work on chasis/platform, integrate Jetson
  • Oliver: Recipes and/or nutritiontal information, bug-test
Posted on

Oliver’s Status Report – 2 Apr 22

This week was an incredibly productive week for me because the primary blockers, such as the development environment and library issues have been tackled in the preceding weeks, leaving me free to implement the meat of the back-end swiftly. I delivered multiple API endpoints and worked closely with Alex and Samuel to integrate the API with their front-end and CV system respectively, producing a cohesive product very rapidly.

One thing I also achieved this week is that I actually refactored the database schema in order to better work with Samuel’s idea of sending a list of items the CV system has predicted, in descending order of probability, and presenting it to the user. The previous schema did consider such a use case but was not well designed for it, and did not store the CV’s predictions in the database at all. By swiftly refactoring (which was made much easier by the Prisma ORM and TypeScript!) I managed to deliver the results we wanted for demo week and beyond.

Going by my personal Gantt Chart, we are now exactly 1 day ahead of schedule. It’s a fantastic sign, and I believe that with this pace, we can work even faster. In the following weeks, I have identified several improvements my API could perform, such as coalescing multiple transactions within a single day, as well as fixing some security vulnerabilities.

Posted on

Samuel’s Status Report – 2 Apr 22

This week was an extremely productive week, and I was able to get a fully working CV system up and running, with various features such as motion detection, stability checks and sending the top 5 predictions to the backend via the API.

FSM – Motion Detection/Stability Check

Last week, I was able to get basic background subtraction/motion detection working that compares the current frame with the previous one to check for changes. This week, I created a simple FSM which would use this motion detection algorithm to check for changes that indicates a user wants to predict an image, wait for stability, and predict on that stable image. Then, when the user removes the fruit, the algorithm similarly detects that change, waits for stability, and returns to a “background state”.

This FSM allows for a stable and robust detection of an item, as opposed to the initial algorithm that performed predictions on a potentially-blurry “delta image”.

 

Fig 1: Prediction after stability

Fig 2: Background State after stability

Fig 3: Ready for next prediction

API Integration

After Oliver was able to get a simple API working, it was time to integrate the CV system with the backend. For this, I used the curlpp library to send a HTTP request in the form of a JSON array of item IDs to the server/backend after a successful prediction.

Normalization

I also attempted to experiment with different normalization techniques (and re-trained the network accordingly), including ImageNet’s original normalization statistics, to improve the robustness of the algorithm against different lighting conditions. However, for some reason, the normalized models always worked worse compared to the non-normalized ones, and therefore I am currently sticking to the unnormalized model that seems surprisingly robust. It has worked amazingly well on different backgrounds, lighting conditions and environments (I’ve tested using monitors and chairs as platforms, at home and on different parts of campus, and it works well in all cases with few problems).

Demo + Future Work

Moreover, we were able to get our algorithm interfaced and working with the front-end and back-end, in preparation for our demo next week. Now that we have a fully working and robust scanning system, we  are currently very much on schedule, and in fact slightly ahead. My next step is to work on

  1. Integrating with the Jetson hardware
  2. Collecting my own datasets
  3. Seeing if there are any bugs with my normalization algorithm that affected its accuracy (because normalization should fix problems not cause them)
Posted on

Team’s Status Report – 2 Apr 22

This week we made much progress as a group in preparation for demo day!

We were all individually able to get all our individual components working AND integrated together:

Individual Components

  • Samuel: Completed CV scanning system, allowing for accurate detection of scan/removal and prediction on stable image
  • Alex: Trained NN and got websockets coded and working/integrated with the API.
  • Oliver: Made a lot of progress this week. Almost fully integrated all available backend API endpoints with the front-end as well as the CV system. Product works end-to-end as a result. Also implemented “live-streaming” of CV predictions to all front-ends

Overall workflow

  1. User places fruit under camera in CV system
  2. CV system detects movement and waits for stable image
  3. CV system predicts on image and sends the top 5 predictions to backend via JSON API
  4. Back-end stores the CV predictions and their relative ranks, and emits a “newTransaction” event to client front-ends.
  5. Front-end receives event from the API and presents a prompt for the user to confirm the CV’s predictions and committing the transaction into the back-end database.

 

We are happy with our current progress / product right now, as everyone in the team worked hard on their individual parts, and were also able to integrate everything together smoothly.

With a working integrated MVP, our team is on track (in fact slightly ahead of schedule) to deliver all that was promised in earlier stages, and it is even possible that we can achieve some of our stretch goals, such as a door sensor to tell whether it is an item addition or removal.

The immediate tasks at hand (in order of priority) are:

  1. Item removal – System currently only supports item addition, but needs to have an interface to support removal. Should be completed quite fast.
  2. Hardware integration – system currently works off a laptop, but want it to work on our Jetson, which currently needs its motherboard fixed. Once the new one arrives, integration should be pretty fast.
  3. Chasis/Platform – Currently propping up the camera on white chairs or monitors facing a white table/table with piece of paper, but eventually need to have it mounted on a proper chasis and platform. However, the algorithm still works despite different environmental conditions, indicating its robustness!
  4. Data collection  – currently we have 14 classes of fruits that the algorithm can robustly detect (including apples, oranges, tomatoes, bananas, pears), but some of these fruits are not commonly bought (eg. pomegranate, kiwi, starfruit).  Would need to collect some data of our own for vegetables like carrots, bell peppers, broccoli, cauliflower etc
  5. Recipes – Was originally a stretch goal, but now something we can possibly do!
Posted on

Alex’s Status Report – 2 Apr 22

This past week, I worked on integrating scans with the API. We got websockets and a few extra We were able to complete the full pipeline of scan -> API -> popup on the frontend.

I also worked on getting the new neural network trained.

We are back on schedule and ready for the demo!