Category: status report (team)

Most of this week was spent preparing for the final presentation, demo and posters, and A LOT of testing. We also managed to install the fully integrated system on an actual fridge!

Since most of the individual work has already been completed, most of our time this week was spent testing our integrated system together and ironing out bugs with the CV, front-end and back-end side. Besides this, we also worked on the final presentation and poster. Both the presentation and the poster seems to have turned out well, and we’re really excited for the final demo and to see how the project fares overall!

Next week, we will continue to work on the final video and poster. However, testing has also revealed some improvements and features that we could add, both on the frontend side (related to UX design like button sizes/placement), and the backend side (email notifications). That being said, we are ahead of schedule (now making reach goals and extra features, i.e. non-essential), and are really excited for our final demo and video!!!

We made quite a bit of progress this week with our project, and are actually mostly prepared for a demo already 😀 We will focus our efforts on simple incremental changes + lots of testing.

We have thus far:

1. 1. Integrated the entire system with our tablet + Jetson
   2. Completed a benchmark accuracy and speed test
   3. Did extensive full-system bug-testing and fixed some bugs in our back-end code with returns and made the system more robust against incorrect quantity data (e.g. removing more fruits than there are in the fridge)
      
   4. Completed training of self-collected classes, including bell peppers, and are potentially looking at adding more classes.
   5. Debugged and fixed classification speed issues on the Jetson
   6. Spray-painted the background white to improve algorithm accuracy
   7. CV algorithm is now more robust against “false triggers”
   8. Nutritional information now available to user

That being said, we have found out some interesting findings and possible improvements that we can make to our project that will make our product even more refined:

Adding email or in-app notifications to remind the user that their food is expiring soon (backend + frontend)
1. Quantity detection using simple white background thresholding and GrabCut (CV, backend interface already in place)
2. Different fruit type selection (see Figure above) (frontend + CV)
3. Smart recipe detection
4. EVEN MORE TESTING TESTING TESTING.

We are currently comfortably ahead of schedule but will keep pressing on in refining our final product 🙂

This week we spent most of our time building the actual platform, bug testing and making minor incremental improvements on the system.

 

Figure 1: Example of Fully-Constructed Platform

Alex: Built the platform and installed camera onto it. Readying Jetson for integration. Started work on nutritional information

Sam: Helped Alex with building of platform. Tested scanning + collected data on actual platform. Working on various normalization algorithms to improve illumination robustness.

Oliver: Conducted more extensive bug testing and fixed a number of bugs in the backend. Made expiry date calculations generate more robust and user-friendly output (i.e. granularity on a daily level instead of on a microsecond level).

We are currently on schedule but need to speed up the integration process to leave more time for testing.

Next week, we will work on integrating the full system, by testing it with the Jetson and installing on a real fridge door. We are also in the midst of training with our self collected dataset and will try that next week.

This week, we were able to make our preliminary demo which went quite well. We were able to show the robustness of our CV algorithm and the full integration of our system (front and back end), through the successful scanning and adding of fruit to the database, and having the added fruit show up on the UI.

There were some slight hiccups during the actual presentation, in particular when we tried to show the remove feature that was added in a rush without much testing. In the actual demo/final presentation this will not happen, and there will probably be some sort of code freeze so that we will only demo robust and tested features.

Since we basically have an MVP complete already, we are slightly ahead of schedule, and have been and are currently working on making the chassis/hardware and adding on some key features.

Summary of some progress this week

• Samuel: Collecting more data for various fruits and vegetables
• Oliver: Fixed remove feature bugs
• Alex: UI touchups, cut wood for chasis and platform

Next week,

• Samuel, Alex: Work on chasis/platform, integrate Jetson
• Oliver: Recipes and/or nutritiontal information, bug-test

This week we made much progress as a group in preparation for demo day!

We were all individually able to get all our individual components working AND integrated together:

Individual Components

• Samuel: Completed CV scanning system, allowing for accurate detection of scan/removal and prediction on stable image
• Alex: Trained NN and got websockets coded and working/integrated with the API.
• Oliver: Made a lot of progress this week. Almost fully integrated all available backend API endpoints with the front-end as well as the CV system. Product works end-to-end as a result. Also implemented “live-streaming” of CV predictions to all front-ends

Overall workflow

1. User places fruit under camera in CV system
2. CV system detects movement and waits for stable image
3. CV system predicts on image and sends the top 5 predictions to backend via JSON API
4. Back-end stores the CV predictions and their relative ranks, and emits a “newTransaction” event to client front-ends.
5. Front-end receives event from the API and presents a prompt for the user to confirm the CV’s predictions and committing the transaction into the back-end database.

 

We are happy with our current progress / product right now, as everyone in the team worked hard on their individual parts, and were also able to integrate everything together smoothly.

With a working integrated MVP, our team is on track (in fact slightly ahead of schedule) to deliver all that was promised in earlier stages, and it is even possible that we can achieve some of our stretch goals, such as a door sensor to tell whether it is an item addition or removal.

The immediate tasks at hand (in order of priority) are:

1. Item removal – System currently only supports item addition, but needs to have an interface to support removal. Should be completed quite fast.
2. Hardware integration – system currently works off a laptop, but want it to work on our Jetson, which currently needs its motherboard fixed. Once the new one arrives, integration should be pretty fast.
3. Chasis/Platform – Currently propping up the camera on white chairs or monitors facing a white table/table with piece of paper, but eventually need to have it mounted on a proper chasis and platform. However, the algorithm still works despite different environmental conditions, indicating its robustness!
4. Data collection  – currently we have 14 classes of fruits that the algorithm can robustly detect (including apples, oranges, tomatoes, bananas, pears), but some of these fruits are not commonly bought (eg. pomegranate, kiwi, starfruit).  Would need to collect some data of our own for vegetables like carrots, bell peppers, broccoli, cauliflower etc
5. Recipes – Was originally a stretch goal, but now something we can possibly do!

We mostly worked individually on our own responsibilities for the project this week:

• Samuel: Fixed issues with neural network training, completed training on Fruits360 dataset with ResNet50 architecture. Currently working on finding better datasets to train on.
• Oliver: Brought API online onto a live server connected to the Internet, and continued to implement API endpoints needed as part of the plan. We now have enough API endpoints to begin the integration process meaningfully
• Alex: Worked with Samuel on researching new datasets and potential fixes for the dataset, as well as running the neural network training and fixing server issues.

 

For next week, we will also be continuing to work on our individual parts but will begin integration through our APIs, especially with the front end.

• Samuel: Will work on finding (or creating our own) and training network on new datasets. Will work on background segmentation (as needed for new datasets or white background).
• Oliver: Will work on integrating the back-end with the front-end created by Alex, so that Alex can be unblocked on that front and can continue work on the front-end. Will also continue with remaining back-end API endpoints
• Alex: Working with Oliver on integrating the front-end and back-end

This week, we primarily focused on our design review report, which was more work than expected. Thankfully, the deadline was extended, and most of the content was already thought through or covered in the presentation; we just needed to spend time writing in out. In particular, we decided to make our block diagram a lot more detailed:

Old Block Diagram

Our old block diagram used in the slides was primarily meant as a summary for visual purposes, and therefore lacked the detail needed for the report.

New Block Diagram

Our new block diagram is a lot more detailed, with specifics regarding algorithms, APIs and data transfer; however, this would have been too confusing for a presentation.

 

Most of our time this week was spent on the design report, and not much was done on the implementation side. However, we are still quite comfortably ahead of schedule since we began implementation early. With regards to the design report, we split up the roles equally, with each team member taking care of the architecture and implementation components related to their specialization:

• Samuel:
  • Architecture, Implementation, Testing (CV + Attachment System)
  • Introduction, Use-Case
  • Trade studies (CV-related)
  • Related work
• Alex:
  • Architecture, Implementation (Front-end/UI)
  • Use-Case, Design requirements
  • Trade Studies (misc)
  • Risk-mitigation
• Oliver:
  • Architecture, Implementation, Testing (Databases/Back-End, APIs,)
  • Project management (Schedule, Responsibilities, Materials)
  • Risk-mitigation
  • Summary

This week, we completed our design review presentation, which we think went quite well.  Our main focus for next week will be the design review report, due Wednesday next week. After Wednesday, we will continue working individually on our various responsibilities to implement the CV, UI and back-end systems  for Samuel, Alex and Oliver respectively.

Currently, we have made good progress on the implementation side, and are slightly ahead of schedule in this sense (see our individual reports for more information):

• Samuel: Completed C++ testing for CV. Discovered that the CNN network found from Medium does not work well and has major flaws with the customized architecture. Will begin work next week training and testing a new ResNet18 or AlexNet model.
• Oliver: Enforced a rigorous common standard in the back-end code-base by integrating automatic linting, type-checking, and even bug catching tools. Brought the code-base to strict, 100%, type-safe standards, setting up the back-end for seamless and co-operative development regardless of each team member’s individual style, and ensuring that code pushed meets high levels of rigor. Will deliver a core set of APIs next week built upon this level of rigor for integration with front-end
• Alex: Completed most of basic UI. Will start collaborating with Oliver once API side is complete. Helped Samuel with the training of the classification algorithm

This week, we worked as a team primarily on finalizing and documenting our design as part of the design review presentation and report. In particular, we finalized our overall architecture and the way in which everything will integrate together. Notably, our original design of a front facing camera with a vertical platform got changed to one where the camera faces downward on a flat white surface. This was to facilitate a more intuitive and non-intrusive scanning process.

Architectural Overview:

> Architecture (Original Handwritten)

The following is a summary of our individual work (for more details, visit the individual status reports):

• Samuel was able to write the code needed to train the neural network classifier (we now have a pretrained model with 98% accuracy), and is working on a C++ application that can use it to classify an image.
• Oliver began work on the backend, and defined the database schema and column relations in Prisma. He is now working on implementing the API endpoints defined in the architecture.
• Alex was able to complete a significant chunk of the front-end design, and get it running at https://capstone.astrasser.com:2096/

Next week, we will continue working independently on our individual portions of developing the CV, UI and backends (more details in the individual reports). As a group we will also start building our fake fridge and scanning platform