Category: Elena’s Status Reports

This week I worked on the final presentation and preparation for the demo video. To be specific, I worked on the slides that we will be using to showcase our project in the demo video.

The other thing that I worked on is the finalizing of the recommendation part’s recipe dataset. For the last two weeks, I was more focused on the testing part, so that I did not finish the setup for the 30 recipes that we planned. Therefore, this week I added the last few recipes to our dataset. Now, we got more variety in recipes and types of cuisine.

The things that I am working on through the weekend with my teammates are filming and editing the video. We already discussed what sections we would like to have in the video, and we are working separately on each section at this moment. We will also be working on the posters together.

We are on our schedule and we pushed to the end of the semester. Next week I will be working on the final report with my teammates.

This week I worked on tests on the ingredient recognition for image recognition part, and the tests for the recommendation system. For the recommendation system, I built the test queries, tested and collected the accuracy, and add error handlers to the functions. For the ingredient recognition, we had two functions before, one of them is using the “label detection” attribute from API response, another is the “object localization” attribute. We used the second one to get the information of ingredients’ location and calculate the corresponding grid index. During the image tests, I noticed that the object localization response can recognize all ingredients with the correct amount, but sometimes can only label the ingredient as “food”. In order to get a more accurate name of ingredients for our project, I am trying to combine the information we can get from both response attributes to raise the accuracy. I am looking forward to finishing the improvement by this weekend for the presentation.

Another part I worked together with my teammates is the presentation slides. As the presenter, i updated our schedule chart, and participated in the creation of the slides, including parts other than the testing.

I think we are on the track. This week we are integrating our system, and after completing the testing we are generally finishing up the project. Next week I will be finalizing the database for recipes and support my teammates with the integration testing and general improvement.

This week I continued working on adding more recipes into our recommendation system. Similar to the previous works, I am searching recipes online and process them by tagging and cleaning data into the designated format.

Another part of this week’s work is designing the test data set for the recommendation part. In general, I had made two ways to test if the recommendation system is working correctly; the first part is a set of requests list as the required format, and this set of requests will be used to manually check if the recommended recipes are correct(only uses the ingredients detected, and satisfy special requirements). Another part of testing will be offering some invalid requests, like missing ingredients, malformed requests, etc, and make sure that the system will not crash and can handle the issues.

In the past weeks, we once tested the image recognition API with the dataset we built. However, we didn’t take the accuracy but only observed that the API works on our test set. Therefore, this week I also started rerunning the API on the dataset in order to get the numeric accuracy. In addition, we have a set of photos we took with real ingredients, which will also be applied to test the API in different situations.

Currently, I am on the schedule; the recommendation part only needs the progress in the number of recipes. For next week’s plan, we will be working on the final presentation, and I will put the most effort into the presentation as the presenter.

For this week I first tested the ingredient detection accuracy on our own image datasets. Among about 30 candidate items chosen before, about 12 items were detected reliably by the API. I did some research about the ingredients that were not detected successfully, some of them were probably not detected due to the overlapping of individual pieces, like raw/cooked shrimps; others may require a perfect angle/shape for correct detection, like pineapples and eggplants. In general, we are able to choose 9 out of 12 items, including beef, broccoli, strawberry, banana, Italian sausage, apple, tomato, onion, carrot, octopus, potato, salmon. It is surprising that the API was able to detect beef and salmon, which are usually sold in small pieces/cuts.

As the next step after deciding the items, I started working on the 30 recipes based on these selected items. For each recipe, I have to manually add tags to them. Currently, the tags that we have include vegan, non-dairy, dessert, dinner, lunch. Also, as suggested by the instructor previously, I will collect a list of the pantry items that are used in these recipes since we assume that users will have plenty of supply for these items.

My progress is still a bit behind due to my personal health issues. Thanks to the support from my teammates, they help taking over the image recognition work and that really helps me catch up with the plan on the image side; on the recommendation side, I plan to finish 10 recipes by this weekend, so that we will be able to integrate the entire system next week. Therefore, the plan for next week will be the teamwork on integrating the entire system.

This week I planned to work on the connections between sub-systems and figure out sending recipes through emails. Unfortunately, I had a fever during the week, and due to the sudden illness, I did not finish everything that I had in my plan.

First, I implemented and eventually tested out the emailing functionality. I am using SMTP in Python to send emails through my Gmail account. My only concern is that both the email address and the password are required for the sender to send the email, which might cause security issues to the sender. Currently, I am putting this information in config.ini and not sharing it when pushing my codes to GitHub/public repository. A screenshot of a successful email sent through SMTP is attached.

For the second part, regarding the connections between sub-systems, I did some research about the projects/previous works including speech and even conversation functionality. For example, I found https://developer.nvidia.com/conversational-ai helped me understand the general pipeline of conversation AI, and https://toptechboy.com/ai-on-the-jetson-nano-lesson-60-make-your-nano-talk-with-text-to-speech/ as a video lesson for better self-practice.

Because of personal health issues, even though I plan to run my own code on Jetson Nano to test a general pipeline, I did not finish it this week. This will make me a bit behind the schedule. In order to catch up with the schedule, I planned to make good use of the time next week, and the week after(spring carnival). I will set a stricter schedule and take care of health issue at the same time so that the happenings will not repeat. For the next week’s deliverables, I plan to add more functionalities to my recommendation part(timestamp), and set up the data processing of the image recognition part.

This week I focused on my work for the recommendation part as well as supporting tests for image recognition. I collected a set of images of the candidate items posted in the previous status report, and my teammate will use the image data set to test the performance of Google Vision on these candidate items.  Some example images are posted below.

For the recommendation part, I built two functionalities. The first function will accept an input including two keys, tags and ingredients. It will output a list of recipe names for users to select from. Some sample input and output is as follows:

Input:

test_request = {‘tag’: [‘non-dairy’];’ingredient’: [‘apple’, ‘orange’,’tuna’]}

Output:

[Apple and orange jam’]

If no recipe can be matched to the request, then it will return the following output:

[‘No results found’]

All samples above are the real inputs and outputs of the system.

Also, after the user selects the recipe they want to use, the next function will accept input of recipe index, and return a list of instructions. A sample output will be the following:

[‘Cut 2 apples and 2 oranges into small chunks’, ‘boil 200ml of water in a pot’,
‘add apples, oranges and 50g of sugar into the pot, and stew with medium heat’, ‘when the content becomes thick, turn off the heat and add 2 table spoons of lemon juice’]

The progress is on schedule. I will use the performance result from Google Vision to decide the 9 items and 30 recipes next week and generate more test cases to test on the recipe recommendations. Also, I am currently working on the email sending recipes to user functionality. There are some issues with the authentication, but I will try to figure it out next week.

 

 

 

Personally, I focused on the work of the design review report.  After reviewing more related works and analyzing the ingredients that might result in high recognition accuracy and lower false cases, I built a table of ingredient candidates that we might count as the final 10 items that we will be recognizing and relating recipes to. The table is attached in the post.

In addition, I learned about the current popular recommendation systems and how to build them from scratch. I followed these tutorials: https://www.toptal.com/algorithms/predicting-likes-inside-a-simple-recommendation-engine and https://www.analyticsvidhya.com/blog/2018/06/comprehensive-guide-recommendation-engine-python/ to help me design the algorithms for a small scale recommendation system.

I am in the right place of the schedule we planned and not behind. For the deliverables for next week, we will first finish up the report together, and next personally I will be focusing on grabbing the recipes related to the candidates listed in the table and finalize our 10 items on the list and 30 recipes as well.

One of the most significant risks that could harm the success of the project is that we may have to work separately in order to make sure that different tasks can be tackled at the same pace. We will manage the risk by ordering two pieces of the same hardware parts so that we will be able to do our own work at the same time. The hardware parts will not be wasted because there is the possibility that we will add a second layer to our fridge so that we will use two hardware pieces.

Another risk that we currently met is that even though we order all parts using the form last Sunday and expected that the orders would have been placed on Tuesday, we did not know that we needed to inform TAs in order for the orders to be confirmed. Thus, we didn’t get the orders placed until Friday, which may influence our schedule. Fortunately, our models and codes will not be run on jetson Nano until we tested them on PC, so the delay in these parts will not be a big problem. For the acrylic boards and the LED grids, we will try to work on the building work together to shorten the time we need for this part of the work.

In general, we do not have any changes to our proposed schedule. This week we worked on the design review presentation together, and we tried to specify the implementations of the software part and support our ideas with related works and details. We also decided on the bill of materials and placed the orders. For the next week, we will work on the design review report and start working on our own tasks as planned.

 

This week I worked with my teammates on the design review presentation. Personally, I first worked on the schematic diagram. I simulated the ideal product in the real-world application by renovating my own fridge, taking a photo of this real fridge, and adding components to the photo(below shows the photo in the process; the final pic exceeds the 8Mb limit and will be displayed in presentation). In addition, I also sketched a diagram of how our final product for proof of concept will look like.

For the recommendation system, I read many related works available online and some popular implementation of recommendation system training networks. From all articles and works, I found a paper that focused on a cooking recipe recommendation system that runs on a consumer smartphone. https://online-journals.org/index.php/i-jim/article/view/3623/3116

I compared their applications of bag of features with my previous related projects to help me build my own training models.

I think we are on the schedule planned before. For the next week, we will first focus on finishing the design review report; if I have extra time, I will start building the baseline model by first implementing the feature extraction and test on the accuracy.

 

This week I looked into the Google Cloud Vision API that we planned to use for the image recognition part. By testing on the API with different images of perishables, I believe that the API will be reliable on photos that we take for real food, as it provides high accuracy on the type of food and precision on the amount. Some example results of oranges are the following.

I think the progress is on schedule, and for the next week, I am planning to research more about the past works on recipe recommendation systems or other recommendation systems based on few features(restriction on recipes, number of people).  Also, I will work with my team on the design review presentation.