Author: yezheng

This week I worked on the final presentation and preparation for the demo video. To be specific, I worked on the slides that we will be using to showcase our project in the demo video.

The other thing that I worked on is the finalizing of the recommendation part’s recipe dataset. For the last two weeks, I was more focused on the testing part, so that I did not finish the setup for the 30 recipes that we planned. Therefore, this week I added the last few recipes to our dataset. Now, we got more variety in recipes and types of cuisine.

The things that I am working on through the weekend with my teammates are filming and editing the video. We already discussed what sections we would like to have in the video, and we are working separately on each section at this moment. We will also be working on the posters together.

We are on our schedule and we pushed to the end of the semester. Next week I will be working on the final report with my teammates.

This week I worked on tests on the ingredient recognition for image recognition part, and the tests for the recommendation system. For the recommendation system, I built the test queries, tested and collected the accuracy, and add error handlers to the functions. For the ingredient recognition, we had two functions before, one of them is using the “label detection” attribute from API response, another is the “object localization” attribute. We used the second one to get the information of ingredients’ location and calculate the corresponding grid index. During the image tests, I noticed that the object localization response can recognize all ingredients with the correct amount, but sometimes can only label the ingredient as “food”. In order to get a more accurate name of ingredients for our project, I am trying to combine the information we can get from both response attributes to raise the accuracy. I am looking forward to finishing the improvement by this weekend for the presentation.

Another part I worked together with my teammates is the presentation slides. As the presenter, i updated our schedule chart, and participated in the creation of the slides, including parts other than the testing.

I think we are on the track. This week we are integrating our system, and after completing the testing we are generally finishing up the project. Next week I will be finalizing the database for recipes and support my teammates with the integration testing and general improvement.

This week I continued working on adding more recipes into our recommendation system. Similar to the previous works, I am searching recipes online and process them by tagging and cleaning data into the designated format.

Another part of this week’s work is designing the test data set for the recommendation part. In general, I had made two ways to test if the recommendation system is working correctly; the first part is a set of requests list as the required format, and this set of requests will be used to manually check if the recommended recipes are correct(only uses the ingredients detected, and satisfy special requirements). Another part of testing will be offering some invalid requests, like missing ingredients, malformed requests, etc, and make sure that the system will not crash and can handle the issues.

In the past weeks, we once tested the image recognition API with the dataset we built. However, we didn’t take the accuracy but only observed that the API works on our test set. Therefore, this week I also started rerunning the API on the dataset in order to get the numeric accuracy. In addition, we have a set of photos we took with real ingredients, which will also be applied to test the API in different situations.

Currently, I am on the schedule; the recommendation part only needs the progress in the number of recipes. For next week’s plan, we will be working on the final presentation, and I will put the most effort into the presentation as the presenter.

For this week I first tested the ingredient detection accuracy on our own image datasets. Among about 30 candidate items chosen before, about 12 items were detected reliably by the API. I did some research about the ingredients that were not detected successfully, some of them were probably not detected due to the overlapping of individual pieces, like raw/cooked shrimps; others may require a perfect angle/shape for correct detection, like pineapples and eggplants. In general, we are able to choose 9 out of 12 items, including beef, broccoli, strawberry, banana, Italian sausage, apple, tomato, onion, carrot, octopus, potato, salmon. It is surprising that the API was able to detect beef and salmon, which are usually sold in small pieces/cuts.

As the next step after deciding the items, I started working on the 30 recipes based on these selected items. For each recipe, I have to manually add tags to them. Currently, the tags that we have include vegan, non-dairy, dessert, dinner, lunch. Also, as suggested by the instructor previously, I will collect a list of the pantry items that are used in these recipes since we assume that users will have plenty of supply for these items.

My progress is still a bit behind due to my personal health issues. Thanks to the support from my teammates, they help taking over the image recognition work and that really helps me catch up with the plan on the image side; on the recommendation side, I plan to finish 10 recipes by this weekend, so that we will be able to integrate the entire system next week. Therefore, the plan for next week will be the teamwork on integrating the entire system.

This week I planned to work on the connections between sub-systems and figure out sending recipes through emails. Unfortunately, I had a fever during the week, and due to the sudden illness, I did not finish everything that I had in my plan.

First, I implemented and eventually tested out the emailing functionality. I am using SMTP in Python to send emails through my Gmail account. My only concern is that both the email address and the password are required for the sender to send the email, which might cause security issues to the sender. Currently, I am putting this information in config.ini and not sharing it when pushing my codes to GitHub/public repository. A screenshot of a successful email sent through SMTP is attached.

For the second part, regarding the connections between sub-systems, I did some research about the projects/previous works including speech and even conversation functionality. For example, I found https://developer.nvidia.com/conversational-ai helped me understand the general pipeline of conversation AI, and https://toptechboy.com/ai-on-the-jetson-nano-lesson-60-make-your-nano-talk-with-text-to-speech/ as a video lesson for better self-practice.

Because of personal health issues, even though I plan to run my own code on Jetson Nano to test a general pipeline, I did not finish it this week. This will make me a bit behind the schedule. In order to catch up with the schedule, I planned to make good use of the time next week, and the week after(spring carnival). I will set a stricter schedule and take care of health issue at the same time so that the happenings will not repeat. For the next week’s deliverables, I plan to add more functionalities to my recommendation part(timestamp), and set up the data processing of the image recognition part.

This week I focused on my work for the recommendation part as well as supporting tests for image recognition. I collected a set of images of the candidate items posted in the previous status report, and my teammate will use the image data set to test the performance of Google Vision on these candidate items.  Some example images are posted below.

For the recommendation part, I built two functionalities. The first function will accept an input including two keys, tags and ingredients. It will output a list of recipe names for users to select from. Some sample input and output is as follows:

Input:

test_request = {‘tag’: [‘non-dairy’];’ingredient’: [‘apple’, ‘orange’,’tuna’]}

Output:

[Apple and orange jam’]

If no recipe can be matched to the request, then it will return the following output:

[‘No results found’]

All samples above are the real inputs and outputs of the system.

Also, after the user selects the recipe they want to use, the next function will accept input of recipe index, and return a list of instructions. A sample output will be the following:

[‘Cut 2 apples and 2 oranges into small chunks’, ‘boil 200ml of water in a pot’,
‘add apples, oranges and 50g of sugar into the pot, and stew with medium heat’, ‘when the content becomes thick, turn off the heat and add 2 table spoons of lemon juice’]

The progress is on schedule. I will use the performance result from Google Vision to decide the 9 items and 30 recipes next week and generate more test cases to test on the recipe recommendations. Also, I am currently working on the email sending recipes to user functionality. There are some issues with the authentication, but I will try to figure it out next week.