Team D3 SmolKat: A Smart Kitchen Assistant
Carnegie Mellon ECE Capstone, Spring 2021 – Yang Yue, Nanxi Li, Elena Gong
This week I helped with the final integration of code, now our prototype can run seamlessly for demo. We are wrapping up for our project at this stage and we are preparing for the final demo and poster session. I was in charge of recording the demo part of the video and a part of the technical details. Everything is good and on schedule. We will have amazing final video soon!
This week I worked with my team to make the final video for my demo on Monday and also preparing for the poster. We also reviewed and prepped the final presentation this week.
Next week, we plan on editing and finalizing the video and poster, and practice before the poster session. We will focus on wrapping up this upcoming week.
This week our team finished the setup of the system in the mini-fridge and tested the integrated system. The most significant risk that could jeopardize the success of the project is how to showcase the entire functionality in the demo video. Since the live demo is only one part of the entire demo video, and our project plays the role in the daily routine of the user, we need to find a proper expression that demonstrates our product in few minutes. This risk is managed by having good communication in our team meeting to design a plan with specific steps and content we want to add to the video. We will add annotations to the live demo part so that we can make sure each step is explained to the audience.
Since it is already the last week of class, and we finished the implementation part of our project, we are not making changes to the existing design, and we don’t change the schedule as the things left are to work on the report, demo, and poster.
An image of the mini-fridge is attached.
This week I worked on the final presentation and preparation for the demo video. To be specific, I worked on the slides that we will be using to showcase our project in the demo video.
The other thing that I worked on is the finalizing of the recommendation part’s recipe dataset. For the last two weeks, I was more focused on the testing part, so that I did not finish the setup for the 30 recipes that we planned. Therefore, this week I added the last few recipes to our dataset. Now, we got more variety in recipes and types of cuisine.
The things that I am working on through the weekend with my teammates are filming and editing the video. We already discussed what sections we would like to have in the video, and we are working separately on each section at this moment. We will also be working on the posters together.
We are on our schedule and we pushed to the end of the semester. Next week I will be working on the final report with my teammates.
This week, I collected user data for our speech recognition system and tested the collected data in our system for both latency and also accuracy. We will use this data for our presentation next week, and this will be useful for showing the performance of our system.
Additionally, I fixed some issues with our image capture on the jetson nano, so that now we can correctly live capture images and process them for our system. Along with that, I worked on setting up our physical demo system (see included photo) with a minifridge and LEDs for the presentation.
In addition to this, I am working on adding to our presentation slides for Monday. This week we are finishing integrating our system and completing testing. We are on track, and should have a ready presentation for Monday. 
This week I finalized the design for LED grid and worked on hardware integration. The LED strips only work with 2 of the pins on Jetson Nano (SPI MISO, apparently not all GPIO pins are equivalent), so I ended up soldering a LED grid from individual LEDs from scratch. It was a lot of designing, wiring and soldering involved. (There are 2 grids lighting up in the photo below.)
I moved our design into a mini fridge for final demo and spend a lot more time soldering and wiring. The camera and microphone is mounted. The code is also ready to be integrated.
I will do some integration test tomorrow, and help Elena prepare the final presentation slides.
The most significant risk that could jeopardize the success of the project is the integration of the system in the mini-fridge. At the beginning of the semester and for the most time of the semester, we are using the clear acrylic board as an imitation of one shelf in the fridge. But we noticed that it might be better to just set up our system in the mini-fridge for a good demo result. Therefore, our team is currently focusing on building up the system in the mini-fridge. Since we set up the camera separately to take the picture of the grid, we have to figure out how to do this in the fridge. The risks are managed by mounting/ taping the camera on the wall of the fridge; we might need to adjust for the best angle and height, but we will try to fix the camera at one point for the best image results.
There are no changes made to the existing design of the system. Since we are coming to the end of the semester, we believe that sticking to our current design and implementation is the best idea.
There are no significant changes to the current schedule. We updated the demo and report timeline according to the course schedule.
This week I worked on tests on the ingredient recognition for image recognition part, and the tests for the recommendation system. For the recommendation system, I built the test queries, tested and collected the accuracy, and add error handlers to the functions. For the ingredient recognition, we had two functions before, one of them is using the “label detection” attribute from API response, another is the “object localization” attribute. We used the second one to get the information of ingredients’ location and calculate the corresponding grid index. During the image tests, I noticed that the object localization response can recognize all ingredients with the correct amount, but sometimes can only label the ingredient as “food”. In order to get a more accurate name of ingredients for our project, I am trying to combine the information we can get from both response attributes to raise the accuracy. I am looking forward to finishing the improvement by this weekend for the presentation.
Another part I worked together with my teammates is the presentation slides. As the presenter, i updated our schedule chart, and participated in the creation of the slides, including parts other than the testing.
I think we are on the track. This week we are integrating our system, and after completing the testing we are generally finishing up the project. Next week I will be finalizing the database for recipes and support my teammates with the integration testing and general improvement.