Carnegie Mellon ECE Capstone, Spring 2021 – Yang Yue, Nanxi Li, Elena Gong
This week I helped with the final integration of code, now our prototype can run seamlessly for demo. We are wrapping up for our project at this stage and we are preparing for the final demo and poster session. I was in charge of recording the demo part of the video and a part of the technical details. Everything is good and on schedule. We will have amazing final video soon!
This week I worked with my team to make the final video for my demo on Monday and also preparing for the poster. We also reviewed and prepped the final presentation this week.
Next week, we plan on editing and finalizing the video and poster, and practice before the poster session. We will focus on wrapping up this upcoming week.
This week our team finished the setup of the system in the mini-fridge and tested the integrated system. The most significant risk that could jeopardize the success of the project is how to showcase the entire functionality in the demo video. Since the live demo is only one part of the entire demo video, and our project plays the role in the daily routine of the user, we need to find a proper expression that demonstrates our product in few minutes. This risk is managed by having good communication in our team meeting to design a plan with specific steps and content we want to add to the video. We will add annotations to the live demo part so that we can make sure each step is explained to the audience.
Since it is already the last week of class, and we finished the implementation part of our project, we are not making changes to the existing design, and we don’t change the schedule as the things left are to work on the report, demo, and poster.
An image of the mini-fridge is attached.
This week I worked on the final presentation and preparation for the demo video. To be specific, I worked on the slides that we will be using to showcase our project in the demo video.
The other thing that I worked on is the finalizing of the recommendation part’s recipe dataset. For the last two weeks, I was more focused on the testing part, so that I did not finish the setup for the 30 recipes that we planned. Therefore, this week I added the last few recipes to our dataset. Now, we got more variety in recipes and types of cuisine.
The things that I am working on through the weekend with my teammates are filming and editing the video. We already discussed what sections we would like to have in the video, and we are working separately on each section at this moment. We will also be working on the posters together.
We are on our schedule and we pushed to the end of the semester. Next week I will be working on the final report with my teammates.
This week, I collected user data for our speech recognition system and tested the collected data in our system for both latency and also accuracy. We will use this data for our presentation next week, and this will be useful for showing the performance of our system.
Additionally, I fixed some issues with our image capture on the jetson nano, so that now we can correctly live capture images and process them for our system. Along with that, I worked on setting up our physical demo system (see included photo) with a minifridge and LEDs for the presentation.
In addition to this, I am working on adding to our presentation slides for Monday. This week we are finishing integrating our system and completing testing. We are on track, and should have a ready presentation for Monday. 
This week I finalized the design for LED grid and worked on hardware integration. The LED strips only work with 2 of the pins on Jetson Nano (SPI MISO, apparently not all GPIO pins are equivalent), so I ended up soldering a LED grid from individual LEDs from scratch. It was a lot of designing, wiring and soldering involved. (There are 2 grids lighting up in the photo below.)
I moved our design into a mini fridge for final demo and spend a lot more time soldering and wiring. The camera and microphone is mounted. The code is also ready to be integrated.
I will do some integration test tomorrow, and help Elena prepare the final presentation slides.
The most significant risk that could jeopardize the success of the project is the integration of the system in the mini-fridge. At the beginning of the semester and for the most time of the semester, we are using the clear acrylic board as an imitation of one shelf in the fridge. But we noticed that it might be better to just set up our system in the mini-fridge for a good demo result. Therefore, our team is currently focusing on building up the system in the mini-fridge. Since we set up the camera separately to take the picture of the grid, we have to figure out how to do this in the fridge. The risks are managed by mounting/ taping the camera on the wall of the fridge; we might need to adjust for the best angle and height, but we will try to fix the camera at one point for the best image results.
There are no changes made to the existing design of the system. Since we are coming to the end of the semester, we believe that sticking to our current design and implementation is the best idea.
There are no significant changes to the current schedule. We updated the demo and report timeline according to the course schedule.
This week I worked on tests on the ingredient recognition for image recognition part, and the tests for the recommendation system. For the recommendation system, I built the test queries, tested and collected the accuracy, and add error handlers to the functions. For the ingredient recognition, we had two functions before, one of them is using the “label detection” attribute from API response, another is the “object localization” attribute. We used the second one to get the information of ingredients’ location and calculate the corresponding grid index. During the image tests, I noticed that the object localization response can recognize all ingredients with the correct amount, but sometimes can only label the ingredient as “food”. In order to get a more accurate name of ingredients for our project, I am trying to combine the information we can get from both response attributes to raise the accuracy. I am looking forward to finishing the improvement by this weekend for the presentation.
Another part I worked together with my teammates is the presentation slides. As the presenter, i updated our schedule chart, and participated in the creation of the slides, including parts other than the testing.
I think we are on the track. This week we are integrating our system, and after completing the testing we are generally finishing up the project. Next week I will be finalizing the database for recipes and support my teammates with the integration testing and general improvement.
This week I am done with the building and testing of the LED grid. It is working as expected. I might make some final adjustment to the sizing of the grids before taping it, but it is mostly done. The adapter we got for LED is not so stable, so I prefer not to light it up for too long at a time so that the LED strip doesn’t get burned like the last time. This also fits the usage model better. (Users will pick up the food from the grids within a minute.)
I have been testing the image recognition part regarding the placement of each object. This part of the algorithm decides which LED grid each item belongs to. I found out that the accuracy rate of this when using google Vision API is highly dependent on the placement of the camera and the lighting. I did some testing on where to mount the camera to get the best result. Here are the results below. I decide to mount the camera at 18 inch above the surface. We will be using this data for a part of the testing plan as well. More testing about the accuracy of this will be added to the testing plan later.
Another problem is that lighting does affect the image recognition accuracy. We will put a light source (a floor lamp) right above the objects to imitate the lighting in the fridge. In this case, the accuracy rate of image recognition should be stable and it will be better for the testing.
I plan on doing the integration test next week and we should be able to have our final product complete.
This week, I looked at collecting data for our presentation next week. Primarily focused on how our system works and interacts with users. To do this, we are doing surveys to gather audio data and testing how our system works with that input. This will be very important for our presentation on testing and validation. Overall, we are on track and will only need to make some minor changes to our code + validation to have a ready presentation. 