Category: Nanxi’s Status Reports

This week I helped with the final integration of code, now our prototype can run seamlessly for demo. We are wrapping up for our project at this stage and we are preparing for the final demo and poster session. I was in charge of recording the demo part of the video and a part of the technical details. Everything is good and on schedule. We will have amazing final video soon!

This week I finalized the design for LED grid and worked on hardware integration. The LED strips only work with 2 of the pins on Jetson Nano (SPI MISO, apparently not all GPIO pins are equivalent), so I ended up soldering a LED grid from individual LEDs from scratch. It was a lot of designing, wiring and soldering involved.  (There are 2 grids lighting up in the photo below.)

I moved our design into a mini fridge for final demo and spend a lot more time soldering and wiring. The camera and microphone is mounted. The code is also ready to be integrated.

I will do some integration test tomorrow, and help Elena prepare the final presentation slides.

This week I am done with the building and testing of the LED grid. It is working as expected. I might make some final adjustment to the sizing of the grids before taping it, but it is mostly done. The adapter we got for LED is not so stable, so I prefer not to light it up for too long at a time so that the LED strip doesn’t get burned like the last time. This also fits the usage model better. (Users will pick up the food from the grids within a minute.)

I have been testing the image recognition part regarding the placement of each object. This part of the algorithm decides which LED grid each item belongs to. I found out that the accuracy rate of this when using google Vision API is highly dependent on the placement of the camera and the lighting. I did some testing on where to mount the camera to get the best result. Here are the results below. I decide to mount the camera at 18 inch above the surface. We will be using this data for a part of the testing plan as well. More testing about the accuracy of this will be added to the testing plan later.

Another problem is that lighting does affect the image recognition accuracy. We will put a light source (a floor lamp) right above the objects to imitate the lighting in the fridge. In this case, the accuracy rate of image recognition should be stable and it will be better for the testing.

I plan on doing the integration test next week and we should be able to have our final product complete.

This week I finished the program for LED, now with the input from the image recognition software, we can control the LEDs and light up the grids we need. The code has been pushed to GitHub. Unfortunately, I burned our LED strip during the testing process and need to order new ones. The new ones will not arrive before demo, so I prepared a demo with single LEDs. (For simple turn on and offs, the LED strip can controlled in the same way as simple LEDs.)

I also took a look at the image recognition portion. By taking a picture with several food items in it, the software can recognize what items are they and where they are (the API is very precise). This information can help us locate which grids the items are in and determine which ones to light up.

I ran into some expected problem this week. Our Jetson Nano stopped working, and we ended up re-flashing it. The second problem is that the available library that is compatible with the LED strips are mostly integrated with RPi’s GPIO library, which makes integrating the LED library on Jetson Nano very hard. I ended up deciding to write our own LED library for Jetson Nano GPIO. This combined with the issue with our Jetson Nano itself, we need more time for the LED grid. This week’s progress is that we have a robust testing scheme for GPIO output, it will be easier to debug the LED algorithm later. If the LED strip does not work out, we will fall back to use individual LEDs to build the grid ourselves.

Single LED for testing

Last week I spent up till Wednesday composing design review with my teammates. It took a lot of time, since we got some feedback from the instructor for our presentation and we changed some of the contents for the report. We got the power supply for Jetson Nano last week, I flashed the sd card and prepared it for use.

This week on Friday we finally got the network adapter, but we are experiencing some difficulty for installing the driver. I am using ethernet cable for now, but need to figure out the adapter situation next week. I tested the GPIO library we wanted to use for LED strip, and a demo application is working. I will spend tomorrow and Monday and testing and debugging our own application.

 

Our hardware arrived this week. I tested that all the peripherals (camera, microphone, speaker) work as expected with the library we want to use. We have not received the power supply we checked out, so we cannot test Jetson Nano. I will follow up on that. The acrylic board and peripheral holder is the same dimension as we expected. With the acrylic board, I have started designing the LED grid. I decided to make them serial while being individually addressable. This will require more logic in our light-up-LED algorithm, but less hardware connection. This implementation only requires 1 GPIO pin. I plan on finishing it and testing it next week. I am on schedule for the project. （Picture: layout of LED strip, will securely it on acrylic board later)

I also worked on design review this week. I prepared the presentation slide and practiced for the presentation. We also started working on the design review report.

This week I worked on the material list of our project and everything is ordered. I refined the design of our project after reading the feedback for our proposal. I designed the recipe delivery method to be step-by-step instruction with the option to skip or repeat. Specific user interface and implementation details have been decided. I also took the chance of preparing my presentation to review our design and specify requirements on a finer level. (Please refer to our slide for my effort.)

We are on schedule for the project. Next week our hardware will arrive and I want to test them with the external library we will use. I will also need to start designing and making the LED grid.

I searched for the hardwares we need. Since we are more likely to use Jetson Nano, I made a list of corresponding hardwares to purchase and their cost. This can be easily changed to target RPi since most of the hardware are compatible. With Amazon, we can receive most of our hardware in 3 days of order. Additionally, I looked into the LED grids that we need. I did not find LED grids that fits our requirements. We need to build the grid from LED strips by ourselves.

Additionally, I watched the demo of many object recognition project targeting Jetson Nano and read the user manual. I think we have a good grasp on what it could do for us.

 

I collaborated with Yang and Elena to further define the scope of the project and the overview of our design while putting together the presentation slides for our  proposal presentation. I worked on the design overview flowchart, and specified our finer design modules under the entire project. The flowchart is in the presentation slide. For the slides, additionally, I was in charge of pages for Use case, Requirements: Accuracy of Recognition Systems and Quality of Recipe Recommendations and Design overview. (Please refer to our slides to see how much effort I put in.)

Our progress is on schedule. For next week, I hope to start getting our hardware peripherals ready and testing out the external library we will use.