Author: kcha

Team Status Report for 4/27/24

Team Status Report for 4/27/24

Risks: Server on router, but has no wifi. App needs wifi for chatgpt. (is this still a problem ?) Changes: Scanner case has a new angle so it points more downwards Progress: Got notifications to work Video feed on the app Finishing up all the 

Kelly’s Status Report for 4/27/24

Kelly’s Status Report for 4/27/24

Work Accomplished: Camera streaming works Scheduled notifications work Helped Sebastian with implementing database functions on flutter Conducted user experience surveys and collected data Progress: Almost done! (on track) Things to complete next week: Work with Sebastian and Michael to finish getting everything working together (scanner, 

Kelly’s Status Report for 4/20/24

Kelly’s Status Report for 4/20/24


Work Accomplished:

  • Custom/ Edit Input popup
    • Added option to edit the inventory items by tapping on the item card. Information automatically populates.
      • Changes to the quantity, picture, etc. will update instantly to view changes
      • Will also save to database, but showing the change instantly at first will reduce latency and require less calls to the database.
    • Added location dropdown menu to choose where the item is stored
      • If the location is sauces/spices, there will be no option for expiration date and quantity, since we will just assume an infinite amount

 

  • Images
    • Changed to String type to handle file paths, urls, and encoded images which will be stored in the database.
    • Used DALL-E 3 to generate initial images to go along with each AI generated recipe. This makes the loading time to generate around 20 seconds, which allows the cooking animation to play until the end
    • Encoded images from file or url using base64Encode to save to database. Decoded images from database to display.
      • Turned the current saved images (20 produce items we chose) that were links into encoded strings by writing to a .txt file, since there are too many characters to print, to save to the database as constants

  • Database functions set up
    • I created made “//TODO:” comments throughout my code ands et up functions where the database will need to be accessed, like deleting, saving, etc. so that integrating the database in the app will be easy to follow by Sebastian.

  • Replaced the upload image button to an edit button
    • Only shown when the recipe is saved
    • User can edit image, ingredients, or instructions of the recipe
      • Changes to the picture, etc. will update on the recipe cart instantly to view changes

 

  • Added more filter options in the recipe generating filter button
    • Only one time under Timing can be selected

  • Video streaming
    • Tried to receive via TCP server address and port and use zmq – didn’t work because one of the plugins would cause errors and there are very minimal documentation
    • Tried to receive via HTTP url – video feed shows when opening on browser, but not on flutter app because failed to load the video.
      • This way works to load a video url, but might not have the functionality to support real time streaming
      • Might be the way we implemented how the video is sent
      • When I searched it up, it might have to do with how it is http instead of https.
      • Might also be because the server address I am testing with is 127.0.0.1 (localhost) and the simulator/iPhone’s localhost would not be the same as the computer’s localhost, so it opens to nothing.
    • When looking up video streaming flutter apps for examples, people usually use external accounts to stream the video and use keys that the service generates
    • Tried to look up to see if there are other people trying to stream directly, but most questions are left unsolved. There is more support for Android and looks like RTSP is trickier for IOS

 


Additional Questions:

New tools/knowledge

  • Figma
  • Flutter
    • UI elements/ widgets
    • Flutter plugins
  • OpenAI integration for ChatGPT and DALL-E

Learning strategies

  • Look at documentation/ official examples
  • Look for online tutorial articles or YouTube videos of people making simple apps revolving around the thing I need to learn
  • Trial and error working on things on my own and see how it turns out

Progress:

  • Completed a few small stretch goals (editing item and recipes, AI generated images)
  • Still working on connecting the scanner video feed to the app – will look into a few different methods.
    • If it does not work, might consider getting a Raspberry Pi Screen to connect the cameras to and mount onto the refrigerator instead of have the video feed viewed on the app.
  • Will do user experience testing via survey of people using the app once the DB is connected so the app works as intended.

Things to complete next week:

  • Connect scanner video feed to the app
  • Verification tests
  • Final presentation slides

 

Team Status Report for 4/6/24

Team Status Report for 4/6/24

Risks: 2GB Jetson Nano lost support in 2023, and so some of the software is fairly out of date (For example, Python 3.6 was the latest version available to be installed through apt, however Python 3.10 is the minimum version for YOLOv8) Changes: Switched back 

Kelly’s Status Report for 4/6/24

Kelly’s Status Report for 4/6/24

Work Accomplished: Integrated OpenAI API to get recipe using current inventory Connected the results from the filter options (Dietary Restrictions & Cuisine) Prompted the LLM and asked to end the recipe with !!! since there was a problem earlier where some long recipes are cut 

Kelly’s Status Report for 3/30/24

Kelly’s Status Report for 3/30/24


Work Accomplished:

  • Made the video button direct to a page that will stream the video feed from the scanners.
    • Added a back button that goes back to the inventory page.
    • For now UI displays “Camera is not active” but once the server is set up, I’ll put the video stream when the camera is active.
  • Made the filter button open up a pop up with options for dietary restrictions and cuisines.
    • This widget is reused between the saved tab and the generate tab.
    • Made the UI look better by having the buttons colored in when selected, as opposed to the default multiple select check boxes.
    • When clicking “Apply Changes” it will return a list of the selected options that we can use to sort saved recipes or ask the LLM for specific recipes.
  • Displayed a warning label before clicking the “Generate Recipe” button.
    • This will warn users to check if their ingredients have gone bad before using them since our app does not have a way to know exactly if ingredients have gone bad besides getting an estimated number of days the produce usually lasts.
    • The animation starts playing after the button is pressed.
  • Implemented a Swipe-to-delete feature for the inventory items and recipes.
    • This will be used if the user accidentally forgets to scan out items or would like to delete saved recipes
    • Might need to implement it to allow users to delete some quantity and not the entire ingredient (not as urgent since main use will be through scanning out)
    • This action will need to update the database.

Here is a video demo for the new UI changes in action

 

  • LLM prompt engineering: made a prompting flow for recipe generation

  • Tested Meta’s Llama2 7b and 13b and Google’s Gemma using this prompting flow
    • Llama2: the only recipes that it knows are different varieties of stir fry, but it includes questionable ingredients in it, like bananas. When I ask for a specific cuisine, it still gives me stir fry and claim it is Italian or Korean when they are the exact same recipe.

    • Gemma: the only recipe that it knows is tomato and garlic soup. It uses ingredients that are not in the inventory. It does not give any other recipe even when I ask for a different one or change the cuisine.

    • Deduced that these LLM Models are very limited for our use case and we do not have enough time to fine-tune/train the models (learning how to and actually doing it)

 

  • Tested the prompts on ChatGPT
    • Worked a lot better
      • Gave a good variety of answers
      • Would change the recipes according to dietary restrictions and cuisine
      • Fixes itself if it uses an incorrect ingredient and I ask it to try again
    • Given the success rate of ChatGPT in creating recipes, our intended use, it significantly outweighs the pros of local LLM models, which are that they are more secure and can be faster.
    • Decided to use ChatGPT for our recipe generation although it is a cloud-based LLM and we originally said we would use a local LLM.

 

  • Set up OpenAI API on Flutter app
    • Tried making a sample app and it should work but we do not have credits so it just gives me “429” rate limit errors.
    • Waiting to purchase credits using our budget
      • I asked on Wednesday. They told me that on Monday someone in ECE purchasing should reach out to me.
      • My plan was to finish it for the interim demo, but now I cannot.
      • Might need to pay out of pocket if this takes too long.

 


Progress:

  • On track with the updated gantt chart.

 


Things to complete next week:

  • Get OpenAI credits
    • Implement prompting flow in the app to generate recipes
    • Display recipes on new UI elements (pop up with recipe name and recipe page with instructions)
  • Create custom item & custom recipe input pop up when clicking + button
  • Cookable toggle on saved recipes page

 

Team Status Report for 3/30/24

Team Status Report for 3/30/24

Risks: Kria KV260 acceleration documentation doesn’t seem the most up to date Changes: Using ChatGPT instead of Llama2 or Gemma ChatGPT has more variety of recipes while Llama2 and Gemma do not work as intended, and we do not have enough time to fine-tune and 

Kelly’s Status Report for 3/23/24

Kelly’s Status Report for 3/23/24

Work Accomplished: Worked on the UI of the Recipes page Figured out a way to customize the tabs to be what I want Made the Sort By buttons a customized radio button so that it toggles. (followed this example: https://blog.logrocket.com/create-and-customize-flutter-radio-buttons/) Reused this widget so that it 

Team Status Report for 3/16/24

Team Status Report for 3/16/24


Risks:

  • Video streaming is too slow. Switching to using a python library, hopefully the RPi Zero’s can handle it.
  • Found online image dataset of produce that isn’t labeled for detection, just classification. Considering if switching to classification may be better for finishing on time.

Changes:

  • Decided to scan items outside of the refrigerator.
    • Will still be close to the opening of the refrigerator door to use feature tracking
    • Camera will point down so that background will not be cluttered
    • might get a tablet to mount on the front of the refrigerator so that user can see video feedback
    • idle mode of the camera will be determined by a circuit that either uses a connection when the door is closed or a photodiode to detect light inside of the refrigerator

Progress:

  • Received the KV260. However, power supply wasn’t included so setup/testing has been delayed.
  • Front end of app almost complete
  • Video stream proof of concept showed promise, however it was bottlenecked by image processing.
  • Sous-Chef databases locally created, Python connector works and can perform CRUD operations.

 

Kelly’s Status Report for 3/16/24

Kelly’s Status Report for 3/16/24

Work Accomplished: Worked on the ethics assignment Worked on the UI of the app to match Figma mockups Used tab bar for the Recipes Page https://pub.dev/packages/buttons_tabbar  This button style is what we want from the Figma mock up, but it is difficult to customize. Added