Team Status Report for March 8, 2025

Project Risks and Mitigation Strategies

At this point the only hardware delay we’re waiting on is the displays. The original displays ordered were cancelled, so similar alternate parts had to be identified and purchased- these come from sellers with much more recent activity, and before the parts were ordered we asked and received confirmation that they were still in stock, so we are much more hopeful that they will arrive when expected. We then, of course, have to hope they work- they fill the same technical niche as the original part, as small, lightweight displays with optical lenses meant for heads-up displays, but since they are slightly different we no longer have proof that the deconstruction we have planned will work exactly as expected.

Beyond hardware, another key area of risk is ensuring the gesture-based interactions function smoothly and intuitively. As part of my work on the gesture system, I have been refining the detection algorithms and ensuring that they align well with user expectations. The primary risk is potential latency or inconsistency in recognizing gestures, especially in dynamic environments. So I am looking vat sensitivity tuning and error correction methods to mitigate this. 

Changes to System Design

No changes to the system design were made this week.

Schedule Progress

Independent schedule progress is addressed in our individual reports. While the order of some tasks has shuffled, our work is roughly on track.

 

Meeting Specific Needs

 

Part A was written by Diya, part B was written by Rebecca, and part C was written by __.

Part A: … with consideration of global factors. 

The product solution we are designing aims to meet the need for intuitive and hands-free interaction in augmented reality environments. By incorporating a gesture-based input system, users can interact naturally without relying on physical controllers, improving accessibility and ease of use. The gesture recognition is designed to be responsive to gestures and adaptive to the users’ input. 

Part B: … with consideration of cultural factors.

Food is, of course, a foundational part of just about every culture everywhere on Earth. It’s one of the first things people may turn to when trying to reconnect as an adult with a culture they may have missed out on, for various reasons, in childhood, or lost track of somewhere along the line- or when reaching out to friends and family of different cultures. But starting to learn how to cook, or learning how to cook in a style one is entirely unfamiliar with, can be a daunting undertaking; by lowering this barrier of entry, we hope that people will be more encouraged to attempt this particular path to connecting with a new culture, be it their own or their loved ones’.

Part C: … with consideration of environmental factors.

[ ]

We are trying to design as low of a cost environmentally as we can. Our physical frame is designed to last for long use, by being as sturdy as possible. We plan on using as small as an EC2 instance as needed to deploy our web app and store any databases. And inherently, cooking at home is better for the environment: less one time use plastics are used for packaging m, there are less delivery emissions, and food waste is lessened if someone is able to easier cook food that they want to eat when they want it. Our recipes included are all simple recipes, that aim to use ingredients that the user will probably already have: also reducing food waste. Overall, home cooking is more environmentally friendly than ordering takeout, which is an option that many people that don’t feel comfortable cooking at home will end up doing- CookAR hopes to bridge that gap and get people to start homecooking. Though there are additional environmental costs associated with creating a new physical glasses product and running a website, we aim to be intentional about what we design and what resources we use in a way that is as environmentally conscious as possible. 

Team Status Report for February 22, 2025

Project Risks and Mitigation Strategies

  • A key bottleneck in the project is the delay in receiving the Raspberry Pis which don’t arrive until Monday. This impacts our ability to test power consumption and system performance. To mitigate this delay, Rebecca has already obtained the SD cards and has flashed OS onto them, so the boards can be booted immediately upon arrival. This will allow us to immediately start testing early in the week.
  • If there are additional delays with hardware setup, we will proceed with software side testing on local machines and simulate hardware behavior to continue the development.
  • Charvi already has profile, following and registration functionality figured out and she is integrating and debugging these components.
  • Diya has set up gesture recognition locally and is currently testing its accuracy. If accuracy issues arise, we will adjust the model parameters, consider alternative gesture recognition models, or refine preprocessing techniques.
  • Rebecca has drafted the headset CAD so a base exists for the mount points and, as mentioned above, prepped the SD cards and found instructions for installing and running OpenCV on a Raspberry Pi, to jumpstart our work on this.

Changes to System Design

  • No changes to the system design were made this week.

Schedule Progress

  • A few of the Rasppi-testing-related tasks expected to be done this week have been pushed to next week on account of the boards not arriving. No other changes have been made. Some of next weeks tasks may be pushed into spring break, on account of this delay and possibly underestimating the time the design report will take to write, but this slack time should catch all of it.

Team Status Report for February 8, 2025

Project Risks and Mitigation Strategies

  • The most significant risk is that the AR display won’t be arriving till mid March so the integration has had to be pushed. The board has an HDMI output so we can test the system using a computer monitor instead of the AR display. If the AR display arrives later than expected we will conduct most of our testing on an external monitor.
  • Two other significant risks are that the gesture recognition algorithm cannot run on the Raspberry Pi, or that the power demand of it running is too high for a reasonably-weighted battery. If either of these are true, we can offload some of the computational power onto the web app via the board’s wireless functionality.

Changes to System Design

  • We are adding a networking and social feature to the web app. This involves adding scoring incentive to completing recipes that are then translated to levels that are displayed on the user profile. Users can follow each other and view each other’s progress on profiles. We will also deploy our application on AWS EC2. This change was necessary to add back complexity into the project since we are directly using the pre-trained model from MediaPipe for gesture recognition.

Schedule Progress

We have reworked and detailed our Gantt chart.

Meeting Specific Needs

Part A was written by Diya, Part B was written by Charvi, and Part C was written by Rebecca.

Please write a paragraph or two describing how the product solution you are designing will meet a specified need…

Part A: … with respect to considerations of public health, safety or welfare.

By using the gesture interaction method, users can navigate recipes without touching the screen and this reduces cross contamination risks especially when handling raw ingredients. Additionally, by allowing the users to cook in a step by step manner, it helps the users to focus on one task at a time which allows beginner cooks to gain confidence. Also, with gesture control user’s will have less distractions such as phones or tablets and can minimize the risk of accidents in the kitchen. Moreover, the social network feature can allow users to track their progress and connect with other beginner cooks to promote a sense of community amongst new cooks.

Part B: … with consideration of social factors.

Our target user group is “new cooks” – this includes people that don’t cook often, younger adults and children that are in new environments where they have to start cooking for themselves, and people that have been bored or confused by cooking on their own. Our CookAR product will allow people to connect with one another on a platform focused on cooking and trying out new recipes, which will lead them to be motivated by like-minded peers to work on furthering their culinary knowledge and reach. In addition, by game-ifying the cooking process by allowing users to level up based on how many new recipes they tried, CookAR will also motivate people to try new recipes and engage with each other’s profiles, which will be displaying the same information about what recipes were tried and how many.

Part C: … with consideration of economic factors.

A lightweight headset made from relatively inexpensive parts- fifteen-dollar Raspberry Pi boards, a few dollars at most for each of the rest of the peripherals; the most expensive part is the FLCoS display, and even that is only a few tens of dollars for a single item off the shelf- is ideal for a target audience comprised of people trying to get into something they haven’t done much or any of before, and so a target audience which is unlikely to want to spend a lot of money on a tool like this. Compared to a more generalized heads-up-display on the market (or rather, formerly on the market) like the Google Glass, which retailed for $1500, this construction is cheap, while still being fairly resilient.

Additionally, this same hardware and software framework could be generalized to a wide variety of tasks with marginal changes, and a hypothetical “going-into-production” variant of this product would very easily be able to swap out the four-year-old Raspberry Pi Zero W for a something taking advantage of those years of silicon development- for instance additional accelerators like an NPU, tailored to our needs- in a manner such that scale offsets the increase in individual part price.

Team Status Report for February 8, 2025

Project Risks and Mitigation Strategies 

  • Gesture Recognition Accuracy and Performance Issues
    • Risk: the accuracy of the gesture detection might be inconsistent or there might be limitations in the model chosen
    • Mitigation: test multiple approaches (MediaPipe, CNNs, Optical Flow) to determine the most robust method and then fine tune the model
    • If vision recognition is very unreliable, explore other sensor based alternatives such as integrating IMUs for gesture detection
  • Microcontroller compatibility
    • Risk: the microcontroller needs to support the real time data processing for the gesture recognition and AR display without latency issues
    • Mitigation: carefully evaluate microcontroller options to ensure compatibility with CV model. The intended camera board is designed for intensive visual processing.
      • If the microcontroller is not suitable for the CV model, we will look into offloading some of the processing power from the microcontroller to the laptop. This may require sending a great deal of data wirelessly and must be approached with caution.

Changes to the System Design 

  • Finalizing the device selection: There are fewer development board options than modules; however, we need the development board as we do not have the time to sink into creating our own environment. So we will be using the ESP32-DevKitC-VE Development Board, which implements a WROVER-E controller. This has the most storage capacity for its form factor and reasonable price.
  • Refining the computer vision model approach: Initially only considered a CNN based classification model for gesture recognition but after more research also testing MediaPipe and Optical Flow for potential improvements

Schedule Progress

Our deadlines do not start until next week, so our schedule remains the same.