Diya’s Status Report for 08/03/2025

Last week, I focused heavily on the design report, contributing significantly to refining the software details and web application requirements. I worked on structuring and clarifying key aspects of our system to ensure that our implementation aligns with our project goals. A major portion of my work involved ironing out details related to gesture recognition, particularly ensuring it aligns with our defined gesture language. This included adjusting parameters, and troubleshooting inconsistencies to improve accuracy. I have attached a photo of an example of the gesture recognition for the defined gesture language in the design project report.

In the upcoming week, my main focus will be on improving the accuracy of gesture recognition. This will involve fine-tuning detection thresholds, reducing latency, and optimizing the system for different environmental conditions to ensure robustness. I will also continue working on refining the design report if needed and contribute to the integration of the gesture system into the broader application.

Charvi’s Status Report for 3/8/25

This week, I primarily worked on the wireframes for the website. They are in the design report.

We did a lot of rework of the web application such as what networking features to include, so the wireframe was very necessary to get our entire group on the same page about what we have to design and what exactly our web app is used for. This will be a good way to have everyone on the same page if we add / change anything additionally.

I am behind schedule- I did not finish all of my goals to have the website working with basic functionality since we redefined our scope for the webapp. I have to spend a lot of time this week actually getting the website coded (especially with all the changes just added).

As mentioned above, this week I aim to have basic functionality of the entire site (or atleast the most important pages (recipe selection, recipe running page) working).

Team Status Report for March 8, 2025

Project Risks and Mitigation Strategies

At this point the only hardware delay we’re waiting on is the displays. The original displays ordered were cancelled, so similar alternate parts had to be identified and purchased- these come from sellers with much more recent activity, and before the parts were ordered we asked and received confirmation that they were still in stock, so we are much more hopeful that they will arrive when expected. We then, of course, have to hope they work- they fill the same technical niche as the original part, as small, lightweight displays with optical lenses meant for heads-up displays, but since they are slightly different we no longer have proof that the deconstruction we have planned will work exactly as expected.

Beyond hardware, another key area of risk is ensuring the gesture-based interactions function smoothly and intuitively. As part of my work on the gesture system, I have been refining the detection algorithms and ensuring that they align well with user expectations. The primary risk is potential latency or inconsistency in recognizing gestures, especially in dynamic environments. So I am looking vat sensitivity tuning and error correction methods to mitigate this. 

Changes to System Design

No changes to the system design were made this week.

Schedule Progress

Independent schedule progress is addressed in our individual reports. While the order of some tasks has shuffled, our work is roughly on track.

 

Meeting Specific Needs

 

Part A was written by Diya, part B was written by Rebecca, and part C was written by __.

Part A: … with consideration of global factors. 

The product solution we are designing aims to meet the need for intuitive and hands-free interaction in augmented reality environments. By incorporating a gesture-based input system, users can interact naturally without relying on physical controllers, improving accessibility and ease of use. The gesture recognition is designed to be responsive to gestures and adaptive to the users’ input. 

Part B: … with consideration of cultural factors.

Food is, of course, a foundational part of just about every culture everywhere on Earth. It’s one of the first things people may turn to when trying to reconnect as an adult with a culture they may have missed out on, for various reasons, in childhood, or lost track of somewhere along the line- or when reaching out to friends and family of different cultures. But starting to learn how to cook, or learning how to cook in a style one is entirely unfamiliar with, can be a daunting undertaking; by lowering this barrier of entry, we hope that people will be more encouraged to attempt this particular path to connecting with a new culture, be it their own or their loved ones’.

Part C: … with consideration of environmental factors.

[ ]

We are trying to design as low of a cost environmentally as we can. Our physical frame is designed to last for long use, by being as sturdy as possible. We plan on using as small as an EC2 instance as needed to deploy our web app and store any databases. And inherently, cooking at home is better for the environment: less one time use plastics are used for packaging m, there are less delivery emissions, and food waste is lessened if someone is able to easier cook food that they want to eat when they want it. Our recipes included are all simple recipes, that aim to use ingredients that the user will probably already have: also reducing food waste. Overall, home cooking is more environmentally friendly than ordering takeout, which is an option that many people that don’t feel comfortable cooking at home will end up doing- CookAR hopes to bridge that gap and get people to start homecooking. Though there are additional environmental costs associated with creating a new physical glasses product and running a website, we aim to be intentional about what we design and what resources we use in a way that is as environmentally conscious as possible. 

Rebecca’s Status Report for March 8, 2025

Report

I have learned that despite being supposedly a very mainstream device, the Raspberry Pi is… remarkably unintuitive. I’m using Raspberry Pi OS Lite to run the Rasppi headless and ssh into it, though for an as-of-yet unclear reason my computer does not seem to be able to resolve the Rasppi’s hostname and I have to use the IP address directly. This has only worked for this week’s development because I have direct access to my router and its IP address assignments at home, and will immediately have to resolve this issue upon returning to campus. Figuring out how to get into the Rasppi took just far, far too long because every single tutorial and answered question and Guide To Headless Rasppis that I could find online assumed that you could resolve the hostname, which is a very reasonable assumption, and simply bizarrely untrue in my case. I don’t know.

The Raspberry Pi OS Imager also doesn’t tell you what the name of the OS you’re using is, and even on the main website it’s just kind of… a throwaway inline parenthetical comment. Despite being the main thing the entire community uses to refer to the major versions of the OS. And so many things changing between them. It’s. This was a conscious decision. Why would you do it this way.

After figuring out the issue and getting into the board, getting it to talk to the camera was relatively simple (though I had the cable in upside down for a bit, which was deeply frustrating to discover after an hour and a half of debugging. So it goes). I’m using the native Raspberry Pi Camera Module, which is, you know, supposed to be the native camera and therefore straightforward to use, but you would just not believe the number of problems I have had because I’m using a native Pi camera instead of a USB camera.

First photograph captured from the Pi camera! It’s blurry and poorly exposed because I’ve left the protective plastic tab over the lens, since it still has to travel back to Pittsburgh. I expect the quality to be better once I take that off.

I also discovered that OpenCV’s primary image capture method VideoCapture(camera_id) is not compatible with libcamera, the regular Raspberry Pi camera library, because of course it isn’t. Surely nobody would ever want to use OpenCV straightforwardly on a minimal Raspberry Pi. Surely that couldn’t be an extremely common desire and mainstream goal. Can’t imagine.

However Picamera2, the Bookworm Python wrapper for libcamera, is configurable enough to be kind of compatible itself with MediaPipe.

(As an aside: all of the libraries I used this week I was able to access via pip, and that also seems to be the simplest way to use MediaPipe, except for Picamera2, which was only accessible with apt; I set the include-system-site-packages flag in my pyvenv.conf to true to be able to use it.)

This is the MediaPipe on Raspberry Pi tutorial I started from. It doesn’t work on its own, because it relies on the OpenCV method that doesn’t work, but I used it and the associated tutorials linked to set up the Python environment (sigh. why did it have to be Python) and MediaPipe installation.

I found this document, which was exactly what I want to do, with the sole caveat that it’s ten years out of date. Picamera has been displaced with Picamera2, which has been significantly streamlined and so the translation isn’t 1:1, and I’m not familiar enough with either library to do a quality translation. Sigh.

I ended up being able to scavenge bits and parts from this document and from the Picamera2 repo examples to make an trial script which captures images off the camera and streams them via OpenCV (in this case over my ssh tunnel, which was very slow, but I hope an amount of that is the ssh streaming and it will speed up when I cut that).

I was able to then graft the working Picamera image-capture script onto the MediaPipe script provided in the first tutorial. I’m just using a generic model right now, not our own custom gesture language, but it is a proof that the software works on the hardware. If only just barely. It ran at this point extraordinarily slowly, and there was truly just an untenable amount of lag between my hand motions and what I saw on the screen, and even more between the motion of the frames on the screen and the MediaPipe overlay. Making it run faster became a critical priority.

Image capture of the MediaPipe hand tracker running on the Raspberry Pi.

I modified the camera configuration to tell the software reading the camera both the resolution that I wanted out of it (which was already there) and the raw native resolution of the camera. This seemed to fix my zoom problems- the camera’s field of view was far smaller than I had expected or wanted; it seemed to have just been cutting out a 640×480 box out of the center of the FOV. With access to the native resolution, it appears to be binning the pixels to the desired resolution much more cleanly. Additionally, I fixed the framerate, which had previously just been at “whatever the software can handle”. Pinning it at 1.5fps sped up MediaPipe’s response time greatly, improved its accuracy, and all of the lag functionally disappeared (even still streaming the output). It also kept the board from getting so dang hot as it was before; Raspberry Pis since the 3 underclock when they hit 60C, and according to my temp gun that’s about where I was hanging before I fixed the framerate, so that was probably also contributing to lag.

Image capture of the MediaPipe hand tracker working on the Raspberry Pi.

1.5fps is a little lower than I wanted it to be, though. I switched the framerate and recognition outputs to feeding to a printline and turned off the streaming, and was able to trivially double my framerate to 3fps. This hits the spec requirement!

If possible, I’d like to try to pull OpenCV entirely out of the script (with the possible exception of its streaming feature for debugging purposes) since Picamera2 seems to have all of the functionality of OpenCV that I’m using, and in a much more lightweight, Raspberry Pi-native library. I believe this may help me improve the responsiveness of MediaPipe, and will certainly make the script cleaner, with fewer redundant, overkill tools. However, since it works just fine as is, this is not a high priority.

Progress Schedule

I’ve shuffled around my tasks slightly, accelerating the work on MediaPipe while pushing off the HDMI output slightly, so I’m ahead on one section while being behind on another. I’ve also had to put off measuring the power consumption of the Rasppi until I had the recognition model working- in retrospect, I don’t know why measuring the power consumption was placed ahead of getting the most power-hungry algorithm working. I’m not particularly worried about the lead time on the battery, so I’m fine with that getting estimated and selected a bit later than expected.

Next Week’s Deliverables

Originally next week was meant to be the MediaPipe recognition week, while this week was for the HDMI out, but this has been flipped; I plan on working on the code which will generate the display images next week. Additionally, I’ll have to figure out how to log into the Rasppi on the school’s internet connection when I don’t know its IP address directly, which may take a nontrivial amount of time.

Team Status Report for February 22, 2025

Project Risks and Mitigation Strategies

  • A key bottleneck in the project is the delay in receiving the Raspberry Pis which don’t arrive until Monday. This impacts our ability to test power consumption and system performance. To mitigate this delay, Rebecca has already obtained the SD cards and has flashed OS onto them, so the boards can be booted immediately upon arrival. This will allow us to immediately start testing early in the week.
  • If there are additional delays with hardware setup, we will proceed with software side testing on local machines and simulate hardware behavior to continue the development.
  • Charvi already has profile, following and registration functionality figured out and she is integrating and debugging these components.
  • Diya has set up gesture recognition locally and is currently testing its accuracy. If accuracy issues arise, we will adjust the model parameters, consider alternative gesture recognition models, or refine preprocessing techniques.
  • Rebecca has drafted the headset CAD so a base exists for the mount points and, as mentioned above, prepped the SD cards and found instructions for installing and running OpenCV on a Raspberry Pi, to jumpstart our work on this.

Changes to System Design

  • No changes to the system design were made this week.

Schedule Progress

  • A few of the Rasppi-testing-related tasks expected to be done this week have been pushed to next week on account of the boards not arriving. No other changes have been made. Some of next weeks tasks may be pushed into spring break, on account of this delay and possibly underestimating the time the design report will take to write, but this slack time should catch all of it.

Rebecca’s Status Report for February 22, 2025

Report

  • The remaining parts (Rasppis, SD cards, camera, HDMI cables for testing) were ordered Monday/Tuesday, and most of them arrived by Friday. What remains to arrive are, unfortunately, the Rasppis themselves, which is bottlenecking my progress.
  • I’ve drafted the CAD as expected (see below) which took. so long. Just so very many more hours than I thought, which, yeah, probably should have seen that one coming. Note for future modelling: do not use splines. Splines are the highway to incompletely constrained sketches. God, why did I use splines.

  • I’ve flashed the SD cards with the Raspberry Pi OS so I can boot them as soon as they arrive (expected Monday). Diya and I can sit down and check the model, and run the tests I need for power draw then.

Progress Schedule 

  • A few of the tasks I expected to be done this Friday/Saturday did not get done because of the delivery delay. I cannot measure the power consumption of a board I do not have.
  • If I don’t get horribly unlucky, this should be done early next week; some of next week’s tasks may end up getting pushed into spring break, but we have that slack time there for that very reason. Most of the time dedicated to this class for the upcoming week is likely to be spent writing the design report.

Next Week’s Deliverables 

  • The design report, obviously, is due at the end of next week. This is a team deliverable.
  • The Mediapipe/OpenCV-on-Rasppi tests I expected to do this week. We’ll know the power consumption, and then I can figure out what kind of battery I’ll need.

Charvi’s Status Report for 2/22/25

This week, I got some more work done on the webapp.

I was able to add profile functionality, with basic profile picture editing, bio adding, and follow / unfollow funtionality. Here are some screenshots:

https://drive.google.com/drive/folders/11YqsSXDr60VctZHMILzMGM5wH-JmnyWg?usp=sharing

(wordpress is seriously dropping the quality so that nothing is readable, so instead I linked it here^^).

It’s still quite barebones, but the functionality works and can be built upon.

Earlier in the week, our team also spent a decent amount of time working on further specifying the design specs for our design review presentation, and also working on the slides themselves.

I was able to add the unfollowing / following and basic networking functionality to the webapp, as well as making the website navigatable via hyperlinks, but it is not as clean as I would like and there are a few (though small) bugs. I also was unable to add the score / level functionality.

Overall, I was not able to get a satisfactory amount of work done on this project this week due to other classwork and other circumstances, so I am running behind.

As for what I have to do to get back on track, this week I really want to be completley done with the app functionality – Diya is tasked with working on parts of the networking (nameley deploying on AWS, and adding more advanced friending features such as requests), so I will focus on the score part and also cleaning up everything else. And then, depending on what my teammates are doing, I will either work on the website front end and make it look better, or help Diya out with the gesture recognition.

 

Diya’s Status Report for 02/22/2024

This past week, I was catching up on a lot of work since I was really sick the previous week and also had a midterm on Thursday. Despite that, I made significant progress on the project. I worked on the design presentation slides and presented them on Monday. Additionally, I have been working on OpenCV gesture recognition, ensuring it runs locally on my computer. The setup is now complete, and I am currently in the process of testing the accuracy of the model. Now that I have the gesture recognition working locally, the project is back on schedule. The progress aligns with our timeline, and I am ready to move forward with the next steps.

For the upcoming week, I plan to

  1. Continue testing the accuracy of the gesture recognition model
  2. Work on Figma design for the website interface.
  3. Start working on the networking portion of the project for the webapp
  4. Begin drafting and finalizing the design review report submission.

Charvi’s Status Report for 2/15/25

This week, I focused on getting a basic webapp with basic functionalities working locally. This included the add recipe functionality, select recipe functionality, as well as log in, log out, and registering users.

I got this done (though it looks quite bare bones)

Here are some screenshots (wordpress is lowering the quality of these photos alot and I cannot find out how to fix this, but essentially there are basic user register, log in, recipe input, and recipe selection functionality):

 

Our team also spent a lot of time this week redefining our project goals, scope, and complexity. This is explained more in the team status report, but what that means for the webapp is that we plan to add a lot of networking functionality, specifically with friending users, veiwing progress of others, and scoring / acheivments. This means there is quite a lot I will need to add / rework into the webapp.

In addition, we decided to no longer use Unity XR development as this seems like overkill for what we want to do (static text images), as it is for XR game development. Thus, we will just generate static images on the application in a simpler way.

My goals for this week according to my previous report were to get the basic MVP for the website done, which I had mostly done (though as mentioned, it will need a rework). I also wanted to work with Unity XR development a little, which I did not do, due to the rework of our goals.

Due to the project goal rework, I would say I am running behind schedule. Though, I don’t think by too much – I can use the existing basic webapp and models for the DB that I have created, just need to move things around and add some more functionality.

My goals for next week are to, as mentioned before, add networking functionality, friending / following functionality, and add score / level / acheivment functionality for completing recipe to the webapp.

Diya’s Status Report for 02/15/2024

This week was quite challenging for me as I was sick for most of it. Last week, I was recovering from a bacterial infection, and unfortunately, I came down with the flu this week, which led to a visit to urgent care. Despite that, I was still able to contribute to the project, particularly in refining our approach to hand gesture recognition and pivoting my role to contribute more effectively.

Initially, I had misunderstood the gesture recognition task, thinking I needed to find and train a dataset myself. However, after further research, I realized that MediaPipe provides a pretrained model with 90% accuracy for gesture recognition, meaning I could directly integrate it without training a new model. This required a shift in my focus, and I pivoted to handling the networking aspect of the project to add complexity and depth to my contribution.

Beyond that, I have been actively involved in facilitating group meetings, translating our use case requirements into quantitative design requirements, and preparing for the design review presentation this week.

Given my health issues, my progress is slightly behind where I initially wanted to be, but I have taken steps to ensure that I am back on track. Since the gesture recognition aspect is now streamlined with MediaPipe, I have moved focus to the networking component, which is a new responsibility. I am catching up by working on setting up the foundational pieces of the social network feature in our web app.

Next week, I plan to make significant progress on the networking component of the project. Specifically, I aim to set up user authentication for the web app to allow users to create accounts, implement user profiles, which will include cooking levels, past recipe attempts, and preferences, and develop a basic social network feature, where users can add friends and view their cooking activities.