Posted on

Team Status Report for 3/23

There are two significant risks – one is the projector, which has been an ongoing risk. The risk of knocking down or dropping the projector is still a concern because we have not finished this portion of the project. However, we have figured a way to somewhat mitigate this risk, which will be discussed in the system design changes paragraph. Another risk is the object reidentification. It might not work as well once implemented due to lighting conditions. Right now, the implementation uses the UI wireframe, which has less uncertainties as in a real kitchen. We will have to do testing to see how much of a problem this will be – if it is one. To manage this, we will experiment with different lighting conditions.

One change that we have made is simplifying the projector mount. Instead of having the user set different angles during calibration, we will instead fit the projection to any angle – within reason (the projector still must be pointed at the table). We are doing this by computing a homography between a sample image taken by the camera and the image on the screen. Now, we do not need to make the projector mount rotate up and down. The change will make the mount cost less because we do not need a rotation mechanism.

No changes to schedule. We are on track.

Posted on

Caroline’s Status Report for 03/23

I completed the user interface for the web application where the user chooses the recipe. This involved designing the user interface on Figma and then implementing it in flutter.

On schedule.

Next week, I will work on connecting this UI to the backend and also work on the network interface on the AGX.

Posted on

Tahaseen’s Status Report for 03/23

This week, I spent some time working with the Xavier and tried to connect the camera to it so that we would be able to do some preliminary testing. However, there were a few roadblocks here that should be resolved this week as we work to onboard the Xavier. On the other hand, I revisited the projection method such that it utilizes the camera and prevents the user from having to mess with the projector setup. This would involve another step in the calibration process that would create the homography based on the existing camera and projector setup. Furthermore, I have designed the first iteration of the hardware mount that will be 3D printed this week unless an existing metal/wood solution is found. There is some concern that the filament will not hold up.

I am on track and this week I plan to work on completing the calibration procedure, mount and sorting out our central compute.

Posted on

Team Status Report for 3/16

The main risk to our project is the projector mount. If not designed properly with the correct material, the projector could fall and break which would compromise the entire system – especially given our budget. In order to prevent this, we have to doubly engineer our system to ensure the safety of the projector. Additionally, we have to design hard stops to indicate our presets in order to make the user experience more intuitive.

Slight changes were made to the mounting after receiving a larger tripod.

No changes to schedule. We are on track to integration.

Posted on

Tahaseen’s Status Report for 03/16

This week, I talked to Marios and acquired a much larger tripod that is perfect for mounting the projector. Since the new one is sturdier than the first, the first will be used to mount the camera + AGX and the second for the projector. This is a positive change from our previous system and will require new brainstorming on mounting designs. Additionally, I outlined the projector calibration and have begun implementing the script for it. Currently the procedure is (1) inform user calibration is taking place, (2) detect table outline through user markers, (3) verify that the projected calibration grid matches an unwarped version and either reimage in software or prompt the user to raise the projector to preset height/angle changes. Caroline and I also discussed the pros and cons of using the homography that came with the Flutter UI vs a Python script that would require a frame by frame input.

I am on track and this week I plan to schedule out large portions of time to implement a working calibration script and design a hardware mount with our new materials. Our team is on track for integration before the interim demo

Posted on

Caroline’s Status Report for 03/16

I finished the majority of the projector UI work and worked on the backend integration of the voice commands and interface. The projector UI connects to a socket, and whenever a voice command such as “play video” is spoken, it goes into a queue that is processed by the socket, then sent to the interface that plays the video. An example is here: https://drive.google.com/file/d/1TG8wGAeKivAeXD9qwv5chu-26XNts_kV/view?usp=drive_link. The voice command and server socket are both processes in Python that are launched in one script. There is also support for changing the color of the boxes depending on if the ingredients are for the current recipe, incorrectly picked up, or not being used.

Status: on schedule

Next week, I will work on warping the image in flutter with the homography Tahaseen gave me. I will also start working on the recipe web app.

Posted on

Tahaseen’s Status Report for 03/09

This week, I placed the order for a stronger projector and received it. I had been testing with a smaller Panasonic projector, but after testing in different light modes, I found that the brightness was insufficient to meet the requirements of the project. Afterwards, I helped my team setup our coding workspace in Bitbucket so that we could track our progress by issue using Jira and regularly merge with each other to avoid conflicting code. This way, integrating everything at the end will be more seamless and we can avoid any downstream issues. There was a shuffle in task assignment so I will no longer be focusing on setting up the backend of the user site. Instead, I will be focusing more time in making sure the projector is correctly and intelligently calibrating to the workspace table. Therefore, I focused on researching potential steps for the projector calibration mode through existing calibration methods for projectors. After reviewing various resources (these 2 were particularly helpful: https://us.seenebula.com/blogs/buying-guides/projector-calibration and https://simplehomecinema.com/2022/07/06/what-you-need-to-calibrate-your-tv-or-projector/), I was able to gain more insight into the types of things I needed to ensure were calibrated for a satisfactory user experience.

I am currently on track with my tasks. Next week, I will have a barebones setup for a calibrator that I can display through the projector. Additionally, I need to think of an alternative mount for the larger projector on the tripod.

Posted on

Caroline’s Status Report for 03/09

I finished the flutter UI. All the elements are placed in the correct spots, and the dynamic elements are working (videos, timers, buttons). I have been working on the integrating with the backend. I was able to use Flask to run a server locally and access the program as a website. Now, I am working on getting the sockets to work between the UI and a python server. I have been having some difficulty with this step due to unclear documentation, but I am still on track.

On schedule.

To Do: Finish implementing the sockets and test with voice commands.

Posted on

Team Status Report for 2/24

The most significant risk is still mounting the projector. We still plan on placing the projector on a tripod angled downwards on the table. We are currently looking for stable tripods and secure attachment options to manage this risk. Potential options include making a CAD model and 3D printing attachments like we are for the camera case or buying off the shelf options. We are still testing the homography and seeing if the warped image is high quality enough to completely use this method. Our contingency plan is still placing the projector at a 90 degree angle if this does not work. There have not been any major design changes since last week. There are no changes to the schedule. We will be planning system integration this week and will implement it after spring break.

Posted on

Caroline’s Status Report for 2/24/24

I accomplished my task of making the voice commands faster and more accurate. I did this by modifying my previous code and searching for tips online on how to make it faster. I had experimented with different Speech-to-Text models, like Vosk and Google Assistant, to see what would happen, but I found that the Whisper + Picovoice Wake Word combination worked the best. I also worked more on the Flutter UI, based on the design that I created from last week. I am on schedule.

I hope to try modifying the voice commands so that actions are actually taken when a command is spoken (ie. actually pausing a video). I will also finish the Flutter UI so that we can test it on different table surfaces and see what it looks like.