Author: choysal

This week, we continued further with integration.

Individual progress this week has been sparse as I have been working on other final projects for other classes.

Diya and I met this Wednesday to come up with a thorough testing plan for all the software parts. We asked Josh for feedback and we got some mostly about the analytics and gesture recogniton part, which we are taking a look at. Our testing plan is linked in the team status report.

I am currently working on running tests and will finish my assigned tests tommorow. The three of us also met to work on the poster which I have also been working on my assigned parts for, we also need to finish the poster by tommorow.

While I am all done with my individual components and integrated as much as possible without the working display, I am behind on my testing assignments as I had a lot of other final projects to work on this week. Other than my final exams I am free to work on this so I will have to grind this out. Tommorow I will finish all the tests and finish my portion of the poster. I will update here if there is anything else to be said on the weekend progress.

 

This week I further worked on the display to webapp connection (as mentioned in my previous report) as well as ironing out a lot of bugs and making good progress towards our final product.

Early this week, Diya and I met to fix a bug in the webapp that led to a recipe item database mixup that was causing problems when calling funcitons on those recipe objects. This was due to merge conficts that were not resolved properly, so we fixed that and got the webapp to work again.

Once that was done, I got the display pi to send information to the webapp and the webapp to send information to the pi. This was what I had ready to demo on Monday.

I also fixed the I2C issue mentioned earlier where the display pi was not showing up as a reciever device to the gesture pi. Now, the connection was working properly.

Once these changes were made, the pi was now successfuly opening a websockets connection with the webapp and able to send information back and forth. However the problem remained that in order for this to run at the same time as the display, there would need to be some sort of multiprocessing used. After talking to rebecca and doing some additional research, the tradeoff of having the pi open for consistent back and forth communication with the webapp would be a pretty big power draw, and is ultimatley unecessary. Thus, I the following pipeline for WiFi communication:

• upon bootup the display pi opens communication with webapp, which then sends acknowledgement.
• display pi waits for a recipe to be selected on the webapp.
• once “start recipe” is selected on the webapp, the recipe data is loaded into the display and the display starts. The websockets connection is closed.
• session data is loaded into a payload
• upon completion of a cooking session, the pi opens another websockets connection with the webapp and sends the payload. The connection is then closed and the glasses are shut off.

While this means that once a recipe is selected it is locked in and cannot be changed, this is a tradeoff that is worth it for lower powerdraw, since if the user wants to change recipes they can simply finish or exit the session then select another recipe.

Once that pipeline was set up and fully functional, I spent some time integrating the WiFi communication with the I2C connection and the gesture recognition signals. This code is tested in pair integrations but not the fully integrated pipeline. More on this later in the report.

I recently met with Diya to test these changes on the analytics functionality she worked on. There were a few bugs created from merge conflicts that we sorted out, but now we are confirmed that this works! The workflow of booting up the glasses, selecting the recipe, running through the receipe and receiving the session data in the webapp works. We ran through a couple of edge cases as well.

The next thing I need to work on (its already done we just need to test actually) is the i2c connection between the pis – specifically checking that the data sends properly and the gestures captured by the camera are changing the display. This works on the code really well, we just need to test this on the hardware. Though I don’t ancipate there being many issues, the biggest thing is setting up the pis and the connections and also getting x forwarding to work so I can acutally test this. I was hoping if Rebecca was done with the display to just quickly test this on the already existing hardware, but this doesn’t seem feasible. I should’ve done this earlier but I was busy with other classes and this seemed like a daunting task.. but I think we should be fine as long as I get this tested and check the full integration pipeline (only thing missing is the physical display on the glasses) within the next few days.

Beyond this, I will continue to fix the issues that have been coming up during integration, and though Diya and I are close to done on the software end (knock on wood), I will also do what I can to help with the hardware. We also have to do the presentation and start our final report and poster so this will also take a lot of time and I will put a lot of effort towards this when I am not activley working on implementation.

The week of carnival (right before interim demo), I coded the I2C integration with the display renderer, which has the raspi receiving and processing signals from the gesture recognition output from the other pi. We discussed this during the interim demo, but at that point there were metallic connection issues so it did not work. As of this thursday those issues have been fixed by Rebecca, and the pis were handed off to me to debug an additional issue with the display pi not showing up as an I2C slave, probably due to a minor issue on the script. I have since taken the pis and have to debug this issue tommorow, but since then I have been working on the wifi connection between the display pi and the webapp, which I will discuss in a later section of this report.

Right after interim demo and at the beginning of this week, Diya and I worked on writing up the features, details, and user / technical requirements for the analytics feature. We discussed this with Tamal and Josh, and continue to further refine this.

This week, my primary work was getting the WiFi connection between the display raspi and the webapp working. Essentially, we want the user to click a button on the webapp and send information about that particular recipe such as steps and ingredients to the glasses, that can then display that information. And after the cooking session is done, send information about the stats of that session (time spent, steps flagged) to the webapp.

Originally, we did not do enough research on how exactly to do this and thought to use bluetooth. Then, realizing this was not feasible for an AWS deployed webapp (our original plan), we switched to possibly deploying the server on the pi itself. We should’ve done a lot more research admittedly way before this, but this is the sitatuion we were in.

Before starting on this task, I did several hours of research on how to best do this, as I have never worked with a raspi before, rarely ever worked with networking, and certianly never deployed a server on a raspi. I quickly realized this was not very feasible on a pico and also not very necessary or scalable, and realized that it would be a better idea to simply have the raspi and the webapp have a websockets connection.

Implementing this involved setting up daphne and channels for websockets requests on the webapp, setting up the websocket routing and comsumer commands, and adding the feature to send information over to the pi. Then on the pi side, sending an initial connection request to the webapp on boot up, setting up a listener on the display script, and listening for and displaying information that was sent. This fully works, and now the pi and webapp are connected and can send and receive info from one another over WiFi.

While doing this I realized that something happened with our webapp, probably because of merge conflicts, that has caused diya and I to have very different model definitions on the webapp (my guess is that when diya started working on the webapp frontend some of the models I made did not get pushed / pulled, and Diya wrote additional definitions for these models that do not match with the old db). This has caused our database to be messed up, and a lot of recipes have mismatched ids and feild names. I am meeting with Diya tommorow to sort this out, should not be too bad hopefully. I have also noticed I am having some issues where my local git clone is not letting me pull (repo does not exist error), which I hope is just a local issue. I can probably delete the clone off of my computer then reclone, but I am waiting on confirmation from my teammates that this is a me issue and not a repo issue.

This coming week, we have a lot of work to do. For me personally, once this issue is sorted out, It will be relativley easy to display the info that the webapp sends to the glasses and vice versa, so we can demo this on Monday. Then, I will figure out the I2C bug. After that, I will see where else the team needs help. I assume this will be on the analytics feature on the webapp, so that will be my task for the next week including random things that will inevitably come up here and there with integration.

I feel like I have done a lot of work this week especially on the WiFi connection so I feel on track in that sense, but I definetley have a lot of work to do in these coming weeks!

This week, our team got some feedback about lack of complexity, specifically on my end as the webapp wasn’t enough complexity wise. As a result, we reshuffled some responsibilities and assigned new ones. I finished what I planned to do, here is a summary:

I completley developed the pygame display which will be displayed on our glasses, including all the instructions (go forward, go back, show ingredients, elaborate steps). This information will be displayed at all times to the user as a HUD, so I made sure to take extra consideration to provide text wrapping, intuitive controls (ex. locking progression on steps when ingredients are up to reduce amount of input confusion, making sure “back” returns to the beginning of the previous step and not the wrapped text), and easy to edit variables for line amount and textbox sizes in case things change when the program is rendered on the actual display. I also added a progress bar that displays at the bottom of the glasses to show how many steps have been completed. I used arbitrary keyboard inputs as placholders, which Diya then attached to the gesture recognition signals. The display output is now fully hooked up to the gesture recognition.

Tommorow, I need to add the functionality of recipe experience levels (exp gained through finishing recipes) and display them on the profiles on the webapp. Diya is currently experiencing some git issues which have temporarily slowed down progress as integration of the changes on her branch are not merging with the others (something to do with virtual environment issues), but we are resolving that and then I will implement this functionality.

We also discussed what we can implement after the demo, and had a discussion about what exactly we want to do with the networking feature.

I had the idea that to integrate the glasses and the webapp networking features more, we should have a feature that allows the user to pick one other person to cook with before starting the recipe, and once it starts, should be able to see their progress on the recipe live on the glasses. This will require some lower level work with WiFi connections / networking, and will also require some work on the arduino and I2C- so I’m excited to work on this after the interim demo.

EDIT: upon discussion today (sunday march 30th), we have decided that an analytics feature may be more useful and focused on the user requirements and use case. More on this in the next report.

In addition, if the display is able to handle the current display, Diya and I thought it would also be cool to be able to display the recipe selection on the glasses. I will be in charge of this if that happens, which will be another task for after interim demo.

This week, I worked on the webapp further.

I have finished functionality for the recipe page, including adding reviews and displaying information about the recipe like ingredients and steps.

I have also finished functionality for the recipe selection page including filtering with tags.

It does not look great, but the functionality works as it should and is pretty complete.

I have currently been testing with manually inputting recipes in a form, which includes inputing name, steps, ingredients, and a tag. This should be pretty easy to migrate to real recipes as I have set up the models so that reading from the JSON file that Diya has scraped from the web into models should allow it to easily update through the whole existing webapp, though I am sure that there will have to be some adjustment.

I have accomplished the goals I set for myself last week, so I would say I am on track.

The next step is spending some time finishing up the recipe running page and adding a score bar (for the player level) on the profile page that updates with completed recipes – this should be quick. I aim to finish this page tommorow, so I can be on track to complete the remaining integration tasks. Then, Diya and I will work on integrating the actual web scraped recipe database into the webapp, which will probably take some time as there will be a few details that have to be adjusted here and there. Once that is done, I can remove all the placeholder data. And lastly, the entire team needs to figure out how to connect this application to the glasses. This includes getting text to display on the glasses, which the three of us will figure out. Once that is done, the webapp portion should be good to go for the interim demo. If there is extra time, Me and/or Diya can work on making the webapp look good (though I will probably spend some time rearranging things to make the website more legible anyways).

 

 

This week, I worked on the ethics assignment and further worked on the webapp.

I have the login and registration completed from before, and I have also worked a lot on the backend this week and done planning of the total website / backend structure.

I have been focusing on the recipe selection page primarily, as well as adding features to the profile page to reflect the changes we made last week.

In addition, after a conversation with staff, we decided to add a feature in which users can request further elaboration on a cooking term in a step that will show up on the display. I’ve accounted for this feature in the webapp structure.

I did not have much time to work on capstone this week as I landed from my flight Monday night and had a large assignment due for another class on Thursday, so I am behind and was not able to complete the functionality of the entire website which was my goal that I assigned for myself last week. However, I do think this was an unrealistic goal, and I think I have more time than I initially thought to work on my end of the project as integration will be happening later than initially planned. In addition, I will have a lot more time to work on capstone this coming week.

This week, I absolutley must complete atleast the recipe selection and recipe running page with steps showing to be on track. I am confident that I can get this done since I have the backend setup and I have more time this week. Once that is done, I will continue working on functionality for the rest of the webapp.

This week, I primarily worked on the wireframes for the website. They are in the design report.

We did a lot of rework of the web application such as what networking features to include, so the wireframe was very necessary to get our entire group on the same page about what we have to design and what exactly our web app is used for. This will be a good way to have everyone on the same page if we add / change anything additionally.

I am behind schedule- I did not finish all of my goals to have the website working with basic functionality since we redefined our scope for the webapp. I have to spend a lot of time this week actually getting the website coded (especially with all the changes just added).

As mentioned above, this week I aim to have basic functionality of the entire site (or atleast the most important pages (recipe selection, recipe running page) working).

This week, I got some more work done on the webapp.

I was able to add profile functionality, with basic profile picture editing, bio adding, and follow / unfollow funtionality. Here are some screenshots:

https://drive.google.com/drive/folders/11YqsSXDr60VctZHMILzMGM5wH-JmnyWg?usp=sharing

(wordpress is seriously dropping the quality so that nothing is readable, so instead I linked it here^^).

It’s still quite barebones, but the functionality works and can be built upon.

Earlier in the week, our team also spent a decent amount of time working on further specifying the design specs for our design review presentation, and also working on the slides themselves.

I was able to add the unfollowing / following and basic networking functionality to the webapp, as well as making the website navigatable via hyperlinks, but it is not as clean as I would like and there are a few (though small) bugs. I also was unable to add the score / level functionality.

Overall, I was not able to get a satisfactory amount of work done on this project this week due to other classwork and other circumstances, so I am running behind.

As for what I have to do to get back on track, this week I really want to be completley done with the app functionality – Diya is tasked with working on parts of the networking (nameley deploying on AWS, and adding more advanced friending features such as requests), so I will focus on the score part and also cleaning up everything else. And then, depending on what my teammates are doing, I will either work on the website front end and make it look better, or help Diya out with the gesture recognition.