Category: Harry’s Status Reports

This week I polished the style for some pages and caught minor issues on the webapp. Since most of my classes are done this week, I will spend more time on the project next week. I will finalize the page styles before we start working on the video and test all of the functionalities of the webapp and fix any issues I find.

May 3 Update:

I was able to finalize page styles and clean up the codebase. I also added a few functionalities.

• display custom recipes on the dashboard page
• added “recipe” page to show individual recipes
• allow users to edit their recipes
• allow users to delete recipes
• allow users to publish their recipes
• shows if an ingredient is in stock on the “recipe” page
• added a “discover recipes” page for users to see published recipes
• added a warning dialog when user tries to delete a device (as shown below)

Similar to device registration, I added reCAPTCHA to the “email grocery list” dialog to prevent spamming.

This week I have been focusing on webapp deployment. I was able to launch an EC2 instance and setup Django and Apache to serve the webapp in the cloud. I also purchased a web domain for the webapp and configured an SSL certificate for it to enable HTTPS and get better security (the webapp can be found at https://b6foodtracker.com). I updated the settings in the GCP console and FB Developer console to make sure OAuth works on the server. We ran into some issues about file permissions and python imports when deploying but ultimately we were able to get them working.

I looked into the Django messages framework and incorporated it into our webapp. Now we are able to pass messages to the user and display them on the webpage. Current four types of messages are supported: info, success, error, and warning.

I also worked on some CSS styling and HTML code restructuring. I worked on multiple pages and was able to complete the sign in page.

I will keep working on styling and help Jay to get the AJAX component working. I will also continuously deploy the latest version of the webapp to the server and monitor if there’s any issues. Next week, when the majority of the tasks are done, we will start testing our project.

This week I have been looking into implementing the API endpoint for communicating with the RPi. I also investigated potential security risks and how to mitigate them. I was able to implement the update_inventory views method with precautions against malicious actors. The current communication protocol should be able to handle the following situations:

1. Someone tries to update another person’s inventory if they manage to get their serial number: we are including a client secret in the POST request for server to validate
2. In case of a data breach where all of the serial number and secret pairs are leaked: we are storing the SHA256 hashed string of the client secret in the database
3. Someone intercepts the POST request and tries to get the client secret: we will use HTTPS to send the requests so that all of the data is encrypted
4. Someone intercepts the POST request and sends it repeatedly to the server (replay attack): we will send a timestamp along with the data and record the latest in the database. Since the request cannot be modified by the attacker, only the first request will be accepted
5. Someone brute force the serial numbers on the registration page: we are planning to add CAPTCHA to the page

This week I have been continuing working on the user registration system. I discovered a bug where users are able to sign in with another account when they are already signed in, which would consistently break the webapp. I figured out a solution by inserting custom functions into the django social auth sign in pipeline. In order to streamline the sign in process for the users and avoid needing to repeatedly call a method for all view methods to retrieve user information, I scrapped the UserInfoFetcher I wrote previously. After investigating through the python-social-auth source code, I was able to create custom functions and insert them into the oauth pipeline. By also creating a custom User model, I am able to save users’ profile pictures and emails (which sometimes are not provided by default) into request.user without needing to manually fetch them. I also worked on some html pages to create a usable webapp and started working on the recipe page where people can get auto-generated grocery lists. In addition, I was able to work with Jay to debug some code and help Keaton setup his Django environment so that he can work on some css/bootstrap.

For next week, I will continue working on the recipe page and try my best to finish a working version before the demo day. I will also start working on the API endpoint for updating inventories. I have come up with a detailed design for the transmission encryption and I will discuss it next week.

I don’t have much to report for this week due to midterms. However, I was able to complete the user registration system. I will keep working on the webapp backend and try to start working on integration next week.

This week I have been looking into setting up the user registration system. Although I was blocked by the Facebook OAuth for two days, after looking into Google and Facebook’s API documentations and python-social-auth’s codebase, I was able to get the sign-in components working. By default, I don’t have access to a user’s email from the Facebook OAuth API (which I need to finish the user registration steps), and I don’t have access to the user’s profile picture from both services (which is something nice to have for the interface), so I looked into ways I could retrieve this information. After digging through many documentations and StackOverflow examples, I was able to figure out how to fetch it from Google and Facebook’s API endpoints. Specifically, I found the correct endpoints from both services so that I could send GET requests with an access token provided from OAuth to get the information I need. As shown below, while the resulting code only consists of a couple lines, it took a lot of documentation reading and trails and errors to find the correct parameters and API endpoints to make it work. Since the ways I can fetch the information are mostly the same for both services, the next step will be coming up with a suitable design pattern to achieve better reusability and scalability in case we need to add more OAuth providers in the future.

Next week I will finish up the user registration system and start working on the view methods for other pages. Since we have designed the class models for the database, the implementation should be trivial when compared to the OAuth component. Specifically, I will create the User model and a corresponding form to send to the user if they log in for the first time to complete their registration. I will also work with Jay to connect the backend logic with the user interface so that we can have a working prototype.

Because of the upcoming midterms and interviews, I wasn’t productive this week. I worked on the design report document with my the teammates and started setting up the AWS account with Jay. I will try to get more done over the next week and the spring break.

This week I have been having conversations with the team members about the overall design of our project and investigating the implementation details. I brainstormed a list of problems we might face and questions we need to answer. For example, what communication protocol we need to use for the components to talk to each other, how to control the camera, etc. After doing some research, I was able to draw the high-level block diagram (see team report). After some investigation, I came up with a design that can utilize different AWS services to handle the backend of our project with scalability at its core. Specifically, the RPi will upload the image to an S3 bucket, which will trigger an AWS Lambda function that houses the CV component. The Lambda function will then send the output (a list of detected objects, for example) to our webapp server. After talking with faculties, we decided not to pursue this path, at least for now, because scalability isn’t really our priority at the moment. However, we might explore with this design after the MVP. For the MVP, we will choose to use only one AWS EC2 instance to run both our webapp and the CV component.

I have also been modifying & drawing more diagrams (flow diagrams, class models, etc.) in preparation for the upcoming design review. For the next week, I will start building the backbone of our webapp and get OAuth working. I will work with Keaton to continue developing the webapp component, and if we can receive the hardware in time, I will also start testing with the camera.

This week I have been discussing with the team members about the implementation details and how our project will look like on a higher level. After the discussion, we have decided to use a Raspberry Pi in place of a Jetson Nano board and hand the image processing steps to the cloud. The reasoning behind this decision is detailed in the team report.

I drew the workflow diagrams so that we can all have a better understanding of the architecture of our project (see team report). I am still working on the implementation diagrams and I will post them once our design is finalized. I began investigating ways the board will communicate with the web server. If we use standard HTTP POST requests, we will need to implement a robust verification system so that one user’s device won’t update another user’s inventory. I also looked into ways users will interact with the system (e.g. how to set it up out of the box, how to reset, etc.). Current candidates are pairing the device with a smart phone via Bluetooth and attaching a touchscreen with a graphical user interface to the fridge. Further discussions on these topics are needed as we might need to rescope our project and adjust the requirements.

Over the next week, I will work with Keaton to perform the benchmarks and Jay to design communication protocols & APIs.