Category: Brandon’s Reports

I finished the frontend of the heat map for the pet activity logs. I switched the heat map library I was using to jsheatmap as it was compatible with TypeScript and had documentation that was easy to understand and implement with. As well, I confirmed a data format between the CV and web application for the pet activity logs. The CV would send an array containing the amount of frames a pet stayed in a specific cell of the room that goes in order from the top left to the bottom right of the room.  This array is sent to the web application, and the web application would keep the tally of the amount a pet spent in a specific cell of the room. Also, we have integrated this task between the CV and the web application.

To implement notifications for a pet entering the forbidden zone, I had to learn about polling, which is making requests to the backend every few seconds to see if the notification flag, which is turned to true by the CV as it makes a POST request whenever the pet moves into the forbidden zone, is set to true. Once the notification flag is true, then a popup is created on the frontend to notify the user that the pet has entered the forbidden zone. After trying a few methods of polling, I found a custom React hook called useInterval that was able to do the task above. We have successfully tested sending a notification from the CV to the web application.

The last task I have started but have not finished is displaying live video feed of the room to the user. We based our implementation on a previous implementation that send a constant loop of arrays of byte data representing the image taken from the CV to the web application locally, but we ran into problems with encoding and decoding the arrays, especially during GET and POST requests.

I am slightly behind schedule as I should have finished displaying the live video feed. I should be able to finish up this task next week as it is a matter of encoding and decoding the images correctly while the data is being passed on between the GET and POST requests.

In addition to finishing up displaying the live video feed of the room, I hope to add authentication by using Google OAuth2.0, and I hope to clean up the user flow of the website and make the website more intuitive, appealing, and user friendly. Lastly, I look to deploy the frontend after all tasks are finished.

The above picture uses fake data, but it is an accurate depiction of the heat map.

The above picture is what is displayed on the frontend when a pet enters the forbidden zone.

This is code currently in progress for displaying live video feed.

This week, I focused on integration and creating the user flow for the website to present for the interim demo. We integrated the CV and Web App by communicating information through sockets, which took some time to understand how to implement sockets and how we can send data between the server and client as we had no prior experience through sockets. Individually, I made sure the forbidden zone data was formatted in a way that was easy to use for the CV. As well, I received a picture from the CV that was used for creating forbidden zones, and I printed a notification for when something enters the forbidden zones. Other than integration, I wanted to show the user flow of the website for the interim demo. Essentially, a user would login and can create a pet model, which is chosen on the dashboard page, for each one of his/her pets. The user would first upload pet images and then choose the forbidden zones for that pet to complete the pet model. After the interim demo, I successfully deployed the backend of the web application, which will allow us to move communication from sockets to GET and POST requests. I explored deploying through Amazon EC2 and then Amazon Elastic Beanstalk after running into some issues with EC2, but I was able to eventually run an EC2 instance of the backend.

As of the updated Gantt chart for the interim demo, my progress is on schedule.

I will be looking into finishing displaying the frontend of the heat map of the pet activity logs and confirming a data format from the CV for these pet activity logs. In terms of integration, we hope to run the same tests as we did in the interim demo but use GET and POST requests instead of sockets. I will start to look into displaying live video feed of the room on the website and displaying a notification on the website if a pet enters the forbidden zone.

In terms of testing I have done on the web application, I have made sure the functionality of uploading pet images and entering forbidden zones work successfully by entering sample data and making sure this data shows up on the backend by running “python manage.py shell”, a Django command to look at what data is in the database, and calling GET requests to the database to see if I can get back the sample pet images and forbidden zone data. As well, we did integration tests between Web App and CV as described above for the interim demo. In the next few weeks, I plan on getting around ten participants to test out my web application and see if the use case requirement of 95% of people can finish the tasks displayed on the web application without external help is satisfied.

Above picture shows the backend is deployed at 54.173.23.127.

I focused on making the task on my web application ready for integration with other parts of the system. Specifically, I accomplished allowing get requests for the pet images and forbidden zones based on the pet name specified in the get request so the ML and CV are able to get this data respectively. As well, I created a dashboard to give the user easy access to creating new forbidden zones and upload pets images for a new pet and view already entered forbidden zones and pet images for existing pets.

I am changing my schedule such that I will be working on tasks that are most relevant for what we will be integrating in our in person meetings. Hence, I will focus on pet activity logs next after finishing integrating forbidden zones as this is one of the core tasks that we hope to integrate between all sub systems.

I hope to finish displaying the frontend of the pet activity logs and work on the integration process for the pet activity logs between the web application and the CV.

This week I worked on creating the frontend and backend of a user designating the forbidden zone for a pet. I learned how to use state and props in React so that I could keep track of the state of each forbidden zone cell. For my forbidden zone example, I created a 10×10 grid (sizing will most likely be changed), and I keep track of the position in the x and y directions and if a user has clicked the cell (which represents a forbidden spot or not). Once a user submits their forbidden zone data, then it sent to the database that is managed by the Django REST framework. A picture of the frontend for the forbidden zone task is shown below.

I was hoping to looking into how to display a frontend for the pet activity logs using the heat map library described from last week, so I am currently behind. I hope to do this for next week and make sure everything is stable for the interim demo.

Like I mentioned above, I hope to create a frontend for displaying pet activity logs using a heat map. As well, I hope to make sure everything I currently have is working for the interim demo, and work on integrate some parts of my web application with other parts of the project.

This work, I worked on the ethics assignment, looked into how to deploy the machine learning aspect of the project onto an Amazon EC2 instance, and worked on implementing choosing forbidden zones as a user and showing pet activity logs to the user. As our team has been investigating where to run the machine learning, such as running it on the web server, I looked into how matching learning can be run on the Amazon EC2 server as I have the most experience with this platform. From my research, there seems to multiple examples of other machine learning models being deployed on Amazon EC2 using a deep learning amazon machine image, so I believe that this is a possible idea. For implementing pet activity logs as a heat map, I have found out that I can use a library called heatmap.js that will allow me to create heat maps based on data taken from OpenCV.

I am behind on work due to homework in other classes, but I will be making sure that all the core tasks for the web application are functional before the interim demo.

I hope to finish implementing forbidden zones and showing pet activity logs in the next week. As well, I hope to create a user interface design based on my user flow diagram I created.

During this week, I spent most of my time working on the design review document with my team. For example, we figured out the details for the testing, verification, and validation as this was one of the points that wasn’t fully addressed in the presentation. As well, I fixed the block diagram for the web application side so that it aligns more with what a block diagram should look like. In addition, I added a flow diagram of the user experience on the web application into the design review document. I have looked into how I can allow users to choose forbidden zones on the web application, but I have not implemented this yet.

I am currently behind schedule, but I will hope to make up the task of implementing the frontend and backend for a user choosing forbidden zones this week.

I hope to fully implement choosing forbidden zones on the frontend and storing this data on the backend. As well, I hope to start looking at how to display pet activity logs on the frontend using data that would be given by the Raspberry Pi.

I have set up the web application that uses React and Django and learned about how to get and send data between React and Django using Axios, which can send GET and POST requests between React and Django, and learned about the Django REST framework, which can take data from the POST requests and create objects based on the Django models to store in the database. Though I have some experience with React and Django, I had no experience with using Axios and the Django REST framework, which took a few hours of learning and debugging to learn the full capabilities of these tools. After learning these tools, I implemented allowing the user to upload images for pet classification. Specifically, I designed a frontend page that allows users to upload images and a Django model to store these images and the type of these images (png, jpeg, etc.). After the user uploads an image, a POST request is called using Axios that sends the image to the Django REST framework, which creates an object based on the Django model specified above that stores the images into the database.

Based on my schedule, I am currently behind on tasks for the web application side. The plan is to push back my tasks back a week as I have other work commitments this week and have been having technical issues with my computer recently, and there are no major tasks that other people are relying on me finishing at the moment so pushing back tasks by a week is okay.

The task I hope to work on this week is to test our ideas related to users choosing forbidden zones for a pet. I want to test both the frontend, specifically users choosing spots, which designate forbidden zones, on a grid overtop a 2D room image, and the backend, specifically storing this data in an array-like structure, implementation of these tasks. Though I am pushing back tasks by a week, I believe this is an important task to do before spring break as it is a core task of the project that other teammates will eventually rely on, and it is important to see what changes I will have to make based on testing our implementation ideas of the forbidden zones this week.

I worked on setting up React integrated with Django. Previously I have used each framework by itself, but I have never used them together so it took some time and debugging to set this up. I looked at my previous implementation of a user uploading images on Django and thought about how to replicate this with React and Django, but I have not implemented this yet. As well, I worked on the Design Review Presentation. Specifically, I thought about the solution approach as it applies to the web application side of the project. I chose React and Django due to my previous experience with these frameworks and how they are the leading frameworks in frontend and backend respectively. One safety concern is related to the privacy of the user, specifically how users will be able to see a live video feed of their room. Hence, a malicious user may be able to see the live video feed of another user, which is an invasion of privacy. But, there are security protection tools supplied by Django that can address these concerns. I have made minor adjustments to the block diagram from last week for the web application side, and I will be finishing up the other points of the Design Review Presentation with my team within the next day. 

Currently, I am behind schedule due to the tests and homeworks of other classes. I will be working on how to use the React and Django frameworks together, and I will implement a user being able to upload pet images to the website to catch up on the schedule.

After I finish working on the tasks that I am behind on, I will be working on one of the core features of the website, which is allowing users to choose forbidden zones for a pet and figuring out how to store this data into the database.

Web Application Development (17-437) covers the engineering principles related to my part of the design process, specifically working with the backend using Django. I have basic experience with React, and I will be learning more about developing with React in the upcoming weeks.

During class, I presented our proposal and did peer reviews of other proposal presentations. Outside of class, our team met to have an initial discussion of the design review for the project. We fleshed out the specific tasks that we each need to complete based on our role for the project, and we figured out how we could integrate all parts of the project together. Personally, this helped me have a greater understanding of what I specifically needed to do for the web application and what data I needed to send to or receive from the Raspberry Pi. After the meeting, I created a block diagram for the Web Application side of the project (as shown in the diagram below). Specifically, I listed the tasks that the user is able to complete on the frontend, and design Django models on the backend that would be used to store data in the database. Finally, I updated the schedule based on what we discussed in the meeting for the design review.

Based on the schedule we created, my progress is on schedule. For next week, the main priority is to finish the design review presentation, specifically finishing the block diagram and discussing testing, verification, and validation. In terms of the web application, I hope to set up a React application integrated with Django. Also, I hope to work on letting users upload images for pet classification, which I have prior experience doing.