Posted on

Brandon’s Status Report for 4/29/2023

At the beginning of the week, we did testing on the system for the final presentation. A test that was relevant to the web application was testing the notification speed to the user after being notified of a pet entering the forbidden zone. We learned that the speed was limited by the polling rate, or rate of GET requests from the frontend to the backend, for checking if a notification should be displayed to the user or not. As well, we worked on updating the block diagram and displaying a demo for the complete solution for the final presentation. In terms of technical implementation, I worked on the live video feed. My initial iteration was displaying an image of the room a frame per second. The CV would send a byte array to the backend, which would be transformed into an image, and the frontend would poll the backend every second using the useInterval React hook and display the image. Though this was easier to do implementation wise and the live video feed would be considered “functioning”, we wanted to find a way to display more frames per second. Hence, we learned about StreamingHttpReponse object in Django. Essentially, this object would take a constantly updating byte array and display the image for the specific byte array. We were able to show more frames/second of the room, but there are some issues with loading the images quickly, which can lead to image flickering. We hope to reduce the image flickering so that users can smoothly view the live video feed.

As of right now, I am behind schedule. I will be working on finishing up the live video feed, finishing up the authentication process, making a user friendly and intuitive website, and deploying the frontend before integration and testing on Tuesday.

Next week, I hope to finishing up the individual tasks for the web application as mentioned above. As well, we look to do integration and testing before the final poster and report.

In addition to the systems test we have done, I have done unit tests for each part of the web application, including choosing forbidden zones, displaying a heat map for the pet activity logs, displaying a notification a user for a pet entering the forbidden zones to name a few, so that system integration is as smooth as possible.

Posted on

Team’s Status Report for 4/29/2023

Risks and Mitigation

We have not tried to get our whole project functioning on an RPi yet. So far the RPi seems to be working as intended, but we have only tried it separately from the project, and have not worked to integrate it yet. This will be attempted starting this coming week, ASAP. If we run into issues we cannot solve, we will reach out, and worst case scenario the ‘final’ project will be run on a laptop. We will also need to redo some tests once the code is running on the RPi, which could lead previously successful tests to fail. We will handle any such cases the way we would handle a failed test on any platform – seek to alter/refine the code to improve efficiency.

Project Changes

There are no changes at this time.

Schedule Changes

A few of our remaining features are a bit behind schedule, as the last week of classes was quite busy – we’d like to improve the live video (very low frame rate), log-in is in progress, and we haven’t had a chance to integrate with the RPi. It is our goal to mostly wrap up these loose ends by the end of this weekend/very beginning of this week, so we can move more exclusively into testing for the week leading up to the live demo.

Testing 

We tested notification speed from the pet entering the forbidden zone on the camera to the CV detecting that the pet has entered the forbidden zone. This was done via slow mo video – to mark the points in time when a ‘pet’ entered a forbidden square in real life, versus when the CV detected this (marked by the camera window closing). We wanted this to be under a second, and our results showed that the detection speed was ~0.625 seconds. 

Then, we tested the notification speed from the pet entering the forbidden zone to the Web App displaying a notification to the user that a pet has entered the forbidden zone. Once again, this was done via slow mo video, marking when the pet was detected by the CV and when the notification first shows on the web app. The goal was for this to occur in under 10 seconds, and the resulting data showed that this process occurred in ~1.125 seconds. We found that the speed from pet entering the forbidden zone to the web app displaying the notification is most limited by the rate of the frontend of the web app making requests to the backend to check if a notification should be created or not.

Next, we tested the accuracy of CV tracking by sampling certain frames and marking where a ‘human’ thinks the pet is versus where the CV thinks the pet is. We wanted this to be with 1ft, and using the the difference in centers of the bounding boxes as our metric, we were usually within 3-6 inches (measured based on knowing what the distance of two spots in the frame would be, i.e. the distance from the cat’s head to its back).

Lastly, we’ve measured the time between when an animal enters the room and when it is picked up by the CV motion detection (also with slow mo video). This took about ~0.75 seconds on average, well less than our goal of 5 seconds. 

 

Note: Many of these tests (any pertaining to tracking speed/frame rate) will be repeated when we switch platforms from a laptop to an RPi. It is our hope that we have sufficient wiggle room in the laptop numbers that any slow downs on the RPi will not bring us outside of our targets. If needed, we may seek out ways to make our design more efficient.



Posted on

Brandon’s Status Report for 4/22/2023

I finished the frontend of the heat map for the pet activity logs. I switched the heat map library I was using to jsheatmap as it was compatible with TypeScript and had documentation that was easy to understand and implement with. As well, I confirmed a data format between the CV and web application for the pet activity logs. The CV would send an array containing the amount of frames a pet stayed in a specific cell of the room that goes in order from the top left to the bottom right of the room.  This array is sent to the web application, and the web application would keep the tally of the amount a pet spent in a specific cell of the room. Also, we have integrated this task between the CV and the web application.

To implement notifications for a pet entering the forbidden zone, I had to learn about polling, which is making requests to the backend every few seconds to see if the notification flag, which is turned to true by the CV as it makes a POST request whenever the pet moves into the forbidden zone, is set to true. Once the notification flag is true, then a popup is created on the frontend to notify the user that the pet has entered the forbidden zone. After trying a few methods of polling, I found a custom React hook called useInterval that was able to do the task above. We have successfully tested sending a notification from the CV to the web application.

The last task I have started but have not finished is displaying live video feed of the room to the user. We based our implementation on a previous implementation that send a constant loop of arrays of byte data representing the image taken from the CV to the web application locally, but we ran into problems with encoding and decoding the arrays, especially during GET and POST requests.

I am slightly behind schedule as I should have finished displaying the live video feed. I should be able to finish up this task next week as it is a matter of encoding and decoding the images correctly while the data is being passed on between the GET and POST requests.

In addition to finishing up displaying the live video feed of the room, I hope to add authentication by using Google OAuth2.0, and I hope to clean up the user flow of the website and make the website more intuitive, appealing, and user friendly. Lastly, I look to deploy the frontend after all tasks are finished.

The above picture uses fake data, but it is an accurate depiction of the heat map.

The above picture is what is displayed on the frontend when a pet enters the forbidden zone.

This is code currently in progress for displaying live video feed.

Posted on

Brandon’s Status Report for 4/8/2023

This week, I focused on integration and creating the user flow for the website to present for the interim demo. We integrated the CV and Web App by communicating information through sockets, which took some time to understand how to implement sockets and how we can send data between the server and client as we had no prior experience through sockets. Individually, I made sure the forbidden zone data was formatted in a way that was easy to use for the CV. As well, I received a picture from the CV that was used for creating forbidden zones, and I printed a notification for when something enters the forbidden zones. Other than integration, I wanted to show the user flow of the website for the interim demo. Essentially, a user would login and can create a pet model, which is chosen on the dashboard page, for each one of his/her pets. The user would first upload pet images and then choose the forbidden zones for that pet to complete the pet model. After the interim demo, I successfully deployed the backend of the web application, which will allow us to move communication from sockets to GET and POST requests. I explored deploying through Amazon EC2 and then Amazon Elastic Beanstalk after running into some issues with EC2, but I was able to eventually run an EC2 instance of the backend.

As of the updated Gantt chart for the interim demo, my progress is on schedule.

I will be looking into finishing displaying the frontend of the heat map of the pet activity logs and confirming a data format from the CV for these pet activity logs. In terms of integration, we hope to run the same tests as we did in the interim demo but use GET and POST requests instead of sockets. I will start to look into displaying live video feed of the room on the website and displaying a notification on the website if a pet enters the forbidden zone.

In terms of testing I have done on the web application, I have made sure the functionality of uploading pet images and entering forbidden zones work successfully by entering sample data and making sure this data shows up on the backend by running “python manage.py shell”, a Django command to look at what data is in the database, and calling GET requests to the database to see if I can get back the sample pet images and forbidden zone data. As well, we did integration tests between Web App and CV as described above for the interim demo. In the next few weeks, I plan on getting around ten participants to test out my web application and see if the use case requirement of 95% of people can finish the tasks displayed on the web application without external help is satisfied.

Above picture shows the backend is deployed at 54.173.23.127.

Posted on

Team’s Status Report for 4/8/2023

Risks and Mitigation

Our primary ‘risk’ currently – although it’s pretty much already happened – is that we will not be able to fully integrate the ML into our project due to unavoidable personal setbacks. The mitigation plan for this is to keep the ML separate for now and focus on integrating the CV and Web App. If things go particularly smoothly we may be able to look into integrating the ML in the last week or two of the project, but for now we will operate on the assumption that this may not happen and act accordingly. 

Project Changes (why & cost)

There aren’t really any changes from what was discussed in the last blog post. As we are proceeding with a more separated project for the time being, we will develop the CV+Web App side of the project with an RPi for the primary hardware. Max will work with the Jetson for the ML side. As these parts are all coming from inventory, this will not affect the budget for our project.

Schedule Changes

As discussed at the interim demo, the primary change to our schedule is the order in which we will focus on integration. Rather than starting with CV+ML as planned, we will be focusing very heavily on integrating CV and Web App. This will allow us to figure out the web app/hardware interactions separately from the ML while that is worked on separately.


Posted on

Brandon’s Status Report for 4/1/2023

I focused on making the task on my web application ready for integration with other parts of the system. Specifically, I accomplished allowing get requests for the pet images and forbidden zones based on the pet name specified in the get request so the ML and CV are able to get this data respectively. As well, I created a dashboard to give the user easy access to creating new forbidden zones and upload pets images for a new pet and view already entered forbidden zones and pet images for existing pets.

I am changing my schedule such that I will be working on tasks that are most relevant for what we will be integrating in our in person meetings. Hence, I will focus on pet activity logs next after finishing integrating forbidden zones as this is one of the core tasks that we hope to integrate between all sub systems.

I hope to finish displaying the frontend of the pet activity logs and work on the integration process for the pet activity logs between the web application and the CV.

Posted on

Team’s Status Report for 4/1/2023

Risks and mitigation

Currently, the most significant risk is that we may be unable to incorporate the Machine Learning part into the system due to personal reasons. Hence, we would be unable to identify different pets in the room and the creation of the different forbidden zones for each pet would not make sense anymore. To mitigate this risk, each person is implementing a different part of the project. So, we could integrate the Web App and CV only if the Machine Learning would not be available for integration, and we could add in the Machine Learning portion if it is finished before the final demo. If the system only includes the Web App and CV, then the creation of one forbidden zone for all pets and detecting if any pet enters the forbidden zone would make the most sense as we would be unable to differentiate the pets without Machine Learning.

Project Changes (why & cost)

If we are unable to add in the Machine Learning part of the project, then we would not have to use a Jetson, which is most beneficial for training the machine learning model, to support integrating the system, which would reduce the cost of the project significantly as the Jetson costs $150 and the internet adapter needed for the Jetson costs $15-30. We would start with two computers (one for Web App and one for CV) to start integrating the system. If we can integrate Machine Learning into the project, then there would be no major changes.

Schedule Changes

We are going to change the order of integration from what was listed on the schedule. We plan to start with integrating Web App and CV, specifically sending forbidden zone data chosen by the user from the Web App for the CV to detect pets that violate the forbidden zone boundaries using sockets for the interim demo. We hope to integrate parts of the Machine Learning into the project for the interim demo as Max said he is feeling better enough to help with integrating.



Posted on

Brandon’s Status Report for 3/25/2023

This week I worked on creating the frontend and backend of a user designating the forbidden zone for a pet. I learned how to use state and props in React so that I could keep track of the state of each forbidden zone cell. For my forbidden zone example, I created a 10×10 grid (sizing will most likely be changed), and I keep track of the position in the x and y directions and if a user has clicked the cell (which represents a forbidden spot or not). Once a user submits their forbidden zone data, then it sent to the database that is managed by the Django REST framework. A picture of the frontend for the forbidden zone task is shown below.

I was hoping to looking into how to display a frontend for the pet activity logs using the heat map library described from last week, so I am currently behind. I hope to do this for next week and make sure everything is stable for the interim demo.

Like I mentioned above, I hope to create a frontend for displaying pet activity logs using a heat map. As well, I hope to make sure everything I currently have is working for the interim demo, and work on integrate some parts of my web application with other parts of the project.

Posted on

Brandon’s Status Report for 3/18/2023

This work, I worked on the ethics assignment, looked into how to deploy the machine learning aspect of the project onto an Amazon EC2 instance, and worked on implementing choosing forbidden zones as a user and showing pet activity logs to the user. As our team has been investigating where to run the machine learning, such as running it on the web server, I looked into how matching learning can be run on the Amazon EC2 server as I have the most experience with this platform. From my research, there seems to multiple examples of other machine learning models being deployed on Amazon EC2 using a deep learning amazon machine image, so I believe that this is a possible idea. For implementing pet activity logs as a heat map, I have found out that I can use a library called heatmap.js that will allow me to create heat maps based on data taken from OpenCV.

I am behind on work due to homework in other classes, but I will be making sure that all the core tasks for the web application are functional before the interim demo.

I hope to finish implementing forbidden zones and showing pet activity logs in the next week. As well, I hope to create a user interface design based on my user flow diagram I created.

Posted on

Brandon’s Status Report for 3/4/2023

During this week, I spent most of my time working on the design review document with my team. For example, we figured out the details for the testing, verification, and validation as this was one of the points that wasn’t fully addressed in the presentation. As well, I fixed the block diagram for the web application side so that it aligns more with what a block diagram should look like. In addition, I added a flow diagram of the user experience on the web application into the design review document. I have looked into how I can allow users to choose forbidden zones on the web application, but I have not implemented this yet.

I am currently behind schedule, but I will hope to make up the task of implementing the frontend and backend for a user choosing forbidden zones this week.

I hope to fully implement choosing forbidden zones on the frontend and storing this data on the backend. As well, I hope to start looking at how to display pet activity logs on the frontend using data that would be given by the Raspberry Pi.