Posted on

Rebecca’s Status Report for 2/25/2023

This past week started with writing the design review slides, and then some preparation to present in the early portion of the week (and also, a midterm). As such, I didn’t begin working on code until the end of the week. I have figured out how to run a variety of OpenCV built-in tracking algorithms on my own computer, on live video feed, with a hardcoded bounding box. See example here!

With regards to the CV coding side, I would say that I’m on schedule. With regards to the hardware rPi, I’m a bit behind, as we had hoped to order it by the end of this past week (but delayed, due to the ongoing debate about rPi versus Jetson). However, this doesn’t have any impact on our ability to draft the software, so I’m not overly concerned about the delay. The plan at this point is to get the rPi this week, before spring break. If we are unable to test it out before spring break, or have trouble working with it in initial tests, then this is something I’ll invest more time into over the break. This should put me comfortably back on schedule.

In terms of deliverables for next, first and foremost I plan to make sure we order the parts we need, and to begin to poke around with using them. For the OpenCV stuff, I have 3 goals: 1) get rid of the hardcoded bounding box in favor of determining them dynamically, via something like pixel differences; 2) explore/experiment a bit more to determine which tracking algorithm seems best for our project; and 3) figure out how it will interface with the ML side (how to determine when to request ID, and how to cutout the important part of the image).

 

Posted on

Max’s Status Report for 2/25/2023

Our group is finishing up some final design implementation choices. The current options do not effect me, so I have been free to continue my work without and hardware hiccups due to changing our main hardware. As such, this week has been primarily implementing the InceptionV4 architecture and setting up a functional dog/cat breed classifier. I am still working on this, but the work is on track.

As for schedule, I am back on track with where I wanted to be. Primary research is completed and implementation is starting. This week I was able to start implementing the InceptionV4 architecture and I am on track for that task.

Posted on

Brandon’s Status Report for 2/25/2023

I have set up the web application that uses React and Django and learned about how to get and send data between React and Django using Axios, which can send GET and POST requests between React and Django, and learned about the Django REST framework, which can take data from the POST requests and create objects based on the Django models to store in the database. Though I have some experience with React and Django, I had no experience with using Axios and the Django REST framework, which took a few hours of learning and debugging to learn the full capabilities of these tools. After learning these tools, I implemented allowing the user to upload images for pet classification. Specifically, I designed a frontend page that allows users to upload images and a Django model to store these images and the type of these images (png, jpeg, etc.). After the user uploads an image, a POST request is called using Axios that sends the image to the Django REST framework, which creates an object based on the Django model specified above that stores the images into the database.

Based on my schedule, I am currently behind on tasks for the web application side. The plan is to push back my tasks back a week as I have other work commitments this week and have been having technical issues with my computer recently, and there are no major tasks that other people are relying on me finishing at the moment so pushing back tasks by a week is okay.

The task I hope to work on this week is to test our ideas related to users choosing forbidden zones for a pet. I want to test both the frontend, specifically users choosing spots, which designate forbidden zones, on a grid overtop a 2D room image, and the backend, specifically storing this data in an array-like structure, implementation of these tasks. Though I am pushing back tasks by a week, I believe this is an important task to do before spring break as it is a core task of the project that other teammates will eventually rely on, and it is important to see what changes I will have to make based on testing our implementation ideas of the forbidden zones this week.


Posted on

Team Report For 2/25/2023

Our biggest risks at the moment are still pertaining to technology use. Specifically, we’ll need to learn a system that none of us have worked with before. And, we’re still slightly uncertain as to whether an rPi will have sufficient computational power for what we need. However at this point, I think the next move will simply be to commit to trying it out, as this is the hardware that we’d prefer for the project (over a more specialized but also more expensive piece of hardware like a Jetson).

The only change to the project that we’re considering at this point is to include a speaker with the system, and the option for the user to request to play a deterrent sound (e.g. dog whistle). This is because, as a use case, we think that reporting bad behavior would also be more useful if there was a way to discourage said behavior, especially if the animal is doing something potentially dangerous. Otherwise, the user will know that something is happening but be completely unable to stop it, which seems unideal. In terms of added costs, this would require us to also include a small and inexpensive speaker to interface with the raspberry pi – probably something we can get from the inventory.

Regarding our schedule, as there has still been some debate going on about whether an rPi would be sufficient for our needs, this hardware hasn’t been requested from inventory yet, which we had been hoping to do by the end of this past week. However, this should not be a major setback, as everything we are doing is in software and can still be developed separately from the rPi. The intention is to get the rPi before spring break, and potentially use that time to catch up on figuring out rPi 101 if we don’t have enough time to do so prior to break. For the web application side, Brandon will be pushing back tasks a week due to other work commitments and technical issues he has been having with his computer. He hopes to work on it this week making sure our ideas on implementing users choosing forbidden zones actually work, which we believe is a core task of the project that should be tested before spring break.

Regarding major achievements, on the Web Application side, Brandon has set up React integrated with Django. He had no experience with Axios and the Django REST framework, which are tools to send data between React and Django, so he took a few hours to learn how to use them. After learning these tools, he implemented allowing the user to upload images for pet classification.

Regarding any adjustments to team work assignment, the potential addition of the speaker will require Brandon to figure out how to communicate a ‘send sound request’ through the web app to the rPi. In terms of receiving that on the rPi side (and figuring out how to incorporate an rPi with a speaker), Rebecca and/or Max will be primarily responsible. 

Posted on

Brandon’s Status Report for 2/18/2023

I worked on setting up React integrated with Django. Previously I have used each framework by itself, but I have never used them together so it took some time and debugging to set this up. I looked at my previous implementation of a user uploading images on Django and thought about how to replicate this with React and Django, but I have not implemented this yet. As well, I worked on the Design Review Presentation. Specifically, I thought about the solution approach as it applies to the web application side of the project. I chose React and Django due to my previous experience with these frameworks and how they are the leading frameworks in frontend and backend respectively. One safety concern is related to the privacy of the user, specifically how users will be able to see a live video feed of their room. Hence, a malicious user may be able to see the live video feed of another user, which is an invasion of privacy. But, there are security protection tools supplied by Django that can address these concerns. I have made minor adjustments to the block diagram from last week for the web application side, and I will be finishing up the other points of the Design Review Presentation with my team within the next day. 

Currently, I am behind schedule due to the tests and homeworks of other classes. I will be working on how to use the React and Django frameworks together, and I will implement a user being able to upload pet images to the website to catch up on the schedule.

After I finish working on the tasks that I am behind on, I will be working on one of the core features of the website, which is allowing users to choose forbidden zones for a pet and figuring out how to store this data into the database.

Web Application Development (17-437) covers the engineering principles related to my part of the design process, specifically working with the backend using Django. I have basic experience with React, and I will be learning more about developing with React in the upcoming weeks.

Posted on

Rebecca’s Status Report for 2/18/2023

This week has been sort of a whirlwind of actually finalizing our design plans. For me, that mostly looked like plenty of research. Near the beginning of the week, I spent a while searching for ways that we might be able to configure this project to work with an FPGA to do the vision and ML algorithms (the best verdict I found, by the way, was to simplify the camera connection as much as possible by doing it as a SPI connection). However, this route was deemed infeasible at the faculty meeting on Wednesday, and so we went back to the original plan of using a Raspberry Pi. As such, I’ve spent the rest of the week seeking information/tutorials on how to use rPi’s in ways that are relevant to our project. I’ve also been working on the Design Review presentation.

Our schedule for this week was still primarily research on my end, so I would say that I’m still on schedule at this time. The intention is acquire the hardware and begin the actual  implementation process starting this coming week. In particular, my deliverables/goals for next week are to start working with the Raspberry Pi – something like being able to run a generic ‘Hello World!’ – and to start exploring the OpenCV library (on my laptop, not the rPi) to get a working video tracker in Python.

Regarding the engineering courses that pertain to my side of the design principles (Computer Vision and being the primary source for getting the rPi working), for the most part there aren’t any. I have some exposure to OpenCV through my Computer Vision course (which is technically in robotics, not engineering), but even that involved much more of learning the computer vision actual algorithms than learning how to use OpenCV, as one would expect. No other engineering courses I’ve taken are even remotely relevant. To compensate for my lack of background in rPi’s, I’ve been reading a lot of internet tutorials over the last few days.

Posted on

Max’s Status Report for 2/18/2023

Our group finally settled on using a Raspberry Pi rather than an FPGA or a Jetson as our hardware. This has meant I no longer need to research neural network implementation on FPGA, but we did receive useful advice in our weekly meeting as a result of our team’s research into this topic. However, this has given me more time to research the exact nature of our neural network. We are definitely going to be moving forward with a CNN. As for what extra layers we will be using on top of our architecture, this is still being determined and will most likely have to be fine tuned over the course of the semester. However, it currently looks like our transfer learning model will implement an initial convolutional layer, followed by 5 to 6 feature layers that will be trained via the user uploaded images of their pet(s).

As for schedule, I am back on track with where I wanted to be. Primary research is completed and implementation is starting. This week I want to start to get a functional transfer learning model as a starting point for testing and training.

Neural networks were covered in 15-301, Introduction to Machine Learning, which I took last year. I have done extra study on various forums looking at examples using the Inception architecture to become more familiar with this particular piece of additional software, but I am already comfortable with the area due to Intro to ML and Computational Neuroscience, which also covered neural networks.

Posted on

Team Status Report for 2/18/2023

Regarding risks, at this point our most significant risks still lie with the technology. Although we’ve been able to find internet tutorials that seem fairly adaptable to our project, this doesn’t change the fact that none of us have worked with a Raspberry Pi before, and so we cannot be certain how that will go until we get our hands on it and start trying. Note: we’re planning to request the part(s) from inventory early this coming week. In terms of managing these risks, we’ve done all we can to find relevant supporting materials online. And, should things not go well, the contingency plan would be  to reach out to course staff (TAs or faculty) sooner rather than later. 

Regarding design changes, at our faculty meeting this week the possibility of using a NVIDIA Jetson was discussed. However, after some team discussion we are still planning to move forward with a Raspberry Pi, as the Jetson is considerably more expensive and definitely outside the consumer price point we’re seeking with this project. We expect that this may cause issues with latency – the ML and CV will definitely not be running with the best possible frame rate – but it is our belief that we can still get the project suitably functional on this hardware. 

Regarding scheduling, at this point, no major schedule changes have occurred. This week has been about finishing up figuring out the details of how our implementation will work. Next week we plan to acquire the hardware and start working on more of the implementation itself. 

Regarding principles of engineering, science, and math:  One key consideration this week was price point. We want to engineer something that accomplishes our goal while remaining affordable. This is what informed our decision regarding our hardware, and pushed us towards using a Raspberry Pi over other more expensive alternatives. With regards to the ML classification, we have decided to implement a convolutional neural network. Neural networks are extremely common in image and speech recognition, and convolutional neural networks are particularly suited to image recognition due to their convolutional layer, a computation that reduces the dimensionality of image analysis without losing data. With the computer vision aspect, the aim is to reduce the ML workload by using simple pixel differences to identify when a moving component enters the frame, sending only that section to be identified only once, and then keeping track of it through the frame via OpenCV functionality after identification has been made. It is our hope that this will be more efficient than trying to run ML identification on every single frame. Information from the Raspberry Pi regarding movement summary and zone notifications will be sent to the user via a web application. With regards to web application development, we plan to use React on the frontend to display user options such as choosing forbidden zones for a pet, which will be implemented using a grid system overtop an image of the home, or request pet activity logs, which is done by organizing the data given by the Raspberry Pi into a heat map or time-sensitive graph, and use Django on the backend to store data related to these user options for each pet. One concern for the web application is privacy as live video feeds of homes can be displayed if requested by the user on the website, which we will address using the security protection features given by Django.

Posted on

Rebecca’s Status Report for 2/11/2023

In the last week I wrote our group’s proposal presentation (see the proposal section of the blog) and did some implementation related research. We also met as a group a few times to spend a few hours figuring out more specifics of our design – in particular, where the boundaries between our roles will lie and how they will interface – and also to discuss possible alterations to our proposal. At this point, our next major steps are to nail down our design as specifically as possible and put together our design presentation. Regarding scheduling, this point in our schedule is still dedicated primarily to research, and I’m happy with my progress on that front. I’ve had some success with figuring out possible ways to work with components we’re unfamiliar with.

Seeing other groups’ proposal presentations this past week was very illuminating. In light of the fact that several other groups have proposed to do some amount of video processing with an FPGA – something we had originally been interested in but were discouraged from early on – we are reconsidering this as an option and doing some research as to the feasibility for our project specifically. In particular, the concern is how we’ll be able to get data onto and off of the FPGA, especially image/video information.  Since we still need to be able to communicate over the internet with our web app, we are currently assuming that we will still need something like a raspberry pi to pass messages back and forth. With regards to communication between the FPGA and rPi, I’ve found an internet project that could be promising to pass wired signals between the two. It focus on simple signals, but I believe the system it  describes could be expanded. It also talks about doing this in parallel, which will definitely be something for us to take advantage of in order to achieve the highest possible frame rate. This is something I will look into further this week.

Posted on

Team Status Report for 2/11/2023

Regarding current risks, none of us have used a Raspberry Pi, and that is a primary piece of hardware in our current implementation. If we move to a solution that implements a Raspberry Pi alongside an FPGA, this may mitigate risks due to the lack of familiarity as we will be using the Raspberry Pi only for communication between the FPGA and the web application. However, there would be new challenges with figuring out how to incorporate the FPGA, specifically the best way to transfer data in and out of it and how to implement a neural network within the FPGA to speed up ML identification.

Regarding changes to our design, we are looking into the use of an FPGA as our main piece of computational hardware paired with another piece of hardware (currently a Raspberry Pi) for communication with the web application. If a 2-piece solution is used it will be more expensive but will significantly increase the computational power, decreasing video buffers and lowering time to classification and notification. This is of particular interest to us because Rebecca has experience working with FPGAs so a solution using an FPGA would allow us to fully utilize these skills. If this is not feasible, either due to cost or implementation complexity, then we can move back to our original plan using only a Raspberry Pi. Although with regards to cost, it is our belief that were this product to actually be marketed, the FPGA would be replaced with a bulk-manufactured digital chip which should cut consumer costs substantially. The FPGA is simply a much more convenient and flexible way to prototype the initial design. We have also specified more parts of our block diagram for the frontend and backend of the web application, as seen in Brandon’s individual report for this week.

Regarding any schedule change, because our current tasks center around researching different implementations for our final design implementation, there are no major changes to our schedule.

Regarding ethics, our project primarily includes considerations for inclusivity and personal privacy. There is a huge variety of homes that have pets, so one of our priorities is trying to keep our product as inclusive as possible with regards to price and ease of use. And personal privacy is of course something that we’re concerned about preserving, as our project requires setting up a camera in one’s home, so we want to minimize the amount of personal information that our system could potentially make more vulnerable.