Category: Team Status Reports

Risks and Mitigation:

We may have a hard time getting a reasonable frame rate on our live video stream, if we cannot find an efficient enough way to send these images. We plan to get it as fast as possible by compressing the data as much as is reasonable, and feel that a somewhat choppy frame rate (few frames a second) would still be acceptable to the project – the most important thing is that the user would be able to get an idea of what their pet is up to. We are still in the fairly early stages of trying to get this feature up and running, so the exact frame rate that we’ll be looking at is still very TBD.

We haven’t had a chance to hook the camera up to the RPi yet (ideally this will be tested by the end of the weekend) so there’s still a risk of mechanical failure with that. We will reach out ASAP if we are unable to troubleshoot the issue, and if it ends up being a flaw with the camera itself, we will order a new one if needed before the end of the week. (The backup backup if it all goes up in flames would be for the ‘final’ version of the project to remain on laptops).

Project Changes:

There aren’t really any changes from the last status report. We will still proceed with a more separated project as we will integrate CV and Web App  with an RPi for the primary hardware. Max will work with the Jetson for the ML side. There will be no changes in cost as the materials come from the ECE inventory.

Schedule Changes:

We will be planning to do heavier integration – particularly, including the RPi – with more realistic testing in the upcoming days, and we hope to get tasks done for each remaining system of the project individually during this week. This should put us on track for the final demo.

1. Risks and mitigation

As we are changing the primary hardware of our project – to an objectively more complex system that we also have no experience with – our main risk at this point will be in using the Jetson Nano. The mitigation for this risk will be to ask others with more experience when we have issues using the Jetson, i.e. course staff and/or other teams. 

1. Project Changes (why & cost)

Using a Jetson Nano will require us to buy some form of internet adapter, which seems like it will use another ~$15-30 of budget for a relatively cheap adapter. It also changes our overall proposed product cost requirement from <$100 to around $180-200 in the current market (i.e., post pandemic), but since our primary hardware is coming from inventory this will have minimal impact on our $600 project budget. Speaker requirements may also change (or be scrapped altogether), but this is tbd as it’s lower on our list of priorities. The reason for this change is because the ML requires greater computational power than an RPi has, and a Jetson seems like an easier/safer option than trying to run the ML on the server.

1. Schedule Changes

Everyone will be devoting a lot of time this week to learning the Jetson Nano system. In particular, we must figure out the general setup, how to connect it to the internet, and potentially look into how to connect it to a speaker. This puts us a bit behind on what we had hoped for in terms of starting to integrate our pieces with the hardware/each other, but if we can get the basics figured out this week then we should be on track to flesh it out more fully next week, and begin testing shortly thereafter.

• Risks and mitigation

At this point, our primary risk relates to the ML, and our ability to run that. Since this will be the most computationally intensive part of our project, making sure that we have enough power to run that algorithm has been our main concern, so that we achieve the time requirements we set out for ourselves. At this point, we are experimenting with/considering 3 different methods to run our ML: 1. Running identification on the pi. This seems the riskiest, and is something we hope to firmly rule in/out by the end of this weekend or beginning of next week. 2. Running identification on the web server, i.e. completely off of the hardware. This is generally the largest unknown, and so it is something we will be doing much more research on in the next few days, but it would have the advantage of significantly reducing the complexity of what’s running on our hardware (and allow us to drive the theoretical consumer cost down even further). 3. Running identification on a Jetson Nano. This is the ‘safest option’, and so it is something that we intend to be prepared for by getting a Jetson from inventory ASAP and trying to learn its setup. However it also adds a bit of cost to the project, as well as a significant amount of theoretical consumer cost, which is why we still intend to look into options 1 and 2. We hope that looking into multiple options will allow us to mitigate the potential risk that one or several of the options proves infeasible. The primary work required to evaluate each involves setting up the platforms and having an operational CNN. 

• Project Changes

We are considering where to run the ML, as detailed above. If we find that we can run it on the Pi, then this wouldn’t be a change. If we end up running it on the server, we will need to change the general structure of how the pi communicates with the web app, but otherwise this shouldn’t be a large change to the project as a whole. If we find that our only choice is to run it on a Jetson, this will be the largest change to the project – we can get a Nano from inventory without needing to use our budget, but would probably also have to acquire a wifi adapter of some sort, which would require a bit of budget. The current impact of the changes we are considering would mostly affect communication (if we intend to host an instance of the ML separate from the hardware) and obviously our hardware.

• Schedule Changes

Trying to get the CV/ML code to run on hardware was always scheduled for this week, so this should stay as intended. There will be additional research needed at the beginning of this week to hopefully narrow down the feasibility. We also intend to acquire a Nano to experiment with ASAP. 

As per usual in the weekly reports, we would say that our biggest risks at the moment pertain to the hardware. The most imminent risk is that we have yet to really dive into trying to set up the rPi (other than reading internet tutorials/projects). As none of us have used an rPi before, there is the chance that this will not go smoothly. Our plan to mitigate this risk is to begin working with the rPi early this week (i.e. trying to run tutorial code), and if we are unable to accomplish what we need with it by the end of the week, we will reach out to course staff. The other, broader hardware risk is that we will find an rPi’s computational abilities insufficient for our project. Again, the plan to mitigate this risk is to begin trying out our algorithms on the rPi sooner rather than later, hopefully starting the week after next (3/20). If we find that it runs infeasibly slow on the rPi, our contingency plan is use a Jetson Nano.

At the moment, no major changes have been made to the existing design. We are considering acquiring a more secure domain for our web app, which would cost around $15, but otherwise shouldn’t affect the implementation or functionality of the project.

Our schedule has not changed at this time.

In terms of tools that we will need to learn for our project, nothing super new or unexpected has cropped up beyond what we’ve been assuming since the beginning of the project. We will all need to learn how to work with the rPi, including how to set up an Apache server to communicate with our web app. Rebecca will be learning how to work with OpenCV beyond what she’s previously done for classes, Max will be learning about working with Inception-v4, and Brandon will be learning more about Django and React.

Our biggest risks at the moment are still pertaining to technology use. Specifically, we’ll need to learn a system that none of us have worked with before. And, we’re still slightly uncertain as to whether an rPi will have sufficient computational power for what we need. However at this point, I think the next move will simply be to commit to trying it out, as this is the hardware that we’d prefer for the project (over a more specialized but also more expensive piece of hardware like a Jetson).

The only change to the project that we’re considering at this point is to include a speaker with the system, and the option for the user to request to play a deterrent sound (e.g. dog whistle). This is because, as a use case, we think that reporting bad behavior would also be more useful if there was a way to discourage said behavior, especially if the animal is doing something potentially dangerous. Otherwise, the user will know that something is happening but be completely unable to stop it, which seems unideal. In terms of added costs, this would require us to also include a small and inexpensive speaker to interface with the raspberry pi – probably something we can get from the inventory.

Regarding our schedule, as there has still been some debate going on about whether an rPi would be sufficient for our needs, this hardware hasn’t been requested from inventory yet, which we had been hoping to do by the end of this past week. However, this should not be a major setback, as everything we are doing is in software and can still be developed separately from the rPi. The intention is to get the rPi before spring break, and potentially use that time to catch up on figuring out rPi 101 if we don’t have enough time to do so prior to break. For the web application side, Brandon will be pushing back tasks a week due to other work commitments and technical issues he has been having with his computer. He hopes to work on it this week making sure our ideas on implementing users choosing forbidden zones actually work, which we believe is a core task of the project that should be tested before spring break.

Regarding major achievements, on the Web Application side, Brandon has set up React integrated with Django. He had no experience with Axios and the Django REST framework, which are tools to send data between React and Django, so he took a few hours to learn how to use them. After learning these tools, he implemented allowing the user to upload images for pet classification.

Regarding any adjustments to team work assignment, the potential addition of the speaker will require Brandon to figure out how to communicate a ‘send sound request’ through the web app to the rPi. In terms of receiving that on the rPi side (and figuring out how to incorporate an rPi with a speaker), Rebecca and/or Max will be primarily responsible. 

Regarding risks, at this point our most significant risks still lie with the technology. Although we’ve been able to find internet tutorials that seem fairly adaptable to our project, this doesn’t change the fact that none of us have worked with a Raspberry Pi before, and so we cannot be certain how that will go until we get our hands on it and start trying. Note: we’re planning to request the part(s) from inventory early this coming week. In terms of managing these risks, we’ve done all we can to find relevant supporting materials online. And, should things not go well, the contingency plan would be  to reach out to course staff (TAs or faculty) sooner rather than later. 

Regarding design changes, at our faculty meeting this week the possibility of using a NVIDIA Jetson was discussed. However, after some team discussion we are still planning to move forward with a Raspberry Pi, as the Jetson is considerably more expensive and definitely outside the consumer price point we’re seeking with this project. We expect that this may cause issues with latency – the ML and CV will definitely not be running with the best possible frame rate – but it is our belief that we can still get the project suitably functional on this hardware. 

Regarding scheduling, at this point, no major schedule changes have occurred. This week has been about finishing up figuring out the details of how our implementation will work. Next week we plan to acquire the hardware and start working on more of the implementation itself. 

Regarding principles of engineering, science, and math:  One key consideration this week was price point. We want to engineer something that accomplishes our goal while remaining affordable. This is what informed our decision regarding our hardware, and pushed us towards using a Raspberry Pi over other more expensive alternatives. With regards to the ML classification, we have decided to implement a convolutional neural network. Neural networks are extremely common in image and speech recognition, and convolutional neural networks are particularly suited to image recognition due to their convolutional layer, a computation that reduces the dimensionality of image analysis without losing data. With the computer vision aspect, the aim is to reduce the ML workload by using simple pixel differences to identify when a moving component enters the frame, sending only that section to be identified only once, and then keeping track of it through the frame via OpenCV functionality after identification has been made. It is our hope that this will be more efficient than trying to run ML identification on every single frame. Information from the Raspberry Pi regarding movement summary and zone notifications will be sent to the user via a web application. With regards to web application development, we plan to use React on the frontend to display user options such as choosing forbidden zones for a pet, which will be implemented using a grid system overtop an image of the home, or request pet activity logs, which is done by organizing the data given by the Raspberry Pi into a heat map or time-sensitive graph, and use Django on the backend to store data related to these user options for each pet. One concern for the web application is privacy as live video feeds of homes can be displayed if requested by the user on the website, which we will address using the security protection features given by Django.