Posted on

Rebecca’s Status Report for 3/4/2023

The week before spring break, pretty much all of the time (and energy) I had allotted for capstone went into the design review report. Other than playing around with the different built-in tracking algorithms a bit more, I did not work directly on the project. In the time since, over break, I have done some research into the concept of motion detection with OpenCV, and found several relevant projects that I intend to use as references. I also found more resources regarding running OpenCV on a Raspberry Pi specifically that I hope will be useful to get that up and running.

Per our schedule, I would say that my progress at this point is slightly behind. In an ideal world I would have started trying to work with the rPi over break, but I simply did not have the energy. The only tasks that I am ‘behind’ on at this point are beginning to learn the set-up of an rPi – basic ‘hello world’ type of stuff and getting tutorial code to run without alteration. To get back on track, I intend to begin working with the rPi early this coming week. If I am unable to get it running reasonably well by Wednesday, I will involve my group mates for aid (we can talk about it during mandatory lab time on Wednesday morning). And if we’re still struggling by Thursday or Friday, I will reach out to course staff, probably Kaashvi. So long as we can begin trying to get our code to run on the rPi by the end of the week, we will be on track.

In the next few days,  I need to start trying to set up the Raspberry Pi. In particular, I will be looking to successfully run some sort of tutorial code by Wednesday. I also aim to get a working ‘rough draft’ of the motion detection + tracking by the end of the week, as per the schedule. I feel I have a good grasp on what tracking and motion detection in OpenCV look like separately, so I believe it should be quite doable to combine these pieces with a few hours of work. I hope to start to get this code (tracking + detection) working on the rPi by the end of the week.

Posted on

Rebecca’s Status Report for 2/25/2023

This past week started with writing the design review slides, and then some preparation to present in the early portion of the week (and also, a midterm). As such, I didn’t begin working on code until the end of the week. I have figured out how to run a variety of OpenCV built-in tracking algorithms on my own computer, on live video feed, with a hardcoded bounding box. See example here!

With regards to the CV coding side, I would say that I’m on schedule. With regards to the hardware rPi, I’m a bit behind, as we had hoped to order it by the end of this past week (but delayed, due to the ongoing debate about rPi versus Jetson). However, this doesn’t have any impact on our ability to draft the software, so I’m not overly concerned about the delay. The plan at this point is to get the rPi this week, before spring break. If we are unable to test it out before spring break, or have trouble working with it in initial tests, then this is something I’ll invest more time into over the break. This should put me comfortably back on schedule.

In terms of deliverables for next, first and foremost I plan to make sure we order the parts we need, and to begin to poke around with using them. For the OpenCV stuff, I have 3 goals: 1) get rid of the hardcoded bounding box in favor of determining them dynamically, via something like pixel differences; 2) explore/experiment a bit more to determine which tracking algorithm seems best for our project; and 3) figure out how it will interface with the ML side (how to determine when to request ID, and how to cutout the important part of the image).

 

Posted on

Team Report For 2/25/2023

Our biggest risks at the moment are still pertaining to technology use. Specifically, we’ll need to learn a system that none of us have worked with before. And, we’re still slightly uncertain as to whether an rPi will have sufficient computational power for what we need. However at this point, I think the next move will simply be to commit to trying it out, as this is the hardware that we’d prefer for the project (over a more specialized but also more expensive piece of hardware like a Jetson).

The only change to the project that we’re considering at this point is to include a speaker with the system, and the option for the user to request to play a deterrent sound (e.g. dog whistle). This is because, as a use case, we think that reporting bad behavior would also be more useful if there was a way to discourage said behavior, especially if the animal is doing something potentially dangerous. Otherwise, the user will know that something is happening but be completely unable to stop it, which seems unideal. In terms of added costs, this would require us to also include a small and inexpensive speaker to interface with the raspberry pi – probably something we can get from the inventory.

Regarding our schedule, as there has still been some debate going on about whether an rPi would be sufficient for our needs, this hardware hasn’t been requested from inventory yet, which we had been hoping to do by the end of this past week. However, this should not be a major setback, as everything we are doing is in software and can still be developed separately from the rPi. The intention is to get the rPi before spring break, and potentially use that time to catch up on figuring out rPi 101 if we don’t have enough time to do so prior to break. For the web application side, Brandon will be pushing back tasks a week due to other work commitments and technical issues he has been having with his computer. He hopes to work on it this week making sure our ideas on implementing users choosing forbidden zones actually work, which we believe is a core task of the project that should be tested before spring break.

Regarding major achievements, on the Web Application side, Brandon has set up React integrated with Django. He had no experience with Axios and the Django REST framework, which are tools to send data between React and Django, so he took a few hours to learn how to use them. After learning these tools, he implemented allowing the user to upload images for pet classification.

Regarding any adjustments to team work assignment, the potential addition of the speaker will require Brandon to figure out how to communicate a ‘send sound request’ through the web app to the rPi. In terms of receiving that on the rPi side (and figuring out how to incorporate an rPi with a speaker), Rebecca and/or Max will be primarily responsible. 

Posted on

Rebecca’s Status Report for 2/18/2023

This week has been sort of a whirlwind of actually finalizing our design plans. For me, that mostly looked like plenty of research. Near the beginning of the week, I spent a while searching for ways that we might be able to configure this project to work with an FPGA to do the vision and ML algorithms (the best verdict I found, by the way, was to simplify the camera connection as much as possible by doing it as a SPI connection). However, this route was deemed infeasible at the faculty meeting on Wednesday, and so we went back to the original plan of using a Raspberry Pi. As such, I’ve spent the rest of the week seeking information/tutorials on how to use rPi’s in ways that are relevant to our project. I’ve also been working on the Design Review presentation.

Our schedule for this week was still primarily research on my end, so I would say that I’m still on schedule at this time. The intention is acquire the hardware and begin the actual  implementation process starting this coming week. In particular, my deliverables/goals for next week are to start working with the Raspberry Pi – something like being able to run a generic ‘Hello World!’ – and to start exploring the OpenCV library (on my laptop, not the rPi) to get a working video tracker in Python.

Regarding the engineering courses that pertain to my side of the design principles (Computer Vision and being the primary source for getting the rPi working), for the most part there aren’t any. I have some exposure to OpenCV through my Computer Vision course (which is technically in robotics, not engineering), but even that involved much more of learning the computer vision actual algorithms than learning how to use OpenCV, as one would expect. No other engineering courses I’ve taken are even remotely relevant. To compensate for my lack of background in rPi’s, I’ve been reading a lot of internet tutorials over the last few days.

Posted on

Team Status Report for 2/18/2023

Regarding risks, at this point our most significant risks still lie with the technology. Although we’ve been able to find internet tutorials that seem fairly adaptable to our project, this doesn’t change the fact that none of us have worked with a Raspberry Pi before, and so we cannot be certain how that will go until we get our hands on it and start trying. Note: we’re planning to request the part(s) from inventory early this coming week. In terms of managing these risks, we’ve done all we can to find relevant supporting materials online. And, should things not go well, the contingency plan would be  to reach out to course staff (TAs or faculty) sooner rather than later. 

Regarding design changes, at our faculty meeting this week the possibility of using a NVIDIA Jetson was discussed. However, after some team discussion we are still planning to move forward with a Raspberry Pi, as the Jetson is considerably more expensive and definitely outside the consumer price point we’re seeking with this project. We expect that this may cause issues with latency – the ML and CV will definitely not be running with the best possible frame rate – but it is our belief that we can still get the project suitably functional on this hardware. 

Regarding scheduling, at this point, no major schedule changes have occurred. This week has been about finishing up figuring out the details of how our implementation will work. Next week we plan to acquire the hardware and start working on more of the implementation itself. 

Regarding principles of engineering, science, and math:  One key consideration this week was price point. We want to engineer something that accomplishes our goal while remaining affordable. This is what informed our decision regarding our hardware, and pushed us towards using a Raspberry Pi over other more expensive alternatives. With regards to the ML classification, we have decided to implement a convolutional neural network. Neural networks are extremely common in image and speech recognition, and convolutional neural networks are particularly suited to image recognition due to their convolutional layer, a computation that reduces the dimensionality of image analysis without losing data. With the computer vision aspect, the aim is to reduce the ML workload by using simple pixel differences to identify when a moving component enters the frame, sending only that section to be identified only once, and then keeping track of it through the frame via OpenCV functionality after identification has been made. It is our hope that this will be more efficient than trying to run ML identification on every single frame. Information from the Raspberry Pi regarding movement summary and zone notifications will be sent to the user via a web application. With regards to web application development, we plan to use React on the frontend to display user options such as choosing forbidden zones for a pet, which will be implemented using a grid system overtop an image of the home, or request pet activity logs, which is done by organizing the data given by the Raspberry Pi into a heat map or time-sensitive graph, and use Django on the backend to store data related to these user options for each pet. One concern for the web application is privacy as live video feeds of homes can be displayed if requested by the user on the website, which we will address using the security protection features given by Django.

Posted on

Rebecca’s Status Report for 2/11/2023

In the last week I wrote our group’s proposal presentation (see the proposal section of the blog) and did some implementation related research. We also met as a group a few times to spend a few hours figuring out more specifics of our design – in particular, where the boundaries between our roles will lie and how they will interface – and also to discuss possible alterations to our proposal. At this point, our next major steps are to nail down our design as specifically as possible and put together our design presentation. Regarding scheduling, this point in our schedule is still dedicated primarily to research, and I’m happy with my progress on that front. I’ve had some success with figuring out possible ways to work with components we’re unfamiliar with.

Seeing other groups’ proposal presentations this past week was very illuminating. In light of the fact that several other groups have proposed to do some amount of video processing with an FPGA – something we had originally been interested in but were discouraged from early on – we are reconsidering this as an option and doing some research as to the feasibility for our project specifically. In particular, the concern is how we’ll be able to get data onto and off of the FPGA, especially image/video information.  Since we still need to be able to communicate over the internet with our web app, we are currently assuming that we will still need something like a raspberry pi to pass messages back and forth. With regards to communication between the FPGA and rPi, I’ve found an internet project that could be promising to pass wired signals between the two. It focus on simple signals, but I believe the system it  describes could be expanded. It also talks about doing this in parallel, which will definitely be something for us to take advantage of in order to achieve the highest possible frame rate. This is something I will look into further this week.