Posted on

Rebecca’s Status Report for 4/29/2023

The very beginning of this week was  spent working on the final presentation slides on Sunday. I didn’t do anything for capstone during the middle of the week (other than attending presentations) as I had many assignments due in other classes that took priority. At the end of the week, I met up with Brandon and we put some time into getting the live video feed working – it has a reasonable framerate now, although often it goes white in between each frame, which creates an awful strobing effect. We’re still looking for ways to improve this.

I’m fairly on schedule at this point other than the RPi. I would’ve liked if we could have begun integration testing with the RPi by now, so this is a bit behind schedule, but it does at least seem to function ok on simple examples. The plan is to get 100% ready for integration tomorrow and Monday, so that we can integrate on Tuesday.

This coming week will therefore start with lots of time working with the RPi. Hopefully Brandon and I can integrate on Tuesday. The goal is to at least re-do the tests we’ve already done on laptops, now on the RPi, in time for the poster on Wednesday. Then the rest of the week will be debugging last features (mostly live video) and performing longer tests/under more realistic conditions.

Posted on

Rebecca’s Status Report for 4/22/2023

Between now and last post, I’ve spent much of my time for this class working with Brandon to integrate the CV & Web App portions. We’ve more or less finished heat maps and zone notifications, using requests as our form of communication. We’ve also begun work on getting a live video feed working, although that isn’t fully there yet. Outside of this, I’ve also begun working on the final presentation, and I have spent a bit of time working with extremely simple examples for the Raspberry Pi.

I am slightly behind where I had hoped to be with the RPi at this point, mostly since I would like to prove that it works with the camera and simple CV code before starting to try to integrate with that. I hope to accomplish this by the end of this weekend, which would put us on track to start including the RPi at the beginning-middle of this week.

In the coming week, I intend to 1) spend more time on the final presentation; 2) work on getting the system working on the RPi rather than laptop(s); 3) work with Brandon to get live video feed up and running. Any time left at the end of the week will hopefully be dedicated to running fuller/more situationally correct tests on the integrated system.

Posted on

Team Status Report for 4/22/2023

Risks and Mitigation:

We may have a hard time getting a reasonable frame rate on our live video stream, if we cannot find an efficient enough way to send these images. We plan to get it as fast as possible by compressing the data as much as is reasonable, and feel that a somewhat choppy frame rate (few frames a second) would still be acceptable to the project – the most important thing is that the user would be able to get an idea of what their pet is up to. We are still in the fairly early stages of trying to get this feature up and running, so the exact frame rate that we’ll be looking at is still very TBD.

We haven’t had a chance to hook the camera up to the RPi yet (ideally this will be tested by the end of the weekend) so there’s still a risk of mechanical failure with that. We will reach out ASAP if we are unable to troubleshoot the issue, and if it ends up being a flaw with the camera itself, we will order a new one if needed before the end of the week. (The backup backup if it all goes up in flames would be for the ‘final’ version of the project to remain on laptops).

Project Changes:

There aren’t really any changes from the last status report. We will still proceed with a more separated project as we will integrate CV and Web App  with an RPi for the primary hardware. Max will work with the Jetson for the ML side. There will be no changes in cost as the materials come from the ECE inventory.

Schedule Changes:

We will be planning to do heavier integration – particularly, including the RPi – with more realistic testing in the upcoming days, and we hope to get tasks done for each remaining system of the project individually during this week. This should put us on track for the final demo.

Posted on

Rebecca’s Status Report for 4/8/2023

The beginning of this week was a lot of time spent starting to integrate between CV and the Web App. Brandon and I worked to find a communication protocol (currently using sockets, later switching to requests) that would enable our pieces to send info, then got the notification pipeline working pretty much in full, though on a limited scale. We are now able to send an image from the camera to the web app, choose forbidden zones on that image, and relay that info back to the CV tracking. When an overlap into the forbidden zone is detected, a notification is sent back to the web app. Starting next week, we will be working on refining the communication (switching to requests) and getting activity logs roughly operational. After this, we should hopefully be able to flesh out the full functionality! I also aim to spend a bit of time tonight working with the RPi, so that hopefully we can get the project running on this platform around the end of next week or slightly thereafter.

With regards to our updated schedule, my progress is in line with what we have planned.

As already slightly mentioned above, the goals for next week will be to accomplish much more integration between the CV and Web App – change to using requests as our more official communication method, and get activity logs roughly working. If we have spare time, we may also begin to look into what it would take to send live video feed. Individually, I also hope to be able to run my tracking code on the RPi (the version without web app integration) by the end of the week (i.e. before Carnival).

Posted on

Rebecca’s Status Report for 4/1/2023

This has not been my most productive week to date, as I tend to spend a lot of time working on Saturday, but was quite busy today with personal business – most of my work time will be offloaded to Sunday, but thus I don’t have major progress to report in this blog. However, I’m content with the status of my portion. There’s one small bug I’m still trying to work out in the CV code, where sometimes when multiple bounding boxes overlap, the smaller bounding box is favored. Other than resolving this, I would say that the tracking and motion detection are working exactly as I had planned. I had hoped to post a video, but my computer kept crashing when I tried to edit the video and I didn’t have time before midnight to figure that out. Perhaps I will edit to add this later. Other than some tweaking of the core CV code, I have spent this week preparing to start integration for the interim demo. Specifically, I’ve written rough drafts of code for where the CV will connect to other pieces (i.e. forbidden zones and tracking activity), and done a bit of research into how we will go about connecting the web app to the CV when they’re hosted on two separate computers (likely sockets).

Other than that one last stubborn little bug, I would say that the core CV is done, which is nice and on schedule. However, we are moderately behind on really being able to dive into integration, including as it relates to the CV. The plan to remedy this is that 1) it will be our primary focus from here on out and 2) we will begin meeting in person, starting tomorrow.

In the next week, it will be full steam ahead on integration. First and foremost, the goal will be to get the CV communicating with the Web App for the interim demo on Wednesday. Depending on how Max is doing, we may try to incorporate the ML as well, though likely still keeping strictly on laptops for the time being. In the latter half of the week (once we can hopefully establish exactly how we will be moving forward with the project), I should be able to turn my focus back to the hardware we’ll be using, as this has slightly fallen on the back burner due to the uncertainty around our project.

Posted on

Rebecca’s Status Report for 3/25/2023

The CV code is pretty much done. All that is left is a bit more tweaking/debugging, and of course integration with the ML (which will be more so a team activity, so that will happen when Max is ready).

My progress with regards to the CV is pretty much on track, other than the little bit of tweaking I’d still like to do. However we are fairly behind on learning to work with the hardware, so that will be almost entirely the focus for this upcoming week. I plan to work on basic set-up for the Jetson, and hopefully we can accomplish enough this week to be back on schedule for fuller integration next week.

In the next week, I intend to focus on setting up the Jetson. I have found a nice step by step for general set up the developer kit which I hope will be doable with minimal struggles. I aspire to start on this early in the week (whenever we get the Jetson), so that we can discuss any major issues in mandatory lab on Wednesday. I also intend to finish the refinements on the CV, hopefully tonight or tomorrow. Then in the later part of the week, we can look at hooking up the camera with the Jetson.

Posted on

Team Status Report for 3/25/2023

  1. Risks and mitigation

As we are changing the primary hardware of our project – to an objectively more complex system that we also have no experience with – our main risk at this point will be in using the Jetson Nano. The mitigation for this risk will be to ask others with more experience when we have issues using the Jetson, i.e. course staff and/or other teams. 

  1. Project Changes (why & cost)

Using a Jetson Nano will require us to buy some form of internet adapter, which seems like it will use another ~$15-30 of budget for a relatively cheap adapter. It also changes our overall proposed product cost requirement from <$100 to around $180-200 in the current market (i.e., post pandemic), but since our primary hardware is coming from inventory this will have minimal impact on our $600 project budget. Speaker requirements may also change (or be scrapped altogether), but this is tbd as it’s lower on our list of priorities. The reason for this change is because the ML requires greater computational power than an RPi has, and a Jetson seems like an easier/safer option than trying to run the ML on the server.

  1. Schedule Changes

Everyone will be devoting a lot of time this week to learning the Jetson Nano system. In particular, we must figure out the general setup, how to connect it to the internet, and potentially look into how to connect it to a speaker. This puts us a bit behind on what we had hoped for in terms of starting to integrate our pieces with the hardware/each other, but if we can get the basics figured out this week then we should be on track to flesh it out more fully next week, and begin testing shortly thereafter.

Posted on

Rebecca’s Status Report for 3/18/2023

Aside from working on the ethics assignment, I’ve spent this week working on basic RPi setup and the mostly fleshed out CV algorithm. Basic RPi setup seems to have been successful, but I have yet to run it on CV-ish examples as we have yet to get a camera for the RPi, unfortunately. I have most of a ‘rough draft’ for the CV algorithm for our project, but am still debugging and filling in a couple blanks – mostly where different pieces (i.e. motion detection versus tracking versus overlap) meet up. I hope to have a full working draft by the end of tomorrow (Sunday 3/19).

My progress is slightly behind – I haven’t been able to work with Pi as fully as I had hoped by this point due to lacking a bit of necessary hardware, and the draft for the overall CV isn’t quite complete. However, if I can finish the CV draft by tomorrow this will put me at a good point. With regards to the Pi, it’s likely that we’ll go ahead with the testing we had hoped for this week regardless, so barring any significant problems that arise when we try to do this, that should also put me back on track for working with the hardware. We also intend to explore other routes for running the ML, so it’s likely that I’ll be doing additional work this week researching/experimenting with those.

In the next week, I plan to have a fully working draft of the CV portion that I can run on my laptop (by early in the week, so I can use the latter half of the week to refine it a bit). I also hope that we will be able to begin some integration testing of the ML algorithm on the Pi or other platforms, though this will also depend on where my team mates are at.

Posted on

Team Status Report 3/18/2023

  • Risks and mitigation

At this point, our primary risk relates to the ML, and our ability to run that. Since this will be the most computationally intensive part of our project, making sure that we have enough power to run that algorithm has been our main concern, so that we achieve the time requirements we set out for ourselves. At this point, we are experimenting with/considering 3 different methods to run our ML: 1. Running identification on the pi. This seems the riskiest, and is something we hope to firmly rule in/out by the end of this weekend or beginning of next week. 2. Running identification on the web server, i.e. completely off of the hardware. This is generally the largest unknown, and so it is something we will be doing much more research on in the next few days, but it would have the advantage of significantly reducing the complexity of what’s running on our hardware (and allow us to drive the theoretical consumer cost down even further). 3. Running identification on a Jetson Nano. This is the ‘safest option’, and so it is something that we intend to be prepared for by getting a Jetson from inventory ASAP and trying to learn its setup. However it also adds a bit of cost to the project, as well as a significant amount of theoretical consumer cost, which is why we still intend to look into options 1 and 2. We hope that looking into multiple options will allow us to mitigate the potential risk that one or several of the options proves infeasible. The primary work required to evaluate each involves setting up the platforms and having an operational CNN. 

  • Project Changes

We are considering where to run the ML, as detailed above. If we find that we can run it on the Pi, then this wouldn’t be a change. If we end up running it on the server, we will need to change the general structure of how the pi communicates with the web app, but otherwise this shouldn’t be a large change to the project as a whole. If we find that our only choice is to run it on a Jetson, this will be the largest change to the project – we can get a Nano from inventory without needing to use our budget, but would probably also have to acquire a wifi adapter of some sort, which would require a bit of budget. The current impact of the changes we are considering would mostly affect communication (if we intend to host an instance of the ML separate from the hardware) and obviously our hardware.

  • Schedule Changes

Trying to get the CV/ML code to run on hardware was always scheduled for this week, so this should stay as intended. There will be additional research needed at the beginning of this week to hopefully narrow down the feasibility. We also intend to acquire a Nano to experiment with ASAP. 

Posted on

Team Status Report for 3/4/2023

As per usual in the weekly reports, we would say that our biggest risks at the moment pertain to the hardware. The most imminent risk is that we have yet to really dive into trying to set up the rPi (other than reading internet tutorials/projects). As none of us have used an rPi before, there is the chance that this will not go smoothly. Our plan to mitigate this risk is to begin working with the rPi early this week (i.e. trying to run tutorial code), and if we are unable to accomplish what we need with it by the end of the week, we will reach out to course staff. The other, broader hardware risk is that we will find an rPi’s computational abilities insufficient for our project. Again, the plan to mitigate this risk is to begin trying out our algorithms on the rPi sooner rather than later, hopefully starting the week after next (3/20). If we find that it runs infeasibly slow on the rPi, our contingency plan is use a Jetson Nano.

At the moment, no major changes have been made to the existing design. We are considering acquiring a more secure domain for our web app, which would cost around $15, but otherwise shouldn’t affect the implementation or functionality of the project.

Our schedule has not changed at this time.

In terms of tools that we will need to learn for our project, nothing super new or unexpected has cropped up beyond what we’ve been assuming since the beginning of the project. We will all need to learn how to work with the rPi, including how to set up an Apache server to communicate with our web app. Rebecca will be learning how to work with OpenCV beyond what she’s previously done for classes, Max will be learning about working with Inception-v4, and Brandon will be learning more about Django and React.