Posted on

Rebecca’s Status Report for 2/18/2023

This week has been sort of a whirlwind of actually finalizing our design plans. For me, that mostly looked like plenty of research. Near the beginning of the week, I spent a while searching for ways that we might be able to configure this project to work with an FPGA to do the vision and ML algorithms (the best verdict I found, by the way, was to simplify the camera connection as much as possible by doing it as a SPI connection). However, this route was deemed infeasible at the faculty meeting on Wednesday, and so we went back to the original plan of using a Raspberry Pi. As such, I’ve spent the rest of the week seeking information/tutorials on how to use rPi’s in ways that are relevant to our project. I’ve also been working on the Design Review presentation.

Our schedule for this week was still primarily research on my end, so I would say that I’m still on schedule at this time. The intention is acquire the hardware and begin the actual  implementation process starting this coming week. In particular, my deliverables/goals for next week are to start working with the Raspberry Pi – something like being able to run a generic ‘Hello World!’ – and to start exploring the OpenCV library (on my laptop, not the rPi) to get a working video tracker in Python.

Regarding the engineering courses that pertain to my side of the design principles (Computer Vision and being the primary source for getting the rPi working), for the most part there aren’t any. I have some exposure to OpenCV through my Computer Vision course (which is technically in robotics, not engineering), but even that involved much more of learning the computer vision actual algorithms than learning how to use OpenCV, as one would expect. No other engineering courses I’ve taken are even remotely relevant. To compensate for my lack of background in rPi’s, I’ve been reading a lot of internet tutorials over the last few days.

Posted on

Max’s Status Report for 2/18/2023

Our group finally settled on using a Raspberry Pi rather than an FPGA or a Jetson as our hardware. This has meant I no longer need to research neural network implementation on FPGA, but we did receive useful advice in our weekly meeting as a result of our team’s research into this topic. However, this has given me more time to research the exact nature of our neural network. We are definitely going to be moving forward with a CNN. As for what extra layers we will be using on top of our architecture, this is still being determined and will most likely have to be fine tuned over the course of the semester. However, it currently looks like our transfer learning model will implement an initial convolutional layer, followed by 5 to 6 feature layers that will be trained via the user uploaded images of their pet(s).

As for schedule, I am back on track with where I wanted to be. Primary research is completed and implementation is starting. This week I want to start to get a functional transfer learning model as a starting point for testing and training.

Neural networks were covered in 15-301, Introduction to Machine Learning, which I took last year. I have done extra study on various forums looking at examples using the Inception architecture to become more familiar with this particular piece of additional software, but I am already comfortable with the area due to Intro to ML and Computational Neuroscience, which also covered neural networks.

Posted on

Team Status Report for 2/18/2023

Regarding risks, at this point our most significant risks still lie with the technology. Although we’ve been able to find internet tutorials that seem fairly adaptable to our project, this doesn’t change the fact that none of us have worked with a Raspberry Pi before, and so we cannot be certain how that will go until we get our hands on it and start trying. Note: we’re planning to request the part(s) from inventory early this coming week. In terms of managing these risks, we’ve done all we can to find relevant supporting materials online. And, should things not go well, the contingency plan would be  to reach out to course staff (TAs or faculty) sooner rather than later. 

Regarding design changes, at our faculty meeting this week the possibility of using a NVIDIA Jetson was discussed. However, after some team discussion we are still planning to move forward with a Raspberry Pi, as the Jetson is considerably more expensive and definitely outside the consumer price point we’re seeking with this project. We expect that this may cause issues with latency – the ML and CV will definitely not be running with the best possible frame rate – but it is our belief that we can still get the project suitably functional on this hardware. 

Regarding scheduling, at this point, no major schedule changes have occurred. This week has been about finishing up figuring out the details of how our implementation will work. Next week we plan to acquire the hardware and start working on more of the implementation itself. 

Regarding principles of engineering, science, and math:  One key consideration this week was price point. We want to engineer something that accomplishes our goal while remaining affordable. This is what informed our decision regarding our hardware, and pushed us towards using a Raspberry Pi over other more expensive alternatives. With regards to the ML classification, we have decided to implement a convolutional neural network. Neural networks are extremely common in image and speech recognition, and convolutional neural networks are particularly suited to image recognition due to their convolutional layer, a computation that reduces the dimensionality of image analysis without losing data. With the computer vision aspect, the aim is to reduce the ML workload by using simple pixel differences to identify when a moving component enters the frame, sending only that section to be identified only once, and then keeping track of it through the frame via OpenCV functionality after identification has been made. It is our hope that this will be more efficient than trying to run ML identification on every single frame. Information from the Raspberry Pi regarding movement summary and zone notifications will be sent to the user via a web application. With regards to web application development, we plan to use React on the frontend to display user options such as choosing forbidden zones for a pet, which will be implemented using a grid system overtop an image of the home, or request pet activity logs, which is done by organizing the data given by the Raspberry Pi into a heat map or time-sensitive graph, and use Django on the backend to store data related to these user options for each pet. One concern for the web application is privacy as live video feeds of homes can be displayed if requested by the user on the website, which we will address using the security protection features given by Django.

Posted on

Rebecca’s Status Report for 2/11/2023

In the last week I wrote our group’s proposal presentation (see the proposal section of the blog) and did some implementation related research. We also met as a group a few times to spend a few hours figuring out more specifics of our design – in particular, where the boundaries between our roles will lie and how they will interface – and also to discuss possible alterations to our proposal. At this point, our next major steps are to nail down our design as specifically as possible and put together our design presentation. Regarding scheduling, this point in our schedule is still dedicated primarily to research, and I’m happy with my progress on that front. I’ve had some success with figuring out possible ways to work with components we’re unfamiliar with.

Seeing other groups’ proposal presentations this past week was very illuminating. In light of the fact that several other groups have proposed to do some amount of video processing with an FPGA – something we had originally been interested in but were discouraged from early on – we are reconsidering this as an option and doing some research as to the feasibility for our project specifically. In particular, the concern is how we’ll be able to get data onto and off of the FPGA, especially image/video information.  Since we still need to be able to communicate over the internet with our web app, we are currently assuming that we will still need something like a raspberry pi to pass messages back and forth. With regards to communication between the FPGA and rPi, I’ve found an internet project that could be promising to pass wired signals between the two. It focus on simple signals, but I believe the system it  describes could be expanded. It also talks about doing this in parallel, which will definitely be something for us to take advantage of in order to achieve the highest possible frame rate. This is something I will look into further this week.

Posted on

Team Status Report for 2/11/2023

Regarding current risks, none of us have used a Raspberry Pi, and that is a primary piece of hardware in our current implementation. If we move to a solution that implements a Raspberry Pi alongside an FPGA, this may mitigate risks due to the lack of familiarity as we will be using the Raspberry Pi only for communication between the FPGA and the web application. However, there would be new challenges with figuring out how to incorporate the FPGA, specifically the best way to transfer data in and out of it and how to implement a neural network within the FPGA to speed up ML identification.

Regarding changes to our design, we are looking into the use of an FPGA as our main piece of computational hardware paired with another piece of hardware (currently a Raspberry Pi) for communication with the web application. If a 2-piece solution is used it will be more expensive but will significantly increase the computational power, decreasing video buffers and lowering time to classification and notification. This is of particular interest to us because Rebecca has experience working with FPGAs so a solution using an FPGA would allow us to fully utilize these skills. If this is not feasible, either due to cost or implementation complexity, then we can move back to our original plan using only a Raspberry Pi. Although with regards to cost, it is our belief that were this product to actually be marketed, the FPGA would be replaced with a bulk-manufactured digital chip which should cut consumer costs substantially. The FPGA is simply a much more convenient and flexible way to prototype the initial design. We have also specified more parts of our block diagram for the frontend and backend of the web application, as seen in Brandon’s individual report for this week.

Regarding any schedule change, because our current tasks center around researching different implementations for our final design implementation, there are no major changes to our schedule.

Regarding ethics, our project primarily includes considerations for inclusivity and personal privacy. There is a huge variety of homes that have pets, so one of our priorities is trying to keep our product as inclusive as possible with regards to price and ease of use. And personal privacy is of course something that we’re concerned about preserving, as our project requires setting up a camera in one’s home, so we want to minimize the amount of personal information that our system could potentially make more vulnerable.

Posted on

Brandon’s Status Report for 2/11/2023

During class, I presented our proposal and did peer reviews of other proposal presentations. Outside of class, our team met to have an initial discussion of the design review for the project. We fleshed out the specific tasks that we each need to complete based on our role for the project, and we figured out how we could integrate all parts of the project together. Personally, this helped me have a greater understanding of what I specifically needed to do for the web application and what data I needed to send to or receive from the Raspberry Pi. After the meeting, I created a block diagram for the Web Application side of the project (as shown in the diagram below). Specifically, I listed the tasks that the user is able to complete on the frontend, and design Django models on the backend that would be used to store data in the database. Finally, I updated the schedule based on what we discussed in the meeting for the design review.

Based on the schedule we created, my progress is on schedule. For next week, the main priority is to finish the design review presentation, specifically finishing the block diagram and discussing testing, verification, and validation. In terms of the web application, I hope to set up a React application integrated with Django. Also, I hope to work on letting users upload images for pet classification, which I have prior experience doing.

 

Posted on

Max’s Status Report for 2/11/2023

Our group has had meetings to further flesh out our design implementation. One main topic that came up was the possibility of using an FPGA over a Raspberry Pi. This was inspired by witnessing other presentations that were performing computation on live video (rotoscoping, multiple object tracking) that is similar to our projects in terms of the level of computation on live video. Due to this, I have been researching the viability of an FPGA in our solution to improve classification speed of our classifier while still allowing easy video feed input to the FPGA and the ability to communicate with the web application. In addition, I have been researching existing cat/dog breed classifiers and working with InceptionV3 (our currently chosen ML architecture) to see if the current architecture will work.

Due to some major personal setbacks this week I am behind schedule, specifically on evaluating our current ML architecture choice. In addition, as of writing this, I am still unsure as to the viability of using an FPGA and am meeting with my group to further discuss this design change. The main priority for this week is to finish our design review presentation, which will first require a final decision as to whether or not we move forward with an FPGA, Raspberry Pi, or something else as our hardware. In addition, a complete evaluation of the InceptionV3 architecture as it compares to other existing architectures is to be completed soon. This should kick off work on implementing a animal classifier on the chosen architecture, which should be available this week.