PoolTracker DDR

This week, the main thing that I worked on was setting up the entire tracking pipeline. This entails not just the tracking algorithm, but also the transfer of data between the backend system and the web application front end. To that end, I set up the infrastructure that would allow our backend to send information as a POST request to the front end. Furthermore, I worked on fine-tuning the tracking algorithm so that it can handle slightly tilted camera angles, more work needs to be done on this front.

For the next week (after spring break), I want to get tracking working. This would mean that I aim to have the backend track a swimmer over the course of multiple laps and send data periodically (when a lap is completed) to the front end. It follows that, we need to get some sort of collision detection with the lane edges working.

Another significant time commitment for this week was writing the mid-term report for our project.

The beginning part of this week revolved around the Design Presentation, which was the presentation I was assigned to give for this semester. I spent a couple hours creating/polishing the slides our team made last week, and another hour or so practicing. I feel that the presentation went fairly well and that we got good feedback which revealed weaknesses in our design.

From a technical perspective, I worked with Adithya on furthering progress on our web app. The first task we wanted to complete was to receive data from a remote Python script via POST requests. We ran into a lot of unexpected complications with this, and discovered that we were missing the database infrastructure needed to store what we had received. After troubleshooting this for quite some time, we cut our losses and decided to launch an entirely new EC2 instance, meaning we had to redo most of the work from last week.

Because of this setback, things have gotten pushed back just a little. However, there was a slack week we built in for spring break, so everything is still able to be on schedule.

Given that we have our design report due, I’m attending the reading discussion on Monday, and the ethics assignment is also due, I know that there will be limited time to work on this project. With the few hours that I do have left, I want to get OpenPose up and running on my laptop.

This week my effort was split in two main areas. Firstly, I worked towards furthering my tracking and detection algorithms. In particular, I worked towards denoising the image since the noice caused by all the splashing by the swimmers was causing our tracker to not track effectively. There are some further enhancements we need to make but this was definitely a step in the right direction.

Secondly, I worked with Jack to further improve the web app by giving it the ability to handle POST requests from clients. This functionality is imperative for our project since the tracker will be sending information to the web app periodically so that it can be displayed back to clients.

Additionally, 4+ hours this week were spent watching other groups’ progress in their projects. This was also useful since we could adapt the professors’ feedback for other groups to ours.

My progress is on track for the week. For the next week, I plan to continue working on both the tracking algorithm and the web app. In particular, I want to have swimmer tracking working before spring break. However, keeping in mind that I have to complete the design report and also the ethics assignment, both of which I expect will detract from the time I have to work on the actual project, I might have to use up some of the “slack” time that I had built into my schedule in the upcoming weeks.

Following our design presentation, we realized that there were some pretty severe issues with our current design. We spoke with a couple TAs and Professor Marios for recommendation of how to move forward, and have made some changes to our approach accordingly.

The biggest risk that we found in our design is with the stroke classification system. Last week, we had gotten rid of OpenPose for limbic detection, but this caused a lot of concern for the course staff. Because of this, we are working in parallel on limbic detection in OpenCV and in OpenPose. Even using OpenPose, there is a large chance that stroke classification will be an intractable task. We still plan to pursue stroke classification as a component of our project, but we must also be prepared to not have completely positive results.

Besides planning to use OpenPose again, we also changed the interface between the backend and our web app such that the backend sends updates at the completion of every length. In our presentation, we said that we would be sending data at the end of every “swim,” but we did not feel that this would be sufficient.

There are no changes to our schedule.

This week, our group showed a lack of progress, spawning from a combination of different issues. First off, we originally planned to use OpenPose, a CMU made limbic-movement detection software to track limbic movements of swimmers, and then classify one of five outputs (four standard strokes + other). For both macOS and Linux (Ubuntu), there were several problems building the software locally, mostly dealing with GPU related issues. As a consequence, we wasted about 1-2 weeks trying to set up a product we now know we’re not going to use. Secondly, a teammate requested the aquatics director to mount a camera on the top of the pool to record aerial swim footage, which was ultimately denied; this left us with no stable swimming footage as online videos’ camera angles change often. Fortunately we found some light.

In particular, these past few days, we learned of and agreed to use OpenCV, a computer vision language available in C++ and Python. Insofar, our team has found no issues pip installing it on Python3. Moreover, we read online and saw demos delineating OpenCV supports all the video processing we plan to do, everything from detecting pool edges, to segmenting the video via lane-lines, to motion detection and limbic movement tracking. We felt hopeful that at least we can build a decent project in a relatively easy programming language with one package. Additionally, we found some stable footage of swimming races that could be used as input.

Lastly, we discussed two more aspects to our project. First, one of our teammates was quite excited to spatially parallelize the code by segmenting the video via lane-line using GPU programming (CUDA), and examining each swimmer independently . Since buying any CUDA appliance is relatively expensive, we discussed whether to rent EC2 P2 instances for the GPU capability with the $600 allocated budget. Secondly, our team discussed how to deploy the project on a web-app, talking details of start/stop triggers to time swimmers, as well as detect when a swimmer is finished on the web-app itself. Fortunately, two of three team members have had web-app development experience, at least one of whom claimed the relative ease of deploying the app.

In short, we are hopeful of the project given recent developments, but still have a long way to go with many potential roadblocks that could arise.

This past week saw a lot of planning and refining of our project ideas. One of the biggest things that I worked on was hashing out the overall architecture of our project, more specifically the what different components of our project would be responsible for and the interfaces between them. I also worked with Adithya to deploy a basic outline of our web app on AWS (pictured below).

We are still on schedule, although our schedule has changed as outlined in our team status update for this week.

For next week, I hope to determine a clear plan for where we’ll get better aerial footage. I will also be working with Adithya to finish swimmer tracking and update the web app if time permits.

For this week, one of the big changes that we made was to switch from using OpenPose to OpenCV. With that in mind, I decided to take upon the challenge of testing OpenCV for our particular context. Attached is a result that I had for a stock footage from the 2016 olympic games.

Secondly, as is documented in our team’s weekly status report for this week, we decided to move around some things in our schedule and as a result, we deployed a basic web application that will serve as a base to our project. I worked on deploying this web app onto an AWS instance along with Jack.

Lastly, another milestone for this week is to create the presentation for the design reviews of next week. I contributed to this along with both my group mates.

For the next week, my personal milestones are to debug my tracking algorithm and implement a detection algorithm. The aim is to have detection and tracking working by the time I write this report next week.

The biggest threat to our project are issues with the swimming pool footage. We haven’t been able to find a pool that is willing to help us get perfectly aerial footage of a swim practice/competition, and there isn’t any footage available online that exactly meets our needs. Because of this, we have altered the scope of our project to require the client to manually outline the lanes in the pool that we should be analyzing, and will place additional constraints on the input video as the need arises.

One major change that we made to our project is abandoning OpenPose for stroke classification. We spent two entire class periods trying to get sample code working, but we couldn’t even get the library to compile. OpenPose is designed to be used on GPUs, and without these resources we decided it would be a more substantial project if we did our own limbic detection with OpenCV.

We are still on schedule to complete our project on time, but we have changed the order in which we are implementing things. Because we are still waiting to hear back about collecting better footage, we decided to push back motion detection in exchange for getting a web app up and running. Even though this wasn’t scheduled until the end of March, we have deployed a very basic outline of PoolTrackerDDR on an AWS instance.