PoolTracker DDR

As of now, the team has a functional web application that tracks swimmers’ splits and strokes. Moreover, it presents the additional functionality of delimiting stroke times on a continuous graph for users to see. Finally, our web-application also suggests workouts to swimmers, and allows clients to interface with it by picking a stroke and distance.

For our stroke classification, the ML portion of the project, we decided to opt for background subtraction + more pattern recognition to determine the appropriate stroke of our swimmer. The team found this approach more effective than the earlier approach of limb detection + relative coordinate input to a NN.

All in all, there is confidence that a decent capstone project will ensue, and we are looking forward to meet those ends…

This week, the team has shown quite a bit of progress on the web application. In particular, it has quite a bit of commendable functionalities such as graphing the splits of swimmers’, as well as suggesting workouts to a swimmer. Moreover, the user has the option of picking a workout, as well as a choosing stroke/distance on the web application.

Unfortunately, we have made little to no progress on the limb detection, and subsequent stroke classification. This is most likely a consequence of the fact that we have not set up OpenPose, the main limb detection library. It is very possible we may not finish the stroke classification and/or limb detection; however, we plan to document all attempts to perform this feat in the final report.

As our deadline comes closer, we have effectively finalized all we need to do to get a sufficient capstone up and running. First, we need to continue work on the web app. In particular, we need to garner data of the wall touches of respective swimmers while sending that data to the web application, which will be plotted on a front-end UI.

Second, we need to work on the actual stroke classification. For this task, we need to input stroke information about respective swimmers into a NN, and then classify one of four strokes. The challenge with this task is figuring out how to do the limb detection itself, which has proven to be very difficult up until this point. Once we have the limb detection, though,  we can proceed to inputting relative positions of joints to the NN, and training our model.

This week, we made some more substantive progress on the actual website. Two of our capstone members have gotten the server to accept real time input from the swimmer, and output a graph of splits on the localhost. We found this task quite challenging, but got it done nonetheless.

The one thing we need to implement is a stroke classifier. In order to do this, we plan to input the relative positions of body parts of a swimmer with respect to the bounding box for “k” frames. The best way to approach this would be to do limb detection, and then input to a NN those floating point values, relative position of body parts with respect to the body parts. Once that has been inputted, we can apply softmax on four output nodes to classify one of four different strokes, and train our model accordingly.

This week, we effectively finalized the last steps needed in order to be successful for this project. First, we concluded that it is important to collect tracking data from swimmers after they touched the wall, and display splits on a graph. We have gotten the website to monitor that data, and our time graph refreshes every 5 seconds or so. Adithya is working on the front-end to back-end integration, while Jack is working on the demo, as well as the web-app.

Next, we have to actually do the ML for the stroke classification. This involves “zooming” into the swimmer, and then tracking the moving swimmer. Based on the limbic motions of the swimmer, our software would ideally be able to classify a swimmer on one of four strokes.

 

This week, there was very limited time to work on our capstone projects due to other commitments, such as midterms. However, we did see some progress from the last few weeks. First, our group managed to set up limbic detection with OpenCV, which serves a key part of our project in stroke classification for pools. Additionally, we managed to segment the video via lanelines, again using OpenCV, and track each swimmer in each lane individually. Finally, we have not yet faced issues with the web-app component of our project, which is nice.

In short, given the recent developments in our work, I’m optimistic that we will have a feasible project up and running by the end of the semester.

This week, our group showed a lack of progress, spawning from a combination of different issues. First off, we originally planned to use OpenPose, a CMU made limbic-movement detection software to track limbic movements of swimmers, and then classify one of five outputs (four standard strokes + other). For both macOS and Linux (Ubuntu), there were several problems building the software locally, mostly dealing with GPU related issues. As a consequence, we wasted about 1-2 weeks trying to set up a product we now know we’re not going to use. Secondly, a teammate requested the aquatics director to mount a camera on the top of the pool to record aerial swim footage, which was ultimately denied; this left us with no stable swimming footage as online videos’ camera angles change often. Fortunately we found some light.

In particular, these past few days, we learned of and agreed to use OpenCV, a computer vision language available in C++ and Python. Insofar, our team has found no issues pip installing it on Python3. Moreover, we read online and saw demos delineating OpenCV supports all the video processing we plan to do, everything from detecting pool edges, to segmenting the video via lane-lines, to motion detection and limbic movement tracking. We felt hopeful that at least we can build a decent project in a relatively easy programming language with one package. Additionally, we found some stable footage of swimming races that could be used as input.

Lastly, we discussed two more aspects to our project. First, one of our teammates was quite excited to spatially parallelize the code by segmenting the video via lane-line using GPU programming (CUDA), and examining each swimmer independently . Since buying any CUDA appliance is relatively expensive, we discussed whether to rent EC2 P2 instances for the GPU capability with the $600 allocated budget. Secondly, our team discussed how to deploy the project on a web-app, talking details of start/stop triggers to time swimmers, as well as detect when a swimmer is finished on the web-app itself. Fortunately, two of three team members have had web-app development experience, at least one of whom claimed the relative ease of deploying the app.

In short, we are hopeful of the project given recent developments, but still have a long way to go with many potential roadblocks that could arise.