This week I was able to set up the joint tracking from a computer camera. I utilized the Python libraries OpenCV and Mediapipe in order to complete this task. With that, I had to learn key features of the libraries, going through the documentation for each and understanding the syntax. After I had a better understanding of how to use the libraries, I went and coded up an algorithm that will naively display your computer’s web camera and capture the different landmarks and points on the player’s body. Once I had this ready, I wanted to test out the feasibility of our application to be played by a regular user, so I went to multiple different locations and ran the script. With that, I found that it is definitely feasible for the player to play from the comfort of their own camera, but the user needs to stand pretty far from the computer (i.e. the user needs a decent amount of space to actually play the game). This isn’t really a problem but is just something we need to consider.
After that, I worked with Rex to see how I should send the coordinates of the user to the Unity system to showcase the user’s joints within our game UI. I first had to figure out how to print out the coordinates of the 32 points that Mediapipe provides for us. After I did that, I organized that data into a JSON that Rex is able to actually interpret in his Unity game. We have already tested our capability to send data from a Python script to a Unity server, so now we have the capability to translate a real-life human’s moves to a character in the game.
In terms of our progress, I feel that we are on a good track. We now have a better understanding of what the capabilities are of our project and how we can go about completing it. We ran into some slight high-level setbacks when it came to switching our project from a game to an application for learning, but I believe we are moving in the right direction.
Next week, I hope to explore further into the algorithm we will use to detect differences between a user’s dance moves and a reference video. At this point, Danny, Rex, and I have worked on different aspects individually, so next week we will focus on incorporating everything we have together and taking the next steps to actually help people learn dance moves from a reference video.
