This week, I worked on getting us set up for the Interim Demo. After meeting with the professor on Monday, we explored how to improve our comparison algorithm. Before, we mostly just had a frame-to-frame comparison which had okay accuracy. With that, we explored how to use DTW, not just for post-processing, but also for real-time feedback. I first started by doing some more research into how other people have used DTW for video processing. I read a few papers on how others used DTW for feedback, and I was able to gain a better understanding of how the algorithm works and why it is suitable for our application.
We incorporated DTW by comparing shorter segments of the input video. The biggest pivot we had with this compared to what we originally planned to do was using DTW for the real-time feedback. We did this by comparing specific segments of the video at a time with DTW, rather than using the entire video. We did this because of the time-complexity of DTW – the longer the segment we choose (our original plan was to make the segment the whole video), the longer it will take, as it has a quadratic time complexity. In this case, we were able to segment the video into smaller chunks, allowing us to use DTW for real-time feedback.
Additionally, I worked on getting test data and planning how our actual interim demo will go. I considered the use-case application of our system, looking at actual dances we would want to replicate. One thing that I found that I personally enjoyed was learning how to do Fortnite dances, which are short and simple dances that can be generally difficult to master. We also played around with uploading these videos to our pipelined system, allowing us to test with other inputs.
Our progress is on schedule. We have two main components: the Unity side that displays both the reference video and the user video human figures to showcase in real-time how the user dances, and the comparison algorithm that actually showcases what parts of your dance moves correspond to a video, providing if you are dancing well or not. Next steps include integrating both of these aspects together for our total final demo in the next few weeks.