This week, I focused on optimizing the two avatars for the UI in our Unity. Specifically, I implemented a parallel processing approach where one avatar receives pose information (in a json form) from a pre-recorded reference video, while the other avatar receives pose data from live capture, parsed through JSON files. Ensuring that these two avatars execute smoothly and simultaneously, this allows us to effectively compare live performances against reference poses in real-time for the user to see what moves they should be doing. The UI also now shows what the CV is actually trying to capture as well. Additionally, I collaborated with my group members to test various comparison algorithms for evaluating the similarity between the reference and live poses. After thorough experimentation, we made a lot of progress with Dynamic Time Warping (DTW) due to its ability to handle temporal variations effectively, and we feel like this resolves the problem regarding the frame by frame misalignment when we do comparison. So, we integrated DTW into our existing ML pipeline, ensuring compatibility with the data structures we are working with. Screenshots of the UI with the two avatars running in parallel, and CV output is shown below.
Left avatar is reference video avatar, right avatar is live input.
Progress on the project is generally on schedule. While optimizing the avatar processing in parallel took slightly longer than anticipated due to synchronization challenges, the integration of the DTW algorithm proceeded decently once we established the data pipeline. If necessary, I will allocate additional hours next week to refine the comparison algorithm and improve the UI feedback for the player.
Next week, I plan on enhancing the feedback for the player. This will involve enhancing the UI to provide more intuitive feedback for the user when their pose deviates from the desired reference pose. Additionally, I aim to fine-tune the DTW implementation to improve accuracy and responsiveness. By the end of the week, the goal is to have a fully functional feedback system that clearly indicates which body parts need adjustment.