Author: rexk

This week, I focused on several important optimizations and validation efforts for our Unity-based dance game project that uses MediaPipe and OpenCV. I worked on fixing a synchronization issue between the reference video and the reference avatar playback. Previously, the avatar was moving slightly out of sync with the video due to timing discrepancies between the video frame progression and the avatar motion updates. After adjusting the frame timestamp alignment logic and ensuring consistent pose data transmission timing, synchronization improved significantly. I also fine-tuned the mapping between the MediaPipe/OpenCV landmark detection and the Unity avatar animation, smoothing transitions and reducing jitter for a more natural movement appearance. Finally, I worked with the team on setting up a more comprehensive unit testing framework and began preliminary user testing to gather feedback on responsiveness and overall play experience.

Overall, we are on schedule relative to our project timeline. While the video-avatar synchronization issue initially caused a small delay, the fix was completed early enough in the week to avoid impacting later goals. Going off of what Peiyu and Tamal said, we have created a questionnaire and are testing with users to receive usability feedback. To ensure we stay on track, we plan to conduct additional rounds of user feedback and tuning for both the scoring system and gameplay responsiveness. No major risks have been identified at this stage, and our current progress provides a strong foundation for the next phase, which focuses on refining the scoring algorithms and preparing for a full playtest.

For testing, we implemented unit and system tests across the major components.
On the MediaPipe/OpenCV side, we performed the following unit tests:

1. Landmark Detection Consistency Tests: Fed known video sequences and verified that the same pose landmarks (e.g., elbow, knee) were consistently detected across frames.
2. Pose Smoothing Validation: Verified that the smoothing filter applied to landmarks reduced jitter without introducing significant additional lag.
3. Packet Loss Handling Tests: Simulated missing landmark frames and confirmed that the system handled them gracefully without crashing or sending corrupted data.

On the Unity gameplay side, we carried out the following unit and system tests:

1. Avatar Pose Mapping Tests: Verified that landmark coordinates were correctly mapped to avatar joints and stayed within normalized bounds as said in design reqs
2. Frame Rate Stability Tests: Ensured the gameplay maintained a smooth frame rate (30 FPS or higher) under normal and stressed system conditions
3. Scoring System Unit Tests: Tested the DTW (Dynamic Time Warping) score calculations to ensure robust behavior across small variations in player movement
4. End-to-End Latency Measurements: Measured the total time from live player movement to avatar motion on screen, maintaining a latency under 100 ms as required in design reqs

From the analysis of these tests, we found that implementing lightweight interpolation during landmark transmission greatly improved avatar smoothness without significantly impacting latency. We also shifted the reference avatar update logic from Unity’s Update() phase to LateUpdate(), which eliminated subtle drift issues during longer play sessions.

This week, I began by implementing more key features and refactoring critical components, as a part of the integration phase of our project. I modified our pose receiving to properly handle CombinedData, which now includes both raw poseData and real-time feedback from the dynamic time warping (DTW) algorithm. This integration required careful coordination with the updated pose_sender.py script, where I also addressed performance issues with regards to a laggy webcam input. Specifically, I optimized the DTW algorithm by offloading computations to a separate thread, reducing webcam lag and improving responsiveness. Additionally, I implemented a new character skin feature compatible with Danny’s pose_sender, allowing for a more customized and engaging user experience.

Progress is mostly on schedule for the integration part. I plan to spend additional hours refining the feedback visualization and testing latency under different system loads. In the coming week, my goal is to complete the UX feature that highlights which body parts are incorrectly matched in real-time during a dance session. This will significantly enhance usability and user learning by making corrections more intuitive and immediate for the final demo as well.

Now that core modules are functioning, I’ve begun transitioning into the verification and validation phase. Planned tests include unit testing each communication component (pose sender and receivers), integration testing across the DTW thread optimization, and utilizing several short dances for testing accuracy of the real-time feedback. To verify design effectiveness, I will analyze frame-by-frame comparisons of live poses against reference poses as well as the DTW algorithm’s window. This would allow me to check timing accuracy, body part correlation, and response latency using python timers in the code; seeing that they adhere to what we outlined in the use-case requirements with regards to timing metrics. I also plan to evaluate user interaction with the feedback system via usability testing in order to see how viable the final demo can be.

This week, I focused on optimizing the two avatars for the UI in our Unity. Specifically, I implemented a parallel processing approach where one avatar receives pose information (in a json form) from a pre-recorded reference video, while the other avatar receives pose data from live capture, parsed through JSON files. Ensuring that these two avatars execute smoothly and simultaneously, this allows us to effectively compare live performances against reference poses in real-time for the user to see what moves they should be doing. The UI also now shows what the CV is actually trying to capture as well. Additionally, I collaborated with my group members to test various comparison algorithms for evaluating the similarity between the reference and live poses. After thorough experimentation, we made a lot of progress with Dynamic Time Warping (DTW) due to its ability to handle temporal variations effectively, and we feel like this resolves the problem regarding the frame by frame misalignment when we do comparison. So, we integrated DTW into our existing ML pipeline, ensuring compatibility with the data structures we are working with. Screenshots of the UI with the two avatars running in parallel, and CV output is shown below.

Left avatar is reference video avatar, right avatar is live input.

Progress on the project is generally on schedule. While optimizing the avatar processing in parallel took slightly longer than anticipated due to synchronization challenges, the integration of the DTW algorithm proceeded decently once we established the data pipeline. If necessary, I will allocate additional hours next week to refine the comparison algorithm and improve the UI feedback for the player.

Next week, I plan on enhancing the feedback for the player. This will involve enhancing the UI to provide more intuitive feedback for the user when their pose deviates from the desired reference pose. Additionally, I aim to fine-tune the DTW implementation to improve accuracy and responsiveness. By the end of the week, the goal is to have a fully functional feedback system that clearly indicates which body parts need adjustment.

This week, I improved the game’s UI/UX pipeline to facilitate smooth selection of reference .mp4 videos. Although initial implementation was partially completed last week, several bugs affecting the UI/UX integration were identified and resolved this week. Users can now intuitively pick and load a video directly from the game interface, simplifying the setup process and enhancing the overall user experience. Furthermore, the video analysis module was extended to handle selected reference videos robustly, effectively translating video movements into coordinates used by the avatar. This enhancement enables accurate real-time performance comparison, seamlessly integrating both live capture and pre-recorded video data.

This is Danny in a Pre-recorded .mp4 (reference mp4) – NOT live capture Additionally, I successfully optimized the avatar‘s leg recreation for our Unity-based OpenCV MediaPipe dance comparison game. Previously, the avatar’s leg movements experienced slight jitter and occasional lagging frames, making the visual representation less smooth as I mentioned in last week’s report. By refining the landmark smoothing algorithm and employing interpolation techniques between key frames, the avatar’s leg animations now  follow the user’s movements better, significantly enhancing overall realism and responsiveness. As a result, the visual feedback loop maintains an ideal frame rate, consistently hovering around 30 fps, matching our outlined design goals.

Currently, our progress aligns well with our original timeline. Next week, I plan to focus on optimizing and integrating the comparison algorithm further alongside Danny and Akul. Our goal is to implement more sophisticated analytical metrics to assess player accuracy comprehensively. Deliverables targeted for completion include a refined comparison algorithm fully integrated into our Unity game pipeline, rigorous testing, and initial documentation outlining the improved analytic metrics.

This week, I made progress by most importantly getting the real time and reference .mp4 video coordinates working with the avatar. I also spent time optimizing the physics of joint movements and improving the accuracy of how OpenCV MediaPipe’s 2D coordinates are mapped onto the 3D avatar. This was a crucial step in making the motion tracking system more precise and realistic, ensuring that the avatar’s movements correctly reflect the detected body positions. Additionally, I worked on expanding the GUI functionality within Unity, specifically implementing the ability to select a reference .mp4 video for analysis. This feature allows users to load pre-recorded videos, which the system processes by extracting JSON-based pose data derived from the .mp4 file. As a result, the dance coach can now analyze both live webcam input and pre-recorded dance performances, significantly enhancing its usability as mentioned before. I have attached a real-time demo video below to showcase the system’s current capabilities in tracking and analyzing movements. Debugging and refining the motion tracking pipeline took considerable effort, but this milestone was essential to ensuring the system’s core functionality is robust and scalable.

(THIS LINK TO DEMO VIDEO EXPIRES ON MONDAY 3/15, please let me know if new link needed)

I am on track with the project timeline, as this was a major development step that greatly improves the system’s versatility. I primarily want to focus on refining the coordinate transformations, adjusting physics-based joint movements for smoother tracking, and enhancing the UI experience. Debugging the .mp4 processing workflow and ensuring proper synchronization between the extracted JSON pose data and Unity’s animation system were also key challenges that I successfully addressed. Looking ahead, my goal for the upcoming week is to refine the UI pipeline further so that the application becomes a polished, standalone application. This includes improving the user interface for seamless video selection, enhancing the visualization of movement analysis, and optimizing performance for smooth real-time feedback. With these improvements, the project is on a solid path toward completion, and I am confident in achieving the remaining milestones on schedule.

This week, I spent a lot of time refining the dance coach’s joint rotations and movements in Unity, making sure they feel as natural and responsive as possible which involves using physics/physiology and rotations. One of the focuses this week was adding logic to recolor the avatar’s mesh based on movement accuracy, giving users clear visual feedback on which parts of their body need adjustment. I also worked on integrating the comparison algorithm with Danny and Akul, the algorithms which evaluates the user’s pose against the reference movements. A major challenge was optimizing the frame rate while ensuring that the physics and physiological equations accurately represent real-world motion. It took a lot of trial and error to fine-tune the balance between performance and accuracy, but it’s starting to come together. I collaborated closely to test and debug these changes, ensuring that it works correctly for basic movements.

Overall, I’d say progress is on schedule, but some of the optimization work took longer than expected. The biggest slowdown was making sure the calculations didn’t introduce lag while still maintaining accurate movement tracking. I also believe that there is more improvement to be made on the rotations of some of the joints, especially the neck to model the movement more accurately. To stay on track, I plan to refine the physics model further and improve computational efficiency so the system runs smoothly even with more complex movements. Next week, I hope to finalize the avatar recoloring mechanism, refine movement accuracy detection, and conduct more extensive testing with a wider range of dance poses. The goal is to make the feedback system more intuitive and responsive before moving on to more advanced features.

Attached below are the demo videos for how the dynamic CV to unity avatar is right now, the physics movements will need to be further tweaked for advanced movement  (Note: speeds are not the same for both GIFs) 

This week, I focused on improving the natural movement of the character in our CV-based dance coach. By manually adjusting the targets that each joint should track, including arms, legs, and neck, I was able to refine the character’s motion to make it feel more fluid and realistic. The attached video demonstrates these improvements, showing how the avatar’s motion now closely mirrors user input in a more natural way given 3 different dance test poses to move to/from. Additionally, I worked on optimizing our UDP network communication, ensuring that packet transmission and reception are stable with good throughput. One major challenge I encountered is normalizing the coordinate data from MediaPipe’s OpenCV-based pose tracking to Unity’s avatar system, as they operate on different scales. To better understand how real-time feedback should be structured, I also consulted 3 friends who enjoy TikTok dancing and learning these trendy dances on the fly. Based on their input, I confirmed that a Just Dance-style single-frame feedback system is intuitive, fun, and easy to engage with, making it an ideal approach for our project, so I’ve decided with our teammates that this should be what we look for in our design requirements as well.

Currently, our progress is on track with our proposed schedule, as we are actively working on synchronizing user input with the reference video using CV techniques. We have also taken feedback from our Design Proposal Presentation this week to incorporate a more refined design requirement for the feedback part of our system. Moving forward, my next major focus will be on normalizing the MediaPipe pose estimation coordinates to correctly align with Unity’s character rig. This is a crucial step in ensuring accurate comparisons between the user’s movements and the reference video. Additionally, this will augment our upcoming tasks of making a real-time feedback system. If any delays arise on my part, I will dedicate additional time to debugging and adjusting the coordinate mappings to keep us aligned with our project timeline.

This week, I focused on synchronizing the inputs from Python into Unity, specifically working on the networking side of our project. I explored different methods to send data efficiently, considering various types of networks and sockets. Initially, I looked into using TCP for reliability, but I quickly realized that for a real-time application like ours that specializes in timing, UDP would be a better fit. Given that our game runs on a local device, the risk of packet loss is minimal, and UDP allows for lower latency, which is crucial for smooth gameplay. I tested different approaches for handling packets, ensuring that the data sent from the CV model reached Unity without excessive delay. The tests confirmed that UDP provided stable packet delivery in our setup, making it the optimal choice. Alongside networking, I spent time analyzing the outputs from our computer vision model, experimenting with both Mediapipe and OpenPose. OpenPose provided detailed joint tracking, but after testing both models, we found that Mediapipe’s output aligned better with our Unity implementation. It provided smoother and more reliable keypoint data, making it easier to work with for our in-game character movement.

After analyzing the CV model’s output, I started integrating the movement data into the character rig I set up last week. While mapping the positional data to the character, I noticed that the movement looked unnatural and stiff. To fix this, I began experimenting with different animation constraints in Unity, such as Two-Bone Inverse Kinematics, Multi-Aim Constraints, Damped Transform, and Rotation Constraints. I tested how each constraint affected the character’s fluidity and responsiveness when applied to different joints. My goal is to make the character’s motion feel more natural while staying true to the user’s real movements. Currently, I am iterating on these adjustments and expect to refine them further next week. In terms of progress, I believe I am on schedule, but refining the animation may take some extra time. Next week, I plan to improve the character’s movement smoothness by fine-tuning constraint settings and experimenting with filtering techniques to reduce jitter in the CV data. I will also help refine the CV pipeline to ensure correct delivery of outputs to Unity, focusing on optimizing how frequently we sample and send the movement data to balance accuracy and performance.