This week, I focused on synchronizing the inputs from Python into Unity, specifically working on the networking side of our project. I explored different methods to send data efficiently, considering various types of networks and sockets. Initially, I looked into using TCP for reliability, but I quickly realized that for a real-time application like ours that specializes in timing, UDP would be a better fit. Given that our game runs on a local device, the risk of packet loss is minimal, and UDP allows for lower latency, which is crucial for smooth gameplay. I tested different approaches for handling packets, ensuring that the data sent from the CV model reached Unity without excessive delay. The tests confirmed that UDP provided stable packet delivery in our setup, making it the optimal choice. Alongside networking, I spent time analyzing the outputs from our computer vision model, experimenting with both Mediapipe and OpenPose. OpenPose provided detailed joint tracking, but after testing both models, we found that Mediapipe’s output aligned better with our Unity implementation. It provided smoother and more reliable keypoint data, making it easier to work with for our in-game character movement.
After analyzing the CV model’s output, I started integrating the movement data into the character rig I set up last week. While mapping the positional data to the character, I noticed that the movement looked unnatural and stiff. To fix this, I began experimenting with different animation constraints in Unity, such as Two-Bone Inverse Kinematics, Multi-Aim Constraints, Damped Transform, and Rotation Constraints. I tested how each constraint affected the character’s fluidity and responsiveness when applied to different joints. My goal is to make the character’s motion feel more natural while staying true to the user’s real movements. Currently, I am iterating on these adjustments and expect to refine them further next week. In terms of progress, I believe I am on schedule, but refining the animation may take some extra time. Next week, I plan to improve the character’s movement smoothness by fine-tuning constraint settings and experimenting with filtering techniques to reduce jitter in the CV data. I will also help refine the CV pipeline to ensure correct delivery of outputs to Unity, focusing on optimizing how frequently we sample and send the movement data to balance accuracy and performance.
