Team C2: danCe-V

Risk Management:

Risk: Comparison algorithm not matching up frame by frame

Mitigation Strategy/Contingency plan: We will attempt to implement a change to our algorithm that takes into account a constant delay between the user and the reference video. If correctly implemented, this will allow the user to not start precisely at the same time as the reference video and still receive accurate feedback. If we are unable to implement this solution, we will incorporate warnings and counters to make sure the users know when to correctly start dancing so that their footage is matched up with the reference video

Risk: Color based feedback not meeting user expectations – continued risk

Mitigation Strategy/Contingency plan: We plan to break down our Unity mesh into multiple parts to improve the visual appeal of the feedback coloring, so that users can more immediately understand what they need to do to correct a mistake. We also plan to incorporate a comprehensive user guide to help with the same purpose.

Design Changes:

There were no design changes this week. We have continued to execute our schedule.

This week I focused on developing the comparison algorithm. Now that we had the code to normalize the points based on different camera angles, we had the capability to create a more fleshed out comparison engine to see if two videos contain the same dance moves. 

I spent my time this week creating a script that will take in two videos (one a reference video, one a user video) and see if the videos match via frame-to-frame comparisons. In our actual final project, the second video will be replaced with real-time video processing, but just for testing’s sake I made it so I could upload two videos. I used two videos of my partner Danny who does the same dance moves at different angles from the camera and at some different times. Using these videos, I had to extract the landmarks, get the pose data, and normalize the data in case there were any differences in camera poses. After that, I parsed through the JSONs, trying to see if each of the JSONs at each comparable frame are similar enough. I then created a side-by-side comparison UI that allows us to tell which frames are similar, and which frames are different. The comparison is pretty good for the most part, but I did find that there were some false positives, so I modified the thresholds and it got better as well.

Overall, our progress seems to be on schedule. The next steps will be integrating this logic into the Unity side instead of just the server side code. Additionally, I will need to change the logic to take inputs from a webcam and a reference video instead of uploading two videos, but this should be trivial. Overall, the biggest thing will be to test our system more thoroughly with more data points and videos. Next week, we will work on testing the system more thoroughly as well as beginning to work on our DTW post-video analysis engine.

I couldn’t upload a less blurry picture due to maximum file upload size constraints so apologies for any blurriness in the following images.

Match

No Match

This week, I concentrated on enhancing our integration capabilities and optimizing our data analysis framework. I collaborated with Rex to troubleshoot and resolve challenges related to data synchronization between our computer vision system and Unity 3D visualization environment. This involved identifying bottlenecks in the data pipeline and implementing solutions to ensure smooth data flow between different components of our stack.

Additionally, I made significant progress on integrating advanced mathematical transformation techniques into our real-time processing framework. This work required careful consideration of performance implications and algorithm design to balance accuracy with computational efficiency. The optimized implementation calculates reference parameters once at initialization and applies these transformations to subsequent data points, rather than recalculating for each new data frame.

This architectural improvement from last week has yielded substantial performance gains and enhanced the robustness of our system when handling variations in input data. This is especially useful when we are transitioning into full real time processing. The visualization tools we’ve implemented have provided valuable insights that continue to guide our development efforts.

This week I focused on integrating my existing work with the 3D Unity framework that Rex has been working on as well as continuing to improve upon the Procrustes Analysis method. The unity visualization allowed me to get a better sense of how the procrustes analysis is working and how I could improve it.

Initially, I was running the normalization algorithm on every single frame and normalizing them to the reference frame. However, this presented a serious problem once we tested the algorithm in Unity. If the test and reference video are not the exact same length in terms of the number of frames, we would be normalizing frames that are not aligned at all. This means that we would have very little temporal distortion tolerance, which negates our premise of doing DTW analysis. It also greatly impacted processing time since a new rotational matrix needed to be computed every single frame.

To improve upon this, I changed the algorithm to calculate procrustes parameters only once based on frame 0, and apply the calculated parameters to each frame afterwards. This solution worked well and greatly improved our processing speed.

 

 

Team Status Report

Risk Management:

Risk: Cosine Similarity Algorithm not yielding satisfactory results on complex dances

Mitigation Strategy/Contingency plan: We will continue working on the algorithm to see if there are improvements to be made given how the CV algorithm processes landmark data. If the Cosine Similarity Method will not work properly, we will fall back to a simpler method using Euclidean distance and use that to generate immediate feedback.

Risk: Color based feedback not meeting user expectations

Mitigation Strategy/Contingency plan: We plan to break down our Unity mesh into multiple parts to improve the visual appeal of the feedback coloring, so that users can more immediately understand what they need to do to correct a mistake. We also plan to incorporate a comprehensive user guide to help with the same purpose.

Design Changes:

There were no design changes this week. We have continued to execute our schedule.

Part A:

 

DanCe-V addresses the global need for accessible and affordable dance education, particularly for individuals who lack access to professional dance instructors due to financial, geographic, or logistical constraints. Traditional dance lessons can be expensive and may not be available in rural regions. DanCe-V makes dance training more accessible, as anyone with an internet connection and a basic laptop is able to use our application. Additionally, the system supports self-paced learning, catering to individuals with varying schedules and learning speeds. This is particularly useful in today’s fast-paced world where flexibility in skill learning is becoming more and more important.

 

Furthermore, as the global fitness and wellness industry grows, DanCe-V aligns with the trend of digital fitness solutions that promote physical activity from home. The system also has potential applications in rehabilitation and movement therapy, offering value beyond just dance instruction. By supporting a variety of dance styles, DanCe-V can reach users across different cultures and backgrounds, reinforcing dance as a universal form of expression and exercise.

 

Part B:

One cultural factor to consider is that dance is deeply intertwined with cultural identity and tradition. DanCe-V recognizes the diversity of dance forms worldwide and aims to support various styles, with possibilities of learning classical Indian dance forms, Western ballroom, modern TikTok dances ballroom, traditional folk dances, and more. By allowing users to upload their own reference videos and not just including a constrained set of sample videos, the system ensures that people from different cultural backgrounds can engage with dance forms that are personally meaningful to them. Additionally, DanCe-V respects cultural attitudes toward dance and physical movement. Some cultures may have gender-specific dance norms or modesty considerations, and the system’s at-home training approach allows users to practice comfortably in a private setting.

Part C:

DanCe-V is an eco-friendly alternative to traditional dance education, reducing the need for transportation to dance studios and minimizing associated carbon emissions. By enabling users to practice from home, it decreases reliance on physical infrastructure such as studios, mirrors, and printed materials, contributing to a more sustainable learning model. Additionally, the system operates using a standard laptop webcam, eliminating the need for expensive motion capture hardware, which could involve materials with high environmental costs.

Furthermore, dance is a style of exercise that does not require extra materials, such as weights, treadmills, or sports equipment. By making dance accessible to a larger audience, DanCe-V can help reduce the production of these materials, which often have large, negative impacts on the environment.

Procrustes Analysis Normalization demo:

Cosine Similarity comparison results:

Over the past week, I focused on two main components: the design report and our single frame comparison. For the design report, I spent time developing the quantitative design requirements, the design trades, and the system implementation. With that, I conducted research into requirements for our design, finding specific reasons as to why we want to make any design decisions. For example, to mitigate latency within our system, we decided to choose only a certain subset of points in the MediaPipe output to decrease latency while maintaining accuracy in our points. I decided to go with just 17 points, as many of the points crowd at the user’s head and toes which isn’t necessary for our specific use case. Additionally, we had an idea of how we would implement our system, but I spent time creating block diagrams to put all of our thoughts together for each aspect of the system. Consequently, throughout the rest of the semester, we will have these diagrams to refer to and continue to adapt if we make any changes, so both us and viewers can better understand our system. For the design trade study, I focused on making sure that all of our decisions were fully intentional in terms of the algorithms/libraries/protocols that we were using. I explored tradeoffs between these aspects and provided concrete reasoning as to why we chose one or the other. 

This week, we also made the goal to get a working MVP of the single frame comparison, where we can take a user input and a reference video to see whether or not their dances are similar, when doing a frame-to-frame comparison. We split up the work into normalizing the points, doing the actual comparison given normalized points, and providing the Unity output. My task was to compute whether or not a frame was similar or not based on two provided jsons that represent the user input and the reference input for that frame. 

The overall algorithm that I used was pretty simple. I first created a helper function to find the euclidean distance between two points in space, which will be given in the json inputs. Then, I loop through each of the points in the jsons, computing the distance between each of them. If the distance is less than a certain threshold (.05 for now), then the similarity for that point is true. I do this for each joint that we are computing, then if 80% of the joints are “similar” enough, then the overall output for that frame is true. These metrics I decided on are very arbitrary, and I think that we will first need to integrate the code fully and test these metrics to get a better idea of what we need for our final project.

Our progress is currently on schedule. By the end of spring break, we will have a MVP of our real-time feedback system, and once that is complete we will begin to work on our multi-frame analysis and overall integrating our system together.