Status Report 9/21/24

This week I presented the project proposal for our group.  It took some time to memorize the content of the slides as much as possible and prepare for any questions.

Post that, I spent some time this week researching our chosen video game engine, Unity. From that, I’ve been able to break toe game development into a few stages. One stage will be developing an environment for the characters to move in, which will eventually be moved towards being a boxing ring.  The other stage will be developing a character that has dynamically controllable limbs and a body to be used later for control through computer vision. The next stage will be collision detection and damage metrics tied towards that. The last stage will be the game UI development.

Things are currently on schedule but where the schedule will deviate is depending on if I get stuck on any of the stages. Unity development on making an environment will begin this week. If this goes well, I hope to begin on character development as well as a deliverable for next week.

 

Status Report 9/28/24

 

This week I worked on unity development. It started out with a lot of research on beginner methods for making a first person controller on unity. Many of the tutorials online were outdated and included packages which no longer are supported currently which made the process much more difficult than expected, but I was able to find a more recent and in-depth tutorial on some basics for making a first person controller in unity,

I started with downloading a free polygon character package from the unity store and properly importing that into my project as shown below.

After getting more familiar with all of the assets involved in this polygon character, I was moved onto making a simple environment for the character to move on as well as updating the cameras location to be at the head of the character

.

 

I was able to implement basic movements using keyboard controls through C# scripts. That can be shown here.  Everything is on schedule for now. I hope to move onto a custom and more in-depth control set that can take advantage of computer vision for movement to do an early integration test next week.

 

Status Report 10/5/24

 

This week unity progress was continued. I first worked on figuring out how animation in unity worked. This required the creation of some more C# scripts for various portions of animation and then testing to ensure the smoothness of the animation. The animation was done using already existing animations of the free character model. This was done to learn the basic ways of game character animating in Unity before moving on to the more in- depth custom model that will be needed. Some short videos of the progression of this animating can be seen in the following three: 1, 2, 3.

Following that I worked on trying to directly map an example media pipeline pose to the game character. To do this I needed to get a full bone rig onto the character model and then choose out of those the ones that are used by media pipe. After that, using another C# script, an attempt on mapping was done.  Below is an image showing a bit of this.

In that attempt, a few issues were identified that will make this mapping more difficult. One will be the need of converting from the coordinate system of media pipeline which can typically have the origin at something like the top left corner of its image to unity’s which has the center of the environment as its origin (0,0,0). From there the difference in dimensions will need to calculated to convert as well as the difference in character dimensions versus the actual person’s that media pipe captures.

The game logic development is slightly behind schedule as the computer vision integration is being heavily focused on currently, but that is not too much of a worry as that should be the simplest part to integrate into the video game. In the coming week, I hope to research some ways to be able to handle all of these things more easily to get the work done in a fast manner as well as handling making smooth animations between two different poses.

Status Report 10/19/24

A good amount of game logic has been developed. The first thing I worked on was collision detection to be able to tell when a character takes and deals damage to another. Because the medium of damage in a boxing game will be through the fists, I first implemented colliders connected to the middle finger bones of each of hand of the character and expanded them over the full hands as shown below.

From here, I set up a simple cube wall with colliders attached and tested if I could detect when the hands touch the wall. I set it to display in the debug log at the bottom in unity and was able to get that working properly.

From here, I moved onto setting up a simple collider on another character that will be the game AI. A simple health system was set up and attached to the AI character and stand in punching animations were set up to emulate the computer vision punches for the future. With all of this, the player character can now hit the AI and the AI’s health will be displayed lowering on a slider in the game view. A short video of this can be shown here.

From here on, in the upcoming week, I will set up a dynamic collider system for the bodies of the player and the AI so that the game is able to detect the difference between the arms, body, and head being hit and apply different damages amounts depending on what is hit. A slider for player HP will also be implemented. These things will be able to be completed quickly given my experience setting them up for different parts previously. Post these things, I will start to develop a simple AI for the enemy so that it can attack as well as a boxing ring environment. In addition to this, I will work with Taiming to integrate the computer vision controls into the game. The interim demo is in the middle of November. By the next status report, I hope to have an unpolished demo with computer vision controls and developed damage calculation (with sliders) for all parts of the body. We will be on schedule if these things are completed by then.

Status Report 10/26/24

More colliders have been added to the bodies of the characters and increased logic to allow for more dynamic damage detection depending on where the character is hit. This was done through the use of object tags in unity and a script that calculates damage based on object tags.  Images of the change in the the script and new colliders can be seen below.

In addition to that, there was collaboration between Taiming and I in trying to connect the computer vision aspect to the movement of the game character. Some simple issues arose due the non familiarity of either of us with C# scripts which are required to work with Unity but those were overcome quickly. A bigger issue was that the translation of pose coordinates from Mediapipe to connecting with a game character as opposed to sphere objects which have no length resulted in deformed movement of the character.  It is good that we are dealing with this now to remain on schedule for having it fixed by the end of the month. I hope to continue implementing any last bits of game logic as well as continue to work on the integration issue with Taiming over this week.

Status Report 11/2/24

I moved back to focusing on finishing up the full game logic. One thing that i spent a good amount of time on was modifying the threshold required for hit detection so that when multiple hits happen within a very small amount of time, due to a fist not being pulled back yet for example, only one hit is registered. Without correcting this, it could be the case that a single punch could fully take one down from full health to nothing.

We are currently moving into our slack time left behind to get a working demo before the 15th so we are behind schedule as far as that goes but still within the planned extra time that we have left.  By next week, I hope to have made the boxing ring environment as well as added the most remaining simple game logic elements needed for a more fleshed out demo.

Status Report 11/10/24

I worked on continued game logic and making the boxing ring environment. The boxing environment has built in colliders so that the players can not move through them. Creating the environment required the creation of materials for the different color schemes to emulate a boxing ring. The game logic is now in its final stages and the last steps are merging the computer vision controls with the game to be ready for the demo. An image of the boxing ring environment is shown below. This boxing ring is fully setup with colliders that will block the player from moving through them. The player is not able to glitch through or get stuck in the edges or walls of the boxing ring. With the colliders being set up as is, this allows for potential future added development such as physics that push back the player from the ropes of the boxing ring when the player collides into them similar to a real world boxing ring environment.

Status Report 11/17/24

Continuing on from last week, I worked more on implementing some of the enemy capabilities in the game. My goal was to be able to get the enemy to be able to use the same punch animations as the player and to be able to damage the player. I first attempted to naively copy the actions tree in the animator for the player onto the animator for the enemy. That did not work due to the lacking dependencies and many other components such as movement that were related to the punching animations for the player. To get around this issue, I decided to fully duplicate my prefab for the player and turn that into an enemy by disabling the movement components but not the punching animation components. Some issues with the damage detection system arose due to this but were resolved after debugging.

The current game has the capabilities to have the enemy punch using the same inputs as the player and damage the player as well. Work will be done in the future to separate the enemy input from the player inputs and automate the process.  Below here is an image showing first the view of the damaged player and enemy and then the console outputs showing that there is complete differentiation between the enemy hitting the player and vice versa. 
Tomorrow, there are plans with the group to meet up and work on integrating all parts together in preparation for the upcoming demo.

Status Report 11/30/24

New capabilities were added to the game in addition to integration testing. There was the addition of added logic to allow for the game to pause when either the player’s health hits 0 or the enemy’s health hits 0 with corresponding screens for both. The screens both come with a ‘Play Again’ Button which resets both the player’s hp and enemy’s hp to full and unpauses the game.

 

Another new capability that was added was AI behavior for the enemy to allow it to automatically punch. With this all basic game functionality has been implemented.

For integrated testing, some bug fixing was done to allow the unity game character to be properly controlled by computer vision. We were able to get that working and see success regarding the fully integrated product.

Regarding testing for the game, I learned through online research about unity’s build in statistic tab that allowed me to see the FPS of the game. In addition to that, i also had to learn about how to use both python and unity’s time functions in conjunction with UDP communication to be able to see the ping in ms between python and unity. Both of these were done through the use of publicly available online resources and trying to run it until it works.

Status Report 12/7/24

Majority of work spent this week was on further integrated testing as well as the final presentation, poster, and video. Regarding changes to game development, a bug was fixed that allowed the player to hit themselves. This bug was not noticed until integrated testing with the computer vision as the prebuilt animations never allowed for the hands to hit the own player’s body. This was fixed with a simple change to the script which checks the tag of the hand and makes sure it can only hit an enemy if its a player hand and vice versa.

Some test results for the video game included that on lower end laptops, we were able to achieve an FPS of 50 which is above our target of 40, but this number goes much higher with an FPS of over 200 being achieved on higher end desktops. In addition to that, a simple ping test was checked for the latency between our UDP communication from python towards Unity and we were able to see that it was 35 ms at the highest.

We are well on track to have everything ready for the demo with plans to work a bit more to have things how we want them.