Category: Henry’s Status Reports

Last week, on Sunday and Monday, I spent a lot of time on getting the paddle integrated with the game.  I created an interface where the IMU sends me a swing type, power, and direction and I use those commands to trigger specific swing animations. It took a while to figure out why the rPi wasn’t accepting bluetooth connections.  Also on Sunday, I had to spend a while on the final presentation, where I pretty much did everything. On Thursday and Friday, I was working with William on swing detection and have a player’s swing reflect accurately on the game.  We were mainly working on when the IMU should start analyzing its data and how to signal the start and end of these analysis periods.  By the end of Friday though, we had made really good progress and were able to have a player swing the paddle and have the correct swing animation trigger on the game with pretty low latency.

Since we have all the components of the game integrated and working, I’m going to be working on making the game more realistic.  When William and I were working on Thursday and Friday, there were a few weird animations in my code that I need to fix and the ball and paddle speed were unrealistically slow.  I’m going to meet up with William on Wednesday after I make these changes to make sure that the IMU still has enough time to sample orientation/acceleration data and correctly identify the swing type/power/direction.  By Wednesday, I’m also going to have scorekeeping set up.   On Thursday, our entire group is going to meet up to do final tests and make adjustments to our game as we see necessary.

I have a lot of work to do next week, but I definitely have enough time to get it done.

This week I’ve been mainly working on adding important features to the game and wrestling with making the bluetooth connection between the rPi and VR headset more robust.  For the game, I worked on adjusting the paddle-ball interactions to give players a more realistic chance of hitting the ball, adding table-ball interactions so that the game can judge whether a player won a point if a ball hits a side twice, and implementing lateral tracking features in the paddle so that the paddle will always track the ball in the x direction.  That last features is implemented in case the Jetson cannot be integrated into the game.  For the bluetooth work, I changed the game so that instead of the rPi sending data to the server and having the server change the paddle orientation, the rPi will send its data to the VR headset, which contacts the server to change the location of the paddle.  I did this because the bluetooth connections between the rPi and our VR headsets connect much more frequently than connections between the rPi and our server.  Right now,  I am doing testing with the latency of the bluetooth connection and workin with William on thresholding to determine the swing type for the game.

Once everything is integrated and tested, we need to try to set up the Jetson and put the finishing touches on game, which includes score-tracking and  smoother animations.  Next week, I’m going to continue to work on integrating the rPi with the game and try to integrate the Jetson as well.

Progress this week is going as planned heading into next week.  I was able to create the graphics and animations for the final game.  This includes the incorporation the graphics for the paddle, table, and ball, animating and properly triggering the different types of swings, and setting the ball-paddle interactions that set the ball’s trajectory based on specific parameters of the hitter’s swing such as orientation and power.  I’m at the point where two players can play each other using the keyboard as an input.  Looking ahead, the biggest goal is to integrate the rPi and Jetson into the game through bluetooth. But some other things I’m expecting to do are refining the animations to look more smooth and realistic and fine-tuning swing thresholds to make gameplay more realistic.

The swing thresholds are something Will and I brainstormed this week.  We were trying to figure out how to deal with paddle-ball interactions and how we would figure out the ball’s direction and velocity after a collision, which was difficult because of the inaccurate velocity calculations from signal processing modules Will found online.  Another concern was with how the paddle movement would look when we integrated the rPi with the game and how laggy it could possibly look.  Our solution was to restrict sampling to a specific time interval after the ball hit the table and measure the orientation, acceleration, and position data to determine what kind of swing the player was performing.  It would then trigger an already animated swing corresponding to the determined swing type. We categorized swing data to swing types by creating thresholds for each piece of data and setting a threshold to a particular swing type.

I am optimistic that we are heading back on schedule because all I have to do next week is to try to integrate the rPi and Jetson data into the game and optimize certain aspects to make it more playable.  Tomorrow, I am going to be working on creating a bluetooth connection between the Jetson and the game so that on Monday, Logan can just load in his CV code and we can start testing the game in its totality.

This week, I was working with William to get the rPi connected with the game.  On Tuesday we were able to send rotational vectors between the rPi and a Mac and print those values out, but on Wednesday during our demo it didn’t work.  The reason why is because when I was importing the game onto my phone, I had to change a lot of build settings, which affected the bluetooth capabilities.  I also began to work on the final version of the graphics this week: incorporating a VR library that helps process the game to be viewed on a Google Cardboard, and designing the tables, paddles, and camera perspective for the final game.  The plan is to get this all implemented by Thursday or Friday.  What I am also going to be working on next week is setting up the Jetson Nano.  Logan hasn’t been able to work on it these past few weeks, so I’m going to take it since time is beginning to run out.  Hopefully by the end of next week I should be able to have the Jetson Nano set up and add bluetooth capabilities to it.  That way I can test the Bluetooth connection between the Jetson and the server.  I am behind schedule right now, but given the relatively light week I have upcoming, I am ready to put over 20 hours working on what I laid out above.

I started this week with the intention of integrating the Jetson into the game.  I spent a while first adding the Arduino Bluetooth library into our Unity code which should give us the ability for the Raspberry Pi and Jetson to send its data to the code.  However, this is where I ran into a few pretty big roadblocks.  The first one, is that when I was trying to connect to the Jetson, I realized that Logan had not set it up yet.  So, something that we have to work on next week is getting the Jetson’s OS set up and get the Bluetooth drivers and capabilities set up to see if it can connect with our game.  Another roadblock/uncertainty is whether the Arduino Library will work or not. The library is only compatible with certain Bluetooth modules, and I didn’t have anything to test with that had those modules.  The only testing I could really do was to see if my code would know if my Airpods were paired with my Mac or not, which it could.  So, on Monday, I am going to work with William to see if we can get the rPi connected.  If we can’t, I’m going to either have to buy a bluetooth module compatible with the Bluetooth library, or I have to write my own code using NET to connect with Bluetooth devices.

Because there were roadblocks with integrating devices into the game, I decided to refine my Unity code more.  The first thing I did was removing the GUI for determining whether we were running code as the server or as the client.  It’s unnecessary work for users and going to be extra work for us in the future since we would’ve had to add some VR pointing capabilities since our game doesn’t have a mouse and keyboard.  The second thing I did was getting the code ready for Raspberry Pi and Jetson inputs.  I added commented out code that sets the paddle location and orientation to these inputs and code that blocks until both pieces of data are received, so that the paddle input data is synchronized.

For my responsibilities, I am falling behind a little bit.  On the Gantt chart, it mentions that I have all of April to integrate the devices, but I want to have everything integrated a few weeks early so I can help out with other stuff we are behind on like building the paddle or adding spin to the game.  The goal to to get the Jetson and Raspberry Pi integrated into the system within the next two weeks.  That means that we can connect to both devices, send/receive data, and have this data be completely synchronized with low latency.

This week I was able to reach my goals of implementing the multiplayer capabilities and adding Bluetooth communication code to give the VR headset the ability to interact with the paddle’s rPi and the camera’s Jetson.  The only thing that I am waiting on right now is the wifi card for the Jetson which I am going to order this weekend.  The wifi card is necessary for bluetooth connection between the Jetson and the Google Cardboard.

The multiplayer capabilities took a lot longer for me to implement than I expected.  Initially, I used an open source library called Riptide Networking to program this, but the problem was that it was a lot more low-level than I wanted it to be, which means more unnecessary work.  An example would be that I would have to explicitly code what information I wanted to send from a server to a client rather than the library taking care of that for me.  I spent a lot of time researching and starting the code using the Riptide library when I decided to change to a multiplayer library called Mirror, which does a lot more work for me than Riptide.  Mirror does a lot of client-server communication, so instead of me explicitly telling the server what to send to the client, Mirror automatically send the server’s game state to the client by itself.  The biggest thing I had to do with Mirror was program the spawn parameters for when players entered a game.  This includes limiting the game to have only two players,  making sure two players spawned on different ends of the table, and starting the game when two players connected to the game.  Another cool thing with Mirror is that the server and client run the same game, all I need to do is assign a computer as a server and the other two computers as clients.  This saved me a lot of time because with Riptide, I would’ve had to create a client version of the game where it read information from the server and reflected it in the graphics.

Going off our Gantt chart my progress is on time, I have about a week to get Logan’s CV information incorporated into my game, which shouldn’t take much time at all.  The biggest risk right now is how fast the wifi card can come in and how quickly we can get it working with the Jetson.  Any free time I have can be spent working on incorporating better graphics and finalizing the UI of the game.

This past week I was working on creating the graphics and game design for our MVP as well as beginning to work on my deliverables for next week.  What I need to do for the MVP is to have the graphics, game interactions, rules, and multiplayer/bluetooth capabilities implemented.  This past week, I was able to get the graphics, game interactions, and rules implemented on Unity, so right now the game looks like a 3D version of pong where two players can control the paddles using the left/right and a/d keys.   The game can also be imported onto an Android and iOS device to be played on a Google Cardboard.  I also began work on creating the multiplayer aspect of the game so that two different devices can play a game against each other rather than having two players play on one device.  I found the necessary libraries and tutorials to integrate that functionality so all I have to do now is implement it and test it.  For next week, the goal is to finish implementing the multiplayer functionality as well as adding Bluetooth capabilities to get data from our cameras and paddles.  By the end of next week, the game should be completely implemented and all that would need to be done to finish the MVP would be plugging in the data from Will and Logan, which should be really easy.  Everything so far seems to be going according to schedule.  I envision there being some debugging and testing time with the bluetooth functionality, but my goals are definitely realistic for next week.

I’ve been really busy with my other classes this week, so I haven’t been able to spend as much time as I would like to on our project this week.  I spent about 4-5 hours this week getting the VR headset set up and getting Unity ready on my computer.  I set up a demo game to see how the headset works, and I should be able to start working on the graphics and network connection next week.  Right now, I am definitely a little behind schedule as I need to set up the network connectivity and MVP graphics before the end of next week, which may be too much to ask for in one week.  The plan  is to get the MVP graphics created by next Saturday and catch up on the network communication functionality during spring break.

This week I have finalized our team’s network communication implementation for our MVP.  Looking into bluetooth communications and Unity plugins for that, I decided that the latencies for paddle-headset and paddle-server communications are pretty unnoticeable.  I did some research into some helpful libraries and found that the Arduino Bluetooth Plugin from the Unity Asset Store is going to help make data transfers easier between headset and paddle and server to paddle since it provides fast bluetooth communications between rPis and Androids/iOS/Mac devices.  For camera-headset communications, there was a hitch because the camera sends paddle position data through a Jetson Nano, which doesn’t have built in bluetooth capabilities.  My solution to this was to buy a wifi card and antennae that will give the Jetson bluetooth capabilities.  I don’t think the plugin is compatible with the Jetson, so I will probably need to program the network communication from camera-headset by myself.

I would say that I am a little bit behind schedule because I need to still set up my Google Cardboard with the Unity Game Engine before I can implement the code for the network communication.  However, I think that I should be able to get caught up and back on schedule within the next two weeks since I project creating the MVP graphics shouldn’t take too long.  Next week, I want to be able to send data from an rPi to an Android and a Jetson to an Android.  If I have time, it would be great if I could get a quick start on the graphics.

This week I was in charge of doing research on/ordering components for our VR headset, bluetooth communications, and game engine.  For the VR headset, I honed in on the Google Cardboard and took a while to see how it worked along with what game engines were compatible with the most recent SDK.  I took about 5-6 hours researching game engines, more specifically Unity and Unreal.  What I mostly focused on were the plugins that were useful to us, especially the Bluetooth plugins that would allow a paddle to communicate with a headset.  The Arduino Bluetooth Plugin for Unity was the most promising plugin I found, but the latency reported from the reviews was too large, which is why I am currently looking at a faster communication channel, Wifi-Direct.  I ended up settling on choosing the Unity game engine because the graphic design is really intuitive and it is compatible with the most recent version of the Google Cardboard SDK while the Unreal game engine is not.

Looking forward, I am on schedule right now but I may fall behind next week because I need to figure out how to send data from paddle to headset and from paddle to server.  The problem is that I may need to end up programming the binding and connections myself instead of using a plugin to reduce latency.  Also, I will also need to spend more time working on some higher-level design planning that will take time away from my work.  But the goal is the create a communication network from paddle to headset and from player to player.