Posted on

Henry’s Status Report for 03/26

This week I was able to reach my goals of implementing the multiplayer capabilities and adding Bluetooth communication code to give the VR headset the ability to interact with the paddle’s rPi and the camera’s Jetson.  The only thing that I am waiting on right now is the wifi card for the Jetson which I am going to order this weekend.  The wifi card is necessary for bluetooth connection between the Jetson and the Google Cardboard.

The multiplayer capabilities took a lot longer for me to implement than I expected.  Initially, I used an open source library called Riptide Networking to program this, but the problem was that it was a lot more low-level than I wanted it to be, which means more unnecessary work.  An example would be that I would have to explicitly code what information I wanted to send from a server to a client rather than the library taking care of that for me.  I spent a lot of time researching and starting the code using the Riptide library when I decided to change to a multiplayer library called Mirror, which does a lot more work for me than Riptide.  Mirror does a lot of client-server communication, so instead of me explicitly telling the server what to send to the client, Mirror automatically send the server’s game state to the client by itself.  The biggest thing I had to do with Mirror was program the spawn parameters for when players entered a game.  This includes limiting the game to have only two players,  making sure two players spawned on different ends of the table, and starting the game when two players connected to the game.  Another cool thing with Mirror is that the server and client run the same game, all I need to do is assign a computer as a server and the other two computers as clients.  This saved me a lot of time because with Riptide, I would’ve had to create a client version of the game where it read information from the server and reflected it in the graphics.

Going off our Gantt chart my progress is on time, I have about a week to get Logan’s CV information incorporated into my game, which shouldn’t take much time at all.  The biggest risk right now is how fast the wifi card can come in and how quickly we can get it working with the Jetson.  Any free time I have can be spent working on incorporating better graphics and finalizing the UI of the game.

Posted on

Team Status Report for 03/19

By far the biggest risk that came up this past week was the issue of accuracy of the position and orientation of the paddle.  The IMU was very inaccurate at the beginning of the week as the orientation would off by a lot, sometimes as bad as 45 degrees.  However, William was able to fine tune the device so that the orientation data was realistic from eyeballing the orientation of the IMU.  We are still going to work through some smaller problems with the IMU that William mentioned in his report as well as do more extensive testing to get quantitative data as to how accurate the IMU is.  Right now, if any issues with the IMU come up, our contingency plan is to either supplement the open source code we downloaded with our own code or to find a new source that has the features we need.   There are also accuracy issues with the camera as well.  Right now, Logan is working on an OpenCV algorithm that creates a bounding box around the paddle and we just take the center of that bounding box as the position the paddle needs to be in.  There’s definitely concern about how accurate this model is, but we believe that this algorithm is accurate enough that it is not noticeable by the player.  As with the IMU, once this algorithm is completed, we need to get quantitative data of how well the camera can track static and moving objects.

One change in our final design is that the game will be run on the server.  Something that came up during our design of the multiplayer capabilities of the game was that our old implementation had two players running the same game on two different devices.  We realized that synchronizing two different devices to show the same game would be too complicated and too risky, so we decided to centralize the game on a server to take away the synchronization headache.  We don’t see any cost in this new solution as the servers we were planning on using for our MVP and final implementation should be able to handle this extra workload.

As of right now, everything is on schedule and we may be looking at completing our MVP as early as the end of next week, assuming no unforeseen problem arise.  The shift of having our game be run on a centralized server is a minimal effort change for us that should not take more than an hour to fix.

Posted on

Henry’s Status Report for 03/19

This past week I was working on creating the graphics and game design for our MVP as well as beginning to work on my deliverables for next week.  What I need to do for the MVP is to have the graphics, game interactions, rules, and multiplayer/bluetooth capabilities implemented.  This past week, I was able to get the graphics, game interactions, and rules implemented on Unity, so right now the game looks like a 3D version of pong where two players can control the paddles using the left/right and a/d keys.   The game can also be imported onto an Android and iOS device to be played on a Google Cardboard.  I also began work on creating the multiplayer aspect of the game so that two different devices can play a game against each other rather than having two players play on one device.  I found the necessary libraries and tutorials to integrate that functionality so all I have to do now is implement it and test it.  For next week, the goal is to finish implementing the multiplayer functionality as well as adding Bluetooth capabilities to get data from our cameras and paddles.  By the end of next week, the game should be completely implemented and all that would need to be done to finish the MVP would be plugging in the data from Will and Logan, which should be really easy.  Everything so far seems to be going according to schedule.  I envision there being some debugging and testing time with the bluetooth functionality, but my goals are definitely realistic for next week.

Posted on

Team Status Report for 02/26

This week all three of us were getting our microcontrollers and hardware set-up for software development.  We had to set up the OS for the raspberry pi and Jetson as well as get the Google Cardboard setup with the Unity game engine.  Since everything this week went smoothly, there aren’t any new issues to deal with or changes necessary to make.  As of right now, all three of us are a little bit behind schedule because we had a hectic week, but the plan is to catch up next week or during spring break.

Posted on

Henry’s Status Report for 02/26

I’ve been really busy with my other classes this week, so I haven’t been able to spend as much time as I would like to on our project this week.  I spent about 4-5 hours this week getting the VR headset set up and getting Unity ready on my computer.  I set up a demo game to see how the headset works, and I should be able to start working on the graphics and network connection next week.  Right now, I am definitely a little behind schedule as I need to set up the network connectivity and MVP graphics before the end of next week, which may be too much to ask for in one week.  The plan  is to get the MVP graphics created by next Saturday and catch up on the network communication functionality during spring break.

Posted on

Team Status Report for 2/19

One issue that concerned us this week was how to communicate the position and orientation of an opponent’s paddle to a player.  We were worried that the latencies from an opponent’s paddle to a player’s headset would be too much and would cause delayed graphics and drastically affect gameplay.  An intermediate fix we have for the MVP is to use bluetooth communication.  Since for our MVP, the distance between players isn’t large, we can have an opponent’s paddle connect to a player’s headset and the latency between the two devices is pretty much negligible.  For the final product, we discussed adding delays to game to send paddle data, not showing the paddle to an opponent at all, or simplifying the game so that players are within range to have bluetooth connections.

We also made some adjustments to our server design.  Before, we were planning on using AWS Gamelift as a gaming server to compute the ping pong ball path.  However, because the server’s rate of sending and receiving data is low we decided to switch the server to either one of our computer or AWS EC2.  The biggest factors in this switch comes down to pricing, lower latency, and ease of implementing.  For example, if our server was one of our computers, we could really quickly set up a bluetooth network for quick communication and our computers should be powerful enough to calculate the ball projection.  If the bluetooth connection between computer and paddle are too slow, we can switch to a different communication protocol or use EC2 which can guarantee single digit millisecond latency.

Our schedule looking forward seems to be on schedule.  What we are going to be working on this week is data transfer and collecting data from sensors.  We should also start implementation of basic graphics for our MVP.

Posted on

Henry’s Status Report for 2/19

This week I have finalized our team’s network communication implementation for our MVP.  Looking into bluetooth communications and Unity plugins for that, I decided that the latencies for paddle-headset and paddle-server communications are pretty unnoticeable.  I did some research into some helpful libraries and found that the Arduino Bluetooth Plugin from the Unity Asset Store is going to help make data transfers easier between headset and paddle and server to paddle since it provides fast bluetooth communications between rPis and Androids/iOS/Mac devices.  For camera-headset communications, there was a hitch because the camera sends paddle position data through a Jetson Nano, which doesn’t have built in bluetooth capabilities.  My solution to this was to buy a wifi card and antennae that will give the Jetson bluetooth capabilities.  I don’t think the plugin is compatible with the Jetson, so I will probably need to program the network communication from camera-headset by myself.

I would say that I am a little bit behind schedule because I need to still set up my Google Cardboard with the Unity Game Engine before I can implement the code for the network communication.  However, I think that I should be able to get caught up and back on schedule within the next two weeks since I project creating the MVP graphics shouldn’t take too long.  Next week, I want to be able to send data from an rPi to an Android and a Jetson to an Android.  If I have time, it would be great if I could get a quick start on the graphics.

Posted on

Henry’s Status Report for 02/12

This week I was in charge of doing research on/ordering components for our VR headset, bluetooth communications, and game engine.  For the VR headset, I honed in on the Google Cardboard and took a while to see how it worked along with what game engines were compatible with the most recent SDK.  I took about 5-6 hours researching game engines, more specifically Unity and Unreal.  What I mostly focused on were the plugins that were useful to us, especially the Bluetooth plugins that would allow a paddle to communicate with a headset.  The Arduino Bluetooth Plugin for Unity was the most promising plugin I found, but the latency reported from the reviews was too large, which is why I am currently looking at a faster communication channel, Wifi-Direct.  I ended up settling on choosing the Unity game engine because the graphic design is really intuitive and it is compatible with the most recent version of the Google Cardboard SDK while the Unreal game engine is not.

Looking forward, I am on schedule right now but I may fall behind next week because I need to figure out how to send data from paddle to headset and from paddle to server.  The problem is that I may need to end up programming the binding and connections myself instead of using a plugin to reduce latency.  Also, I will also need to spend more time working on some higher-level design planning that will take time away from my work.  But the goal is the create a communication network from paddle to headset and from player to player.

Posted on

Team Status Report for 02/12

Our biggest risk with our product right now is latency, specifically how long it takes for data from the paddle to be processed and sent to the headset.  We are focusing on two aspects to improve latency: communication channels and communication protocols.   For communication channels, we are looking at bluetooth plugins to send paddle orientation data to our headset and AWS servers.  Depending on how fast bluetooth is, we could switch to WiFi-direct among other methods if bluetooth’s latency is too high.  For communication protocols, we are focusing on what data to send and where/when to send it.  Last week we had conversation about sending paddle data to a server, which can project the movement of the ping pong ball and send those projections to each player’s headset.  This creates a buffer of data that limits the amount of network communication and thus lowers latency.  What we are focusing on now is how to hide the delay where the server needs to calculate and send the ball projection data.  We are still working on finalizing the protocols right now, but if there are certain latencies that we cannot optimize, we can sacrifice some of the realism of the game to allow for smoother gameplay.  An example could be having the ball pause for a second at one end of the table to allow the server to calculate the projected movement of the ball towards the player.

Moving forward, we are going to continue to work on our communication protocols as well as researching faster communication channels.  We also received a shipment of our hardware, which we plan on testing out to get comfortable with how it works and look for any problems that we may need to work around.  Everything so far is on schedule.