Category: Team Status Reports

The biggest risks that we have in the upcoming week is the latency of a swing being reflected on the graphics and how accurately the graphics reflect the motion of the swing.  If there is too much latency from data transfer between the VR headset and the paddle, we can work on shortening the sampling period to try to identify and analyze as little IMU data as possible before sending information to the headset.  We don’t expect there to be problems with how accurately the graphics reflect the motion of the swing, but if problems do come up, we need to figure out exactly what motions correspond to what swing types and make sure to explain that to the users in a help screen.

Nothing this week has been done to change how the system works.  How it works now is that the game sends the paddle a signal of when to start sampling data.  The IMU samples an amount of data and analyzes to figure out the swing type, power, and direction and sends these swing parameters to the game, where it triggers the proper swing animation and paddle-ball interaction.  The game is run on a dedicated server, but the paddle is controlled by each player and the player’s VR headset sends paddle state to the server and transitively the other player as well.

Last week, William and Henry were able to get the IMU integrated into the game and player swings can now be seen on the game with low latency.  Next week, everyone is going to be working on making the game more playable.  This includes getting the final testing metrics, adjusting animations, tuning thresholds for swing power and direction, and adding features like scoring into the game.

This week, we continued the road towards integrating towards our final project. William worked on thresholding for the paddle using the IMU, and has found reliable trends that we can use to discretize the swings into a few different categories. The plan is now to send a notification from the game engine to the paddle system through bluetooth as to when the swing tracking window begins, and then for the paddle to respond to the game engine with whether the swing connected and what type of swing should be recorded.

Now, the integration that needs to be done involves setting up a reliable connection with bluetooth, and then sending the aformentioned data between the paddle and game engine. We also need to see how the latencies affect transmission, so that we can tune the parameters as to how much of a buffer we need to allow for real-time transmission of the swing result. Once we work those out, we can gather the data on the raspberry pi to determine the swing type.

As for the CV, we are planning to leave that out until we get the IMU system integrated, as the CV is behind on the testing and integration onto the Jetson. If we get the IMU up and running soon, then we can consider re-integrating the CV.

The biggest risk is the same as it has been the past two weeks and that is the integration of the Jetson and the CV code.  The bluetooth connection between the Jetson and the game has not been set up yet, but Henry is going to try to get that done tomorrow, and Logan should have the CV code done by Monday.  However, we have a new contingency plan in case we are unable to integrate the CV code.  In this case, we would make the game similar to Wii Sports Tennis, where the paddle will follow the ball automatically and the game then becomes more timing and swing-type oriented.

As mentioned in Henry and William’s report, the biggest change to the system is how much the game is going to depend on the data inputs of the Jetson and rPi.  As of last week, the plan was to set the orientation and position of the paddle in the game with the orientation and position data from the sensors.  The problem with this is that is that we worried about the possibilities of lag and having sudden movements not be registered.  We also ran into the problem of determining the velocity of the ball after it has been hit with a paddle because of the inaccurate velocity deductions from the IMU.  To solve both of these problems, we decided to simplify the game to only read the data from the rPi and Jetson during a specific time interval after the ball hits the table and using this data to determine what type of swing a player is performing from a set of predefined swing types.  Once a swing type is determined, it triggers an animation that Henry has already programmed and if the ball collides with the paddle, the paddle adds a velocity vector to the ball based on the swing type.

The biggest goals for next week are the refine the graphics, integrate the rPi and Jetson with the game, and finish fabrication of the paddles.  We have a lighter workload next week compared to this week, but we still have a bit of catching up to do to produce a quality final product.

Right now what we have to do is build the paddle, integrate the rPi and Jetson to the server, finish up the final graphics, finish the computer vision, and add spin capabilities to the game.  Our biggest concern right now is the bluetooth connection between the Jetson and the server because Logan is still working on the computer vision stuff and was not able to set up the Jetson and its bluetooth capabilities.  So Henry is going to try to get that set up this week.  For the bluetooth connection with the rPi, we had the bluetooth connection working temporarily, but it stopped working during the demo.  We think we have located the problem so that is something we are going fix and do more extensive testing on next week.  Henry is also going to try to finish up the final graphics next week, which means adding the VR processing to be compatible with the Google Cardboard and creating more realistic paddles, balls, tables, and better camera angles.  William is working on getting accurate acceleration and velocity data from his IMU so that players can add spin to the ball.

As of right now, we are beginning to fall behind on a few tasks.  We need to get the Jetson and Raspberry Pi integrated, we need to get the paddle built, and we need to add the aspect of spin into our game.  The biggest risk that we are looking at is the integration of the microcontrollers with the Unity game engine.  Henry and William will test to see if the Raspberry Pi can integrate with Unity on Monday, and if it cannot work, we will have two alternate approaches.  The first is to add a new bluetooth module to the Raspberry Pi that allows connection with the bluetooth library we are using on Unity.  The second is to write code ourselves that creates Bluetooth connections.

Even with these hiccups, we don’t see any reason to change the system of our product at all, and our schedule is going to remain the same since we foresaw possible difficulties with the integration.  Referencing Henry’s post, he is going to finish integration of the Raspberry Pi and Jetson within 2 weeks.  William is going to be working on building the paddle and helping Henry with integrating the Raspberry Pi.  And Logan is going to finish up his OpenCV code and work on adding the spin capabilities to the game as well.

As of the end of this week, the individual components of the MVP are completed and next week we are going to finish integrating the components together.  The IMU isn’t a part of the MVP, but there has been considerable progress in making the data output more consistent and accurate.  The game itself has been completed too.  What’s new from this week are the multiplayer and bluetooth capabilities that allow two players to play each other and connect their paddles to the game.  The computer vision algorithm for tracking the lateral position of the paddle is about completed as well.  Looking ahead, the plan for next week is to send the lateral position of the paddle to the Google Cardboards running the game, so that two players can control their respective paddles in a game of Pong.  We are projecting there to be a few calibration adjustments to allow for centering of the paddle and we are going to test for the latency of the paddle movement as well.  For this week, no changes were made to the product design.

The biggest risk that we face is buying a wifi card that either isn’t compatible with the libraries we are using or the Jetson Nano itself.   While this risk is mitigated with good product research, a contingency plan we have is to buy multiple wifi cards at a time so we have backups in case one fails.

By far the biggest risk that came up this past week was the issue of accuracy of the position and orientation of the paddle.  The IMU was very inaccurate at the beginning of the week as the orientation would off by a lot, sometimes as bad as 45 degrees.  However, William was able to fine tune the device so that the orientation data was realistic from eyeballing the orientation of the IMU.  We are still going to work through some smaller problems with the IMU that William mentioned in his report as well as do more extensive testing to get quantitative data as to how accurate the IMU is.  Right now, if any issues with the IMU come up, our contingency plan is to either supplement the open source code we downloaded with our own code or to find a new source that has the features we need.   There are also accuracy issues with the camera as well.  Right now, Logan is working on an OpenCV algorithm that creates a bounding box around the paddle and we just take the center of that bounding box as the position the paddle needs to be in.  There’s definitely concern about how accurate this model is, but we believe that this algorithm is accurate enough that it is not noticeable by the player.  As with the IMU, once this algorithm is completed, we need to get quantitative data of how well the camera can track static and moving objects.

One change in our final design is that the game will be run on the server.  Something that came up during our design of the multiplayer capabilities of the game was that our old implementation had two players running the same game on two different devices.  We realized that synchronizing two different devices to show the same game would be too complicated and too risky, so we decided to centralize the game on a server to take away the synchronization headache.  We don’t see any cost in this new solution as the servers we were planning on using for our MVP and final implementation should be able to handle this extra workload.

As of right now, everything is on schedule and we may be looking at completing our MVP as early as the end of next week, assuming no unforeseen problem arise.  The shift of having our game be run on a centralized server is a minimal effort change for us that should not take more than an hour to fix.

This week all three of us were getting our microcontrollers and hardware set-up for software development.  We had to set up the OS for the raspberry pi and Jetson as well as get the Google Cardboard setup with the Unity game engine.  Since everything this week went smoothly, there aren’t any new issues to deal with or changes necessary to make.  As of right now, all three of us are a little bit behind schedule because we had a hectic week, but the plan is to catch up next week or during spring break.

One issue that concerned us this week was how to communicate the position and orientation of an opponent’s paddle to a player.  We were worried that the latencies from an opponent’s paddle to a player’s headset would be too much and would cause delayed graphics and drastically affect gameplay.  An intermediate fix we have for the MVP is to use bluetooth communication.  Since for our MVP, the distance between players isn’t large, we can have an opponent’s paddle connect to a player’s headset and the latency between the two devices is pretty much negligible.  For the final product, we discussed adding delays to game to send paddle data, not showing the paddle to an opponent at all, or simplifying the game so that players are within range to have bluetooth connections.

We also made some adjustments to our server design.  Before, we were planning on using AWS Gamelift as a gaming server to compute the ping pong ball path.  However, because the server’s rate of sending and receiving data is low we decided to switch the server to either one of our computer or AWS EC2.  The biggest factors in this switch comes down to pricing, lower latency, and ease of implementing.  For example, if our server was one of our computers, we could really quickly set up a bluetooth network for quick communication and our computers should be powerful enough to calculate the ball projection.  If the bluetooth connection between computer and paddle are too slow, we can switch to a different communication protocol or use EC2 which can guarantee single digit millisecond latency.

Our schedule looking forward seems to be on schedule.  What we are going to be working on this week is data transfer and collecting data from sensors.  We should also start implementation of basic graphics for our MVP.

Our biggest risk with our product right now is latency, specifically how long it takes for data from the paddle to be processed and sent to the headset.  We are focusing on two aspects to improve latency: communication channels and communication protocols.   For communication channels, we are looking at bluetooth plugins to send paddle orientation data to our headset and AWS servers.  Depending on how fast bluetooth is, we could switch to WiFi-direct among other methods if bluetooth’s latency is too high.  For communication protocols, we are focusing on what data to send and where/when to send it.  Last week we had conversation about sending paddle data to a server, which can project the movement of the ping pong ball and send those projections to each player’s headset.  This creates a buffer of data that limits the amount of network communication and thus lowers latency.  What we are focusing on now is how to hide the delay where the server needs to calculate and send the ball projection data.  We are still working on finalizing the protocols right now, but if there are certain latencies that we cannot optimize, we can sacrifice some of the realism of the game to allow for smoother gameplay.  An example could be having the ball pause for a second at one end of the table to allow the server to calculate the projected movement of the ball towards the player.

Moving forward, we are going to continue to work on our communication protocols as well as researching faster communication channels.  We also received a shipment of our hardware, which we plan on testing out to get comfortable with how it works and look for any problems that we may need to work around.  Everything so far is on schedule.