Posted on

Team Status Report for 04/30

The biggest risks that we have in the upcoming week is the latency of a swing being reflected on the graphics and how accurately the graphics reflect the motion of the swing.  If there is too much latency from data transfer between the VR headset and the paddle, we can work on shortening the sampling period to try to identify and analyze as little IMU data as possible before sending information to the headset.  We don’t expect there to be problems with how accurately the graphics reflect the motion of the swing, but if problems do come up, we need to figure out exactly what motions correspond to what swing types and make sure to explain that to the users in a help screen.

Nothing this week has been done to change how the system works.  How it works now is that the game sends the paddle a signal of when to start sampling data.  The IMU samples an amount of data and analyzes to figure out the swing type, power, and direction and sends these swing parameters to the game, where it triggers the proper swing animation and paddle-ball interaction.  The game is run on a dedicated server, but the paddle is controlled by each player and the player’s VR headset sends paddle state to the server and transitively the other player as well.

Last week, William and Henry were able to get the IMU integrated into the game and player swings can now be seen on the game with low latency.  Next week, everyone is going to be working on making the game more playable.  This includes getting the final testing metrics, adjusting animations, tuning thresholds for swing power and direction, and adding features like scoring into the game.

Posted on

Henry’s Status Report for 04/30

Last week, on Sunday and Monday, I spent a lot of time on getting the paddle integrated with the game.  I created an interface where the IMU sends me a swing type, power, and direction and I use those commands to trigger specific swing animations. It took a while to figure out why the rPi wasn’t accepting bluetooth connections.  Also on Sunday, I had to spend a while on the final presentation, where I pretty much did everything. On Thursday and Friday, I was working with William on swing detection and have a player’s swing reflect accurately on the game.  We were mainly working on when the IMU should start analyzing its data and how to signal the start and end of these analysis periods.  By the end of Friday though, we had made really good progress and were able to have a player swing the paddle and have the correct swing animation trigger on the game with pretty low latency.

Since we have all the components of the game integrated and working, I’m going to be working on making the game more realistic.  When William and I were working on Thursday and Friday, there were a few weird animations in my code that I need to fix and the ball and paddle speed were unrealistically slow.  I’m going to meet up with William on Wednesday after I make these changes to make sure that the IMU still has enough time to sample orientation/acceleration data and correctly identify the swing type/power/direction.  By Wednesday, I’m also going to have scorekeeping set up.   On Thursday, our entire group is going to meet up to do final tests and make adjustments to our game as we see necessary.

I have a lot of work to do next week, but I definitely have enough time to get it done.

Posted on

Henry’s Status Report for 04/23

This week I’ve been mainly working on adding important features to the game and wrestling with making the bluetooth connection between the rPi and VR headset more robust.  For the game, I worked on adjusting the paddle-ball interactions to give players a more realistic chance of hitting the ball, adding table-ball interactions so that the game can judge whether a player won a point if a ball hits a side twice, and implementing lateral tracking features in the paddle so that the paddle will always track the ball in the x direction.  That last features is implemented in case the Jetson cannot be integrated into the game.  For the bluetooth work, I changed the game so that instead of the rPi sending data to the server and having the server change the paddle orientation, the rPi will send its data to the VR headset, which contacts the server to change the location of the paddle.  I did this because the bluetooth connections between the rPi and our VR headsets connect much more frequently than connections between the rPi and our server.  Right now,  I am doing testing with the latency of the bluetooth connection and workin with William on thresholding to determine the swing type for the game.

Once everything is integrated and tested, we need to try to set up the Jetson and put the finishing touches on game, which includes score-tracking and  smoother animations.  Next week, I’m going to continue to work on integrating the rPi with the game and try to integrate the Jetson as well.

Posted on

Team Status Report 04/16

The biggest risk is the same as it has been the past two weeks and that is the integration of the Jetson and the CV code.  The bluetooth connection between the Jetson and the game has not been set up yet, but Henry is going to try to get that done tomorrow, and Logan should have the CV code done by Monday.  However, we have a new contingency plan in case we are unable to integrate the CV code.  In this case, we would make the game similar to Wii Sports Tennis, where the paddle will follow the ball automatically and the game then becomes more timing and swing-type oriented.

As mentioned in Henry and William’s report, the biggest change to the system is how much the game is going to depend on the data inputs of the Jetson and rPi.  As of last week, the plan was to set the orientation and position of the paddle in the game with the orientation and position data from the sensors.  The problem with this is that is that we worried about the possibilities of lag and having sudden movements not be registered.  We also ran into the problem of determining the velocity of the ball after it has been hit with a paddle because of the inaccurate velocity deductions from the IMU.  To solve both of these problems, we decided to simplify the game to only read the data from the rPi and Jetson during a specific time interval after the ball hits the table and using this data to determine what type of swing a player is performing from a set of predefined swing types.  Once a swing type is determined, it triggers an animation that Henry has already programmed and if the ball collides with the paddle, the paddle adds a velocity vector to the ball based on the swing type.

The biggest goals for next week are the refine the graphics, integrate the rPi and Jetson with the game, and finish fabrication of the paddles.  We have a lighter workload next week compared to this week, but we still have a bit of catching up to do to produce a quality final product.

Posted on

Henry’s Status Report for 04/16

Progress this week is going as planned heading into next week.  I was able to create the graphics and animations for the final game.  This includes the incorporation the graphics for the paddle, table, and ball, animating and properly triggering the different types of swings, and setting the ball-paddle interactions that set the ball’s trajectory based on specific parameters of the hitter’s swing such as orientation and power.  I’m at the point where two players can play each other using the keyboard as an input.  Looking ahead, the biggest goal is to integrate the rPi and Jetson into the game through bluetooth. But some other things I’m expecting to do are refining the animations to look more smooth and realistic and fine-tuning swing thresholds to make gameplay more realistic.

The swing thresholds are something Will and I brainstormed this week.  We were trying to figure out how to deal with paddle-ball interactions and how we would figure out the ball’s direction and velocity after a collision, which was difficult because of the inaccurate velocity calculations from signal processing modules Will found online.  Another concern was with how the paddle movement would look when we integrated the rPi with the game and how laggy it could possibly look.  Our solution was to restrict sampling to a specific time interval after the ball hit the table and measure the orientation, acceleration, and position data to determine what kind of swing the player was performing.  It would then trigger an already animated swing corresponding to the determined swing type. We categorized swing data to swing types by creating thresholds for each piece of data and setting a threshold to a particular swing type.

I am optimistic that we are heading back on schedule because all I have to do next week is to try to integrate the rPi and Jetson data into the game and optimize certain aspects to make it more playable.  Tomorrow, I am going to be working on creating a bluetooth connection between the Jetson and the game so that on Monday, Logan can just load in his CV code and we can start testing the game in its totality.

Posted on

Team Status Report for 04/10

Right now what we have to do is build the paddle, integrate the rPi and Jetson to the server, finish up the final graphics, finish the computer vision, and add spin capabilities to the game.  Our biggest concern right now is the bluetooth connection between the Jetson and the server because Logan is still working on the computer vision stuff and was not able to set up the Jetson and its bluetooth capabilities.  So Henry is going to try to get that set up this week.  For the bluetooth connection with the rPi, we had the bluetooth connection working temporarily, but it stopped working during the demo.  We think we have located the problem so that is something we are going fix and do more extensive testing on next week.  Henry is also going to try to finish up the final graphics next week, which means adding the VR processing to be compatible with the Google Cardboard and creating more realistic paddles, balls, tables, and better camera angles.  William is working on getting accurate acceleration and velocity data from his IMU so that players can add spin to the ball.

Posted on

Henry’s Status Report 04/10

This week, I was working with William to get the rPi connected with the game.  On Tuesday we were able to send rotational vectors between the rPi and a Mac and print those values out, but on Wednesday during our demo it didn’t work.  The reason why is because when I was importing the game onto my phone, I had to change a lot of build settings, which affected the bluetooth capabilities.  I also began to work on the final version of the graphics this week: incorporating a VR library that helps process the game to be viewed on a Google Cardboard, and designing the tables, paddles, and camera perspective for the final game.  The plan is to get this all implemented by Thursday or Friday.  What I am also going to be working on next week is setting up the Jetson Nano.  Logan hasn’t been able to work on it these past few weeks, so I’m going to take it since time is beginning to run out.  Hopefully by the end of next week I should be able to have the Jetson Nano set up and add bluetooth capabilities to it.  That way I can test the Bluetooth connection between the Jetson and the server.  I am behind schedule right now, but given the relatively light week I have upcoming, I am ready to put over 20 hours working on what I laid out above.

Posted on

Team Status Report for 03/02

As of right now, we are beginning to fall behind on a few tasks.  We need to get the Jetson and Raspberry Pi integrated, we need to get the paddle built, and we need to add the aspect of spin into our game.  The biggest risk that we are looking at is the integration of the microcontrollers with the Unity game engine.  Henry and William will test to see if the Raspberry Pi can integrate with Unity on Monday, and if it cannot work, we will have two alternate approaches.  The first is to add a new bluetooth module to the Raspberry Pi that allows connection with the bluetooth library we are using on Unity.  The second is to write code ourselves that creates Bluetooth connections.

Even with these hiccups, we don’t see any reason to change the system of our product at all, and our schedule is going to remain the same since we foresaw possible difficulties with the integration.  Referencing Henry’s post, he is going to finish integration of the Raspberry Pi and Jetson within 2 weeks.  William is going to be working on building the paddle and helping Henry with integrating the Raspberry Pi.  And Logan is going to finish up his OpenCV code and work on adding the spin capabilities to the game as well.

Posted on

Henry’s Status Report for 04/02

I started this week with the intention of integrating the Jetson into the game.  I spent a while first adding the Arduino Bluetooth library into our Unity code which should give us the ability for the Raspberry Pi and Jetson to send its data to the code.  However, this is where I ran into a few pretty big roadblocks.  The first one, is that when I was trying to connect to the Jetson, I realized that Logan had not set it up yet.  So, something that we have to work on next week is getting the Jetson’s OS set up and get the Bluetooth drivers and capabilities set up to see if it can connect with our game.  Another roadblock/uncertainty is whether the Arduino Library will work or not. The library is only compatible with certain Bluetooth modules, and I didn’t have anything to test with that had those modules.  The only testing I could really do was to see if my code would know if my Airpods were paired with my Mac or not, which it could.  So, on Monday, I am going to work with William to see if we can get the rPi connected.  If we can’t, I’m going to either have to buy a bluetooth module compatible with the Bluetooth library, or I have to write my own code using NET to connect with Bluetooth devices.

Because there were roadblocks with integrating devices into the game, I decided to refine my Unity code more.  The first thing I did was removing the GUI for determining whether we were running code as the server or as the client.  It’s unnecessary work for users and going to be extra work for us in the future since we would’ve had to add some VR pointing capabilities since our game doesn’t have a mouse and keyboard.  The second thing I did was getting the code ready for Raspberry Pi and Jetson inputs.  I added commented out code that sets the paddle location and orientation to these inputs and code that blocks until both pieces of data are received, so that the paddle input data is synchronized.

For my responsibilities, I am falling behind a little bit.  On the Gantt chart, it mentions that I have all of April to integrate the devices, but I want to have everything integrated a few weeks early so I can help out with other stuff we are behind on like building the paddle or adding spin to the game.  The goal to to get the Jetson and Raspberry Pi integrated into the system within the next two weeks.  That means that we can connect to both devices, send/receive data, and have this data be completely synchronized with low latency.

Posted on

Team Status Report for 03/26

As of the end of this week, the individual components of the MVP are completed and next week we are going to finish integrating the components together.  The IMU isn’t a part of the MVP, but there has been considerable progress in making the data output more consistent and accurate.  The game itself has been completed too.  What’s new from this week are the multiplayer and bluetooth capabilities that allow two players to play each other and connect their paddles to the game.  The computer vision algorithm for tracking the lateral position of the paddle is about completed as well.  Looking ahead, the plan for next week is to send the lateral position of the paddle to the Google Cardboards running the game, so that two players can control their respective paddles in a game of Pong.  We are projecting there to be a few calibration adjustments to allow for centering of the paddle and we are going to test for the latency of the paddle movement as well.  For this week, no changes were made to the product design.

The biggest risk that we face is buying a wifi card that either isn’t compatible with the libraries we are using or the Jetson Nano itself.   While this risk is mitigated with good product research, a contingency plan we have is to buy multiple wifi cards at a time so we have backups in case one fails.