Posted on

William’s Status Report for 4/30

This week, I was able to perform steps to integrate with Henry’s game engine. First of all, I solved some technical difficulties by finding a new library for the Madgwick Algorithm in python instead of C++, so that I could run the entire program in one Python process instead of worrying about running the bluetooth python script and the C++ sensor fusion processes in parallel. I tested the library and tuned the beta constant in that library so that the changes to the IMU orientation would reflect quickly enough. I also researched and found some code that removed the acceleration due to gravity from our IMU acceleration measurements, which allows for the acceleration data to be centered around 0 no matter what orientation the paddle is in.

Armed with those improvements, I was able to integrate my bluetooth functionality into the IMU program to connect with Henry’s game engine. We worked on tuning the thresholds for the paddle swing, including the thresholds for orientation and acceleration, as well as the amount of time we sample swing data over. After observing that data, we have found thresholds that work.

Next week, I will continue performing further tests on the integration, as well as work on the final report and the demo video.

Posted on

Logan’s Status Report for 4/30

This week I continued working on the report. Currently, the text for the first 3 sections is completed and I have begun working on the next sections. In addition to this I worked on updating our block diagrams and adding more detail to them, as well as collect references for our report as per the feedback on our design report.

This week I will finalize the report, as well as complete the product pitch for the poster in preparation for the public demo.

Posted on

Team Status Report for 04/30

The biggest risks that we have in the upcoming week is the latency of a swing being reflected on the graphics and how accurately the graphics reflect the motion of the swing.  If there is too much latency from data transfer between the VR headset and the paddle, we can work on shortening the sampling period to try to identify and analyze as little IMU data as possible before sending information to the headset.  We don’t expect there to be problems with how accurately the graphics reflect the motion of the swing, but if problems do come up, we need to figure out exactly what motions correspond to what swing types and make sure to explain that to the users in a help screen.

Nothing this week has been done to change how the system works.  How it works now is that the game sends the paddle a signal of when to start sampling data.  The IMU samples an amount of data and analyzes to figure out the swing type, power, and direction and sends these swing parameters to the game, where it triggers the proper swing animation and paddle-ball interaction.  The game is run on a dedicated server, but the paddle is controlled by each player and the player’s VR headset sends paddle state to the server and transitively the other player as well.

Last week, William and Henry were able to get the IMU integrated into the game and player swings can now be seen on the game with low latency.  Next week, everyone is going to be working on making the game more playable.  This includes getting the final testing metrics, adjusting animations, tuning thresholds for swing power and direction, and adding features like scoring into the game.

Posted on

Henry’s Status Report for 04/30

Last week, on Sunday and Monday, I spent a lot of time on getting the paddle integrated with the game.  I created an interface where the IMU sends me a swing type, power, and direction and I use those commands to trigger specific swing animations. It took a while to figure out why the rPi wasn’t accepting bluetooth connections.  Also on Sunday, I had to spend a while on the final presentation, where I pretty much did everything. On Thursday and Friday, I was working with William on swing detection and have a player’s swing reflect accurately on the game.  We were mainly working on when the IMU should start analyzing its data and how to signal the start and end of these analysis periods.  By the end of Friday though, we had made really good progress and were able to have a player swing the paddle and have the correct swing animation trigger on the game with pretty low latency.

Since we have all the components of the game integrated and working, I’m going to be working on making the game more realistic.  When William and I were working on Thursday and Friday, there were a few weird animations in my code that I need to fix and the ball and paddle speed were unrealistically slow.  I’m going to meet up with William on Wednesday after I make these changes to make sure that the IMU still has enough time to sample orientation/acceleration data and correctly identify the swing type/power/direction.  By Wednesday, I’m also going to have scorekeeping set up.   On Thursday, our entire group is going to meet up to do final tests and make adjustments to our game as we see necessary.

I have a lot of work to do next week, but I definitely have enough time to get it done.

Posted on

Logan’s Status Report for 4/23

This week I worked on the Final Report as well as prepared for the Final Presentation which I am responsible for. Next week I will continue to work on the report, and either assist my teammates integrating the Jetson or on other tasks such as graphics depending on our mitigation plans.

Posted on

William’s Status Report for 4/23

This week, I continued trying to push my end of the integration to-do list to prepare for the integration. I tried a few different design concepts to implement the IMU side of the communications, which involves gathering data in a specified time frame, and then using the data to determine a swing type that will be sent to the game engine via bluetooth.

One concept that I tried was to accomplish the data collection and analysis entirely in the same C++ program that reads IMU data. For now, this idea is in the back seat, as I could not find a reliable C++ timer mechanism that will track time for us. Thus, I resorted to just moving the data collection to the same python script that will handle bluetooth connections. To do so, the Rpi will run two concurrent processes, one running the IMU data reading and Madgwick algorithm, and the other taking in that data as input through a unix pipe and then when the time is right, analyzing a window of that data. There was a slight issue of how the python timer jumps by quarter second intervals occasionally, which can throw of data collection, so I tried to adjust the implementation of data collection to rely on the python timer as little as possible, so that the error cannot propagate through different stages of the data analysis process.

For the data analysis process, during the swing window which is of differential time, I am taking the average orientation of the paddle during that window to establish what type of swing (lob, smash, topspin, slice) is being executed. To determine the speed of the paddle, and whether the paddle is swung at all, I am using the acceleration data. Through empirical testing, I found that the acceleration in the Z-axis of the IMU has an upward peak followed by a downward peak during a forehand, and a downward peak followed by a upward peak during a backhand. Thus, the goal is to identify if such peaks take place during the swing window, which can help us decide whether the swing occurred in time and what direction and power it was swung with. To help eliminate noise, I used a Scipy butterworth filter to smooth out the data, and then to determine the peak, we can first track what the acceleration reads before the window, and then integrate the values during the window to see what direction of peak we have. This is something that will have to be tuned going forward by testing it with the game engine in real time.

I also used CAD to design and subsequently 3d-print a box to hold our IMU system, which will be attached to a ping-pong paddle for gameplay.

Posted on

Team Status Report for 4/23

This week, we continued the road towards integrating towards our final project. William worked on thresholding for the paddle using the IMU, and has found reliable trends that we can use to discretize the swings into a few different categories. The plan is now to send a notification from the game engine to the paddle system through bluetooth as to when the swing tracking window begins, and then for the paddle to respond to the game engine with whether the swing connected and what type of swing should be recorded.

Now, the integration that needs to be done involves setting up a reliable connection with bluetooth, and then sending the aformentioned data between the paddle and game engine. We also need to see how the latencies affect transmission, so that we can tune the parameters as to how much of a buffer we need to allow for real-time transmission of the swing result. Once we work those out, we can gather the data on the raspberry pi to determine the swing type.

As for the CV, we are planning to leave that out until we get the IMU system integrated, as the CV is behind on the testing and integration onto the Jetson. If we get the IMU up and running soon, then we can consider re-integrating the CV.

Posted on

Henry’s Status Report for 04/23

This week I’ve been mainly working on adding important features to the game and wrestling with making the bluetooth connection between the rPi and VR headset more robust.  For the game, I worked on adjusting the paddle-ball interactions to give players a more realistic chance of hitting the ball, adding table-ball interactions so that the game can judge whether a player won a point if a ball hits a side twice, and implementing lateral tracking features in the paddle so that the paddle will always track the ball in the x direction.  That last features is implemented in case the Jetson cannot be integrated into the game.  For the bluetooth work, I changed the game so that instead of the rPi sending data to the server and having the server change the paddle orientation, the rPi will send its data to the VR headset, which contacts the server to change the location of the paddle.  I did this because the bluetooth connections between the rPi and our VR headsets connect much more frequently than connections between the rPi and our server.  Right now,  I am doing testing with the latency of the bluetooth connection and workin with William on thresholding to determine the swing type for the game.

Once everything is integrated and tested, we need to try to set up the Jetson and put the finishing touches on game, which includes score-tracking and  smoother animations.  Next week, I’m going to continue to work on integrating the rPi with the game and try to integrate the Jetson as well.

Posted on

Team Status Report 04/16

The biggest risk is the same as it has been the past two weeks and that is the integration of the Jetson and the CV code.  The bluetooth connection between the Jetson and the game has not been set up yet, but Henry is going to try to get that done tomorrow, and Logan should have the CV code done by Monday.  However, we have a new contingency plan in case we are unable to integrate the CV code.  In this case, we would make the game similar to Wii Sports Tennis, where the paddle will follow the ball automatically and the game then becomes more timing and swing-type oriented.

As mentioned in Henry and William’s report, the biggest change to the system is how much the game is going to depend on the data inputs of the Jetson and rPi.  As of last week, the plan was to set the orientation and position of the paddle in the game with the orientation and position data from the sensors.  The problem with this is that is that we worried about the possibilities of lag and having sudden movements not be registered.  We also ran into the problem of determining the velocity of the ball after it has been hit with a paddle because of the inaccurate velocity deductions from the IMU.  To solve both of these problems, we decided to simplify the game to only read the data from the rPi and Jetson during a specific time interval after the ball hits the table and using this data to determine what type of swing a player is performing from a set of predefined swing types.  Once a swing type is determined, it triggers an animation that Henry has already programmed and if the ball collides with the paddle, the paddle adds a velocity vector to the ball based on the swing type.

The biggest goals for next week are the refine the graphics, integrate the rPi and Jetson with the game, and finish fabrication of the paddles.  We have a lighter workload next week compared to this week, but we still have a bit of catching up to do to produce a quality final product.

Posted on

Henry’s Status Report for 04/16

Progress this week is going as planned heading into next week.  I was able to create the graphics and animations for the final game.  This includes the incorporation the graphics for the paddle, table, and ball, animating and properly triggering the different types of swings, and setting the ball-paddle interactions that set the ball’s trajectory based on specific parameters of the hitter’s swing such as orientation and power.  I’m at the point where two players can play each other using the keyboard as an input.  Looking ahead, the biggest goal is to integrate the rPi and Jetson into the game through bluetooth. But some other things I’m expecting to do are refining the animations to look more smooth and realistic and fine-tuning swing thresholds to make gameplay more realistic.

The swing thresholds are something Will and I brainstormed this week.  We were trying to figure out how to deal with paddle-ball interactions and how we would figure out the ball’s direction and velocity after a collision, which was difficult because of the inaccurate velocity calculations from signal processing modules Will found online.  Another concern was with how the paddle movement would look when we integrated the rPi with the game and how laggy it could possibly look.  Our solution was to restrict sampling to a specific time interval after the ball hit the table and measure the orientation, acceleration, and position data to determine what kind of swing the player was performing.  It would then trigger an already animated swing corresponding to the determined swing type. We categorized swing data to swing types by creating thresholds for each piece of data and setting a threshold to a particular swing type.

I am optimistic that we are heading back on schedule because all I have to do next week is to try to integrate the rPi and Jetson data into the game and optimize certain aspects to make it more playable.  Tomorrow, I am going to be working on creating a bluetooth connection between the Jetson and the game so that on Monday, Logan can just load in his CV code and we can start testing the game in its totality.