Right now what we have to do is build the paddle, integrate the rPi and Jetson to the server, finish up the final graphics, finish the computer vision, and add spin capabilities to the game. Our biggest concern right now is the bluetooth connection between the Jetson and the server because Logan is still working on the computer vision stuff and was not able to set up the Jetson and its bluetooth capabilities. So Henry is going to try to get that set up this week. For the bluetooth connection with the rPi, we had the bluetooth connection working temporarily, but it stopped working during the demo. We think we have located the problem so that is something we are going fix and do more extensive testing on next week. Henry is also going to try to finish up the final graphics next week, which means adding the VR processing to be compatible with the Google Cardboard and creating more realistic paddles, balls, tables, and better camera angles. William is working on getting accurate acceleration and velocity data from his IMU so that players can add spin to the ball.
Logan’s Status Report for 4/10
During the start of the week I continued to test the CV with video to see what kind of performance we could get at full framerate. Since we were not able to power the Jetson, I used my laptop. While it doesn’t have the same specs as the Jetson, it would appear that computationally, we should be able to use at least half of the maximum framerate without latency issues. The next step is to test whether our transmission capabilities will also allow this, and then see how the Jetson performs.
I also requested the power supply part for the Jetson.
Henry’s Status Report 04/10
This week, I was working with William to get the rPi connected with the game. On Tuesday we were able to send rotational vectors between the rPi and a Mac and print those values out, but on Wednesday during our demo it didn’t work. The reason why is because when I was importing the game onto my phone, I had to change a lot of build settings, which affected the bluetooth capabilities. I also began to work on the final version of the graphics this week: incorporating a VR library that helps process the game to be viewed on a Google Cardboard, and designing the tables, paddles, and camera perspective for the final game. The plan is to get this all implemented by Thursday or Friday. What I am also going to be working on next week is setting up the Jetson Nano. Logan hasn’t been able to work on it these past few weeks, so I’m going to take it since time is beginning to run out. Hopefully by the end of next week I should be able to have the Jetson Nano set up and add bluetooth capabilities to it. That way I can test the Bluetooth connection between the Jetson and the server. I am behind schedule right now, but given the relatively light week I have upcoming, I am ready to put over 20 hours working on what I laid out above.
William’s Status Report for 4/9
This week, I was able to finally get the Bluetooth module up and running on my raspberry pi. The pi can now listen for connections on startup, and then once a device connects to the pi, the pi can send data from the IMU to that device. I tested this with a bluetooth terminal on my phone, and the bluetooth has been reliable. When I tested with the Unity module with Henry, however, it has not consistently connected, so we still need to trouble shoot why that is the case. Moving forward, I will need to also test how the latency works in real time with the sending of data between devices.
I have also begun looking into ways to extract velocity and maybe even position data from the IMU on top of the current orientation data. I am researching how to use quaternions and rotation matrices to make the IMU data relate to the global axes as opposed to just the IMU axes. This is something I will have to look into more, and then also implement and test moving forward.
Team Status Report for 03/02
As of right now, we are beginning to fall behind on a few tasks. We need to get the Jetson and Raspberry Pi integrated, we need to get the paddle built, and we need to add the aspect of spin into our game. The biggest risk that we are looking at is the integration of the microcontrollers with the Unity game engine. Henry and William will test to see if the Raspberry Pi can integrate with Unity on Monday, and if it cannot work, we will have two alternate approaches. The first is to add a new bluetooth module to the Raspberry Pi that allows connection with the bluetooth library we are using on Unity. The second is to write code ourselves that creates Bluetooth connections.
Even with these hiccups, we don’t see any reason to change the system of our product at all, and our schedule is going to remain the same since we foresaw possible difficulties with the integration. Referencing Henry’s post, he is going to finish integration of the Raspberry Pi and Jetson within 2 weeks. William is going to be working on building the paddle and helping Henry with integrating the Raspberry Pi. And Logan is going to finish up his OpenCV code and work on adding the spin capabilities to the game as well.
Henry’s Status Report for 04/02
I started this week with the intention of integrating the Jetson into the game. I spent a while first adding the Arduino Bluetooth library into our Unity code which should give us the ability for the Raspberry Pi and Jetson to send its data to the code. However, this is where I ran into a few pretty big roadblocks. The first one, is that when I was trying to connect to the Jetson, I realized that Logan had not set it up yet. So, something that we have to work on next week is getting the Jetson’s OS set up and get the Bluetooth drivers and capabilities set up to see if it can connect with our game. Another roadblock/uncertainty is whether the Arduino Library will work or not. The library is only compatible with certain Bluetooth modules, and I didn’t have anything to test with that had those modules. The only testing I could really do was to see if my code would know if my Airpods were paired with my Mac or not, which it could. So, on Monday, I am going to work with William to see if we can get the rPi connected. If we can’t, I’m going to either have to buy a bluetooth module compatible with the Bluetooth library, or I have to write my own code using NET to connect with Bluetooth devices.
Because there were roadblocks with integrating devices into the game, I decided to refine my Unity code more. The first thing I did was removing the GUI for determining whether we were running code as the server or as the client. It’s unnecessary work for users and going to be extra work for us in the future since we would’ve had to add some VR pointing capabilities since our game doesn’t have a mouse and keyboard. The second thing I did was getting the code ready for Raspberry Pi and Jetson inputs. I added commented out code that sets the paddle location and orientation to these inputs and code that blocks until both pieces of data are received, so that the paddle input data is synchronized.
For my responsibilities, I am falling behind a little bit. On the Gantt chart, it mentions that I have all of April to integrate the devices, but I want to have everything integrated a few weeks early so I can help out with other stuff we are behind on like building the paddle or adding spin to the game. The goal to to get the Jetson and Raspberry Pi integrated into the system within the next two weeks. That means that we can connect to both devices, send/receive data, and have this data be completely synchronized with low latency.
Logan’s Status Report for 4/2
This week I was able to get the CV portion of our MVP working on some test images, and start testing the video input of the CV code. The performance on video input (given the numerous frames) is potentially an issue, a possible solution is to ignore most of the frames, only looking at every third (for example) frame. This should still give us enough resolution to get information on the swing while cutting down on the data we need to process.
William’s Status Report for 4/2
This week, I began trying to integrate the bluetooth on raspberry pi so it can send data to the VR device and server for the MVP. I’ve found some Rpi resources like bluez and bluetoothctl that can be used, but I’m still facing some technical difficulties connecting to devices, as sometimes the connection does not stay after pairing. This is something I am still looking into still. I also researched the mechanism for sending data, and I think I have found the plan of writing to a file that will be read by the bluetooth module. I plan to test out the data transfer using a bluetooth terminal app on my phone when I get the bluetooth working.
My goal is to get the bluetooth up and running as soon as possible next week so I can integrate it with the other subsystems and begin seeing how the IMU data translates to the graphics.
Team Status Report for 03/26
As of the end of this week, the individual components of the MVP are completed and next week we are going to finish integrating the components together. The IMU isn’t a part of the MVP, but there has been considerable progress in making the data output more consistent and accurate. The game itself has been completed too. What’s new from this week are the multiplayer and bluetooth capabilities that allow two players to play each other and connect their paddles to the game. The computer vision algorithm for tracking the lateral position of the paddle is about completed as well. Looking ahead, the plan for next week is to send the lateral position of the paddle to the Google Cardboards running the game, so that two players can control their respective paddles in a game of Pong. We are projecting there to be a few calibration adjustments to allow for centering of the paddle and we are going to test for the latency of the paddle movement as well. For this week, no changes were made to the product design.
The biggest risk that we face is buying a wifi card that either isn’t compatible with the libraries we are using or the Jetson Nano itself. While this risk is mitigated with good product research, a contingency plan we have is to buy multiple wifi cards at a time so we have backups in case one fails.
Logan’s Status Report for 3/26
This week I continued to work on the CV with our new strategy of color recognition as opposed to the previous strategy of contours. Around the middle of the week I ran into some issues but I reached out to Professor Savvides for help and he gave me some tips about hue/HSV and pointed me to some resources which continue to be helpful.
Despite the unforeseen challenges this portion of the MVP should be done by Monday so we can proceed with integrating next week.