Posted on

Product Pitch

Team A1: Blokus | Members: Jonathan Nee, Aria Zhang, Nadine Bao

Do you love board games? Do you miss playing them with friends in person? Do you find online board games unsatisfying?

Introducing: A socially-distanced game of Blokus, complete with flashing lights, guaranteed to captivate your attention for hours on end. Play with up to 3 other friends, and jostle to be king of the board.

Our Blokus game detects your moves using computer vision and displays all player moves on your board using LEDs. Play Blokus online with friends, interacting with physical board and pieces, from the safety of your home.

Final Demo Video (Link)

 

Posted on

Aria Zhang’s Status Update for 12/5/20

Progress Update:

Last week, over Thanksgiving break I updated the server code so that it has all the functionality promised from back during the beginning of the semester. The main thread of the server acts as the lobby manager, and accepts incoming communication with clients. The clients also send over the lobby number that they want to join, and while the lobby manager server is waiting for replies from clients, it also sends the waiting clients updates to game sessions: whether there is a new game session created by another player or if another player joined a lobby. The games that is displayed to each player is dependent on the currently created games and also the player’s list of saved games. This information is fed back to all the waiting clients in real time. When a session is filled up, the lobby manager spawns a thread and then passes off all the information to the game session thread including whether or not this was a resumed game. If the game is a resumed game, the lobby manager has to load the pieces from that saved game by querying MongoDB. From then on the game session thread is responsible for replaying piece information between the players in its game session and the game manager thread can then keep accepting new connections so multiple ongoing games are possible.

Back to the game session thread, if the game session thread notices that this is a resumed game, then it will replay the pieces that were stored in the database related to the game back to the players, the client software now also has a replay state, where it reads pieces from the server and doesn’t accept any current player inputs until after the game session server sends a end of replay packet. From then on the client software can function as normally. The client software also got an upgrade to the GUI, there are now new display screens so that the software is more user friendly. For example the client can see which game sessions are available to join in real time. If the client has any games saved over, that is also displayed to the client. Note that players only see their own saved games, not the ones of other people. These saved games can only start when the people who were originally in the session all join back into the saved game. Other GUI updates include a win screen and a loading screen. The player can now also “give up” by pressing the “L” key and the player can also choose to save the game by clicking the “S” key. For a game to end all the players have to press the “L” key indicating that they ran out of moves to play. If any player hits the “S” key the game is automatically saved and ended, the game session manager takes all the pieces from the game and persists it to the database, and also updates the players info to include the game session number into the player’s saved game list.

This week I’m working on some more testing and slight modification since I’m developing on a Mac, but my teammates are on Windows and it seems like the sockets work differently than expected especially now when I’ve made the sockets nonblocking. This week I am working to make the server more robust as well as working on the final presentation and filming some gameplay.

Change of Plans:

No change of plans, things are on track.

Deliverables:

  • Upgraded server software
  • Upgrade GUI
  • Replay Game Mechanic
  • Server Deployed with MongoDB C++ driver

Plans for Next Week:

Wrapping up the server testing, and working on the final proposal and filming more videos.

Posted on

Nadine Bao’s Status Update for 12/5/2020

Progress:

In the past two weeks, Aria and I finished integration between Arduino and Client Software. Now, the client software sends 3 chars representing row, col, and color via serial communication to the Arduino.  This was done using the SerialPort Library which allows for communication between C++ and the Arduino’s serial port.

Wire connections broke between two rows of the LED, so I had to replace some LEDs and solder the connections again.

This week also consisted for game testing. We played many rounds of Blokus using. CV piece detection to ensure the setup was working as intended.  The setup consisted of the LED Blokus Board  and camera.  We captured a video and images of our working game.

I also worked on preparing the final presentation. 

Deliverables:

  • Video of gameplay
  • Client + Arduino Integration
  • Board repairs

Tasks for next week:

  • Finish presentation
  • Film and edit demo video

 

Posted on

Team Status Update for 12/4/2020

Over Thanksgiving break, we caught up where we fell slightly behind in, and completed Arduino and client software integration.

This week the team worked on integration between CV and the completed Blokus board with LEDs on a single board (Nadine’s). This is for the final demo. The original plan we had was to show both sending from my board (without holes) to Nadine’s board and show the LEDs lighting up, as well as having a complete setup with CV and LEDs on a single board, but upon some feedback we got we have decided to scrap the former and just go for the latter, as it kind of encompasses both at once.

The network interface was done this week, massive props to Aria for getting everything working on that end, leading to a very smooth user interface that differed from the original Blokus game only in setting up the client connection.

We also worked on getting our final presentation done, and getting some nice images from what we have 🙂

Plans for next week:

We will setup our Blokus board with camera used for CV and film a demo video, as well as start on our report.

Posted on

Jonathan Nee’s Status Update for 12/4/20

Progress Update:

I spent most of this week on integration issues. We faced many this week as a team. Firstly, we had to change the color of the edge detection of the grid from black to purple. This is predominantly a concern because black is pretty common in backgrounds, and just for general robustness and safety we made the change. Secondly, we faced a long delay in setup with CV on Nadine’s computer. CV setup gave me a hard time at the start, and we struggled on it for too long on her computer as well just for the setup. Even after resorting to sending her my entire project, there were still issues. For the purposes of demoing, we are going to have the setup using my computer instead for now, and see if we can get it working with hers eventually. It is difficult to move the setup to my place as well, because the soldering tools are at her place and it would be impossible to fix things up right if they were to break in transit or if there was an accident at my place. Out of an abundance of caution, we are going to handle it this way for now.

The other major time sink this week was the presentation preparation. We took way longer than expected to get a couple of images for the presentation, and for some reason had too little slides even with a lot of content, and had to add more to beef it up.

Other than these issues, I am on track, everything is more on less done on my side, just the final week’s worth of videos and reports to go!

Change of Plans:

None.

Plan for Next Week:

Get presentation videos. Start writing the report.

Posted on

Aria Zhang’s Status Update for 11/21/20

Progress Update:

This week I was working on getting MongoDB set up for the project. The server can now run locally and communicate with MongoDB Atlas and persist pieces that it receives to the database called “capstone”. Inside that database I have a collection called “player_info” which will be used later for the lobby management code to verify an account and also list out what games can be resumed for that particular account. I have another collection called “games” which contains all the games that have been saved aka they have not been finished. Inside this collection, the entries have fields such as the default “_id” which can help us uniquely identify games, “current_player_turn” which has a number 0-3,  “blue”,  “red”, “green”, and “yellow”. The entries with colors in them contain a list of pieces that the server received for a particular color. Again all this piece information is only persisted at the end of the game in order to reduce latency.

I’m running into some slight issues when I try and deploy the same server to EC2. I’m have trouble with using the mongocxx drivers on EC2 so this is something that I unfortunately did not figure out in time for this status pose. However, it is currently able to run locally so hopefully I can figure it out soon.

Change of Plans:

It’s taking me a bit longer to deploy the server + db to EC2, but with thanksgiving coming up I can use the extra time to work on capstone so I think I’m still on track.

Deliverables:

  • Communication between server and MongoDB locally

Plan for Next Week:

Finishing up the server software and make the UI more user friendly as suggested in the midpoint demo.

Posted on

Team Status Update for 11/21/2020

This week the team worked on integration between  CV and the completed Blokus board with LEDs. We were able to accurately detect pieces when the LEDs were on. The slight bleeding from the LEDs did not lead to incorrect piece detection.

Board construction was finalized this week, although repairs may be necessary as the soldered connections may break when the Blokus board is being setup or adjusted.

Optimizations in CV detection were also make to help us fall under our latency goal.  This mainly involved removing steps in image processing that would not drastically our piece detection accuracy.

Plans for next week:

We are a behind on Arduino and client software integration, so we will push to get that finished as soon as possible. Afterwards, we will setup our Blokus board with camera used for CV and film a demo video.

Posted on

Nadine Bao’s Status Update for 11/21/2020

Progress:

This week I was able to integrate the LED lights into the Blokus board. This process consisted of attaching and positioning LEDs on to a square board that is then placed under the Blokus boards. The main challenging was lining up the lights with the holes on the board. Some of the holes do not match the lights perfectly, but we do not think it greatly impacts the player’s ability to indicate which square is lit up.

I was also able to meet up with Jonathan to test out how the LED lights would impact the computer vision. Once thing we did not expect to be a problem was the holes on the board. The CV detects these wholes as black which is the same color it uses to find the four corners of the board. This is fixed by changing the color used to detect the corner. For example, in our trial,  we changed it to purple because purple is not a player color or a color in the game’s environment (ex. table, floor). We also checked if the slight bleeding from each LED would affect whether or not the CV would detect coordinates that were not part of the players move. This is important in the case that that we decide to light up the LEDs under the player’s moves. Bleeding did not affect the correctness of coordinates read as it did not surpass the threshold for coverage of a square. Similar tests were done to see if player color would be accurately detected even when player pieces are surrounded by different colored lights. Player pieces were still accurately detected.

Deliverables:

  • Working Blokus board.
  • Testing with CV.

Tasks for next week:

We are behind on the serial communication between  Arduino and client game code.  To account for this, we will put in more hours during Thanksgiving break. After serial communication is successful, we should have a playable  game experience to demonstrate.

Posted on

Jonathan Nee’s Status Update for 11/21/20

Progress Update:

I have created a sheet here as I track how to get the times down. As suggested, I am creating 2 versions, 1 without first reducing the size of the image, that is more efficient in terms of timing. However, this image is less appropriate and takes up too much room on the screen for the demonstration, so for purposes of the demonstration only, we will keep the scaled image as represented by the slow time in the left column. So far, by removing that as well as the morphological transform of the black detection corners, I have gotten the time to within the range we want, from 60-120ms. This “fast” version of the code still retains accuracy of the image detection and there is no compromise on accuracy. The morphological transformation is no longer required as the image is bigger and the pixels already provide a good enough sample to get bounding rectangles from. In addition, the reason why the time for calculating if each cell is red has in fact dropped is due to additional testing as to how few pixels are required to accurately determine if each cell is red or not, and currently for the fast version, by ignoring the 5 pixels on the borders again accuracy is not compromized. This could be further reduced, but it does not really add to the bottom line significantly. Further reductions in time would probably have to come from hsv or filtering colors, both of which are necessary. I could try downres instead of resizing which might be more efficient next, but I suspect that would not be that great a time saving either. I am currently looking into whether rgb would provide a good enough range for the color choices I need, given how different the colors are from each other for this project.

In addition, I also met with Nadine to test out how detection works for the LEDs. This testing is now important because we are considering lighting up LEDs of the respective colors when a valid piece has been placed, as a way to show when the system has registered the piece placement for the other players. As such, we need to ensure any minimal bleeding does not end up detecting additional squares. Our testing currently shows that the area being detected is very small, and that it should be entirely possible to set a higher threshold for each grid cell to exclude any issues from bleeding.

Change of Plans:

None. Still pushing towards getting that time down.

Plan for Next Week:

Start writing the report and getting presentation videos. Take a short break for Thanksgiving.

Posted on

Aria Zhang’s Status Update for 11/14/20

Progress Update:

This week I worked on deploying the server code so that the clients can connect to an EC2 instance that is running the server which forwards a player’s move to all the other players in the game. I was going to try and do an intermediate step and run the server locally for our demo, but I realized that clients outside my network would not be able to connect to a local server because it would be behind a firewall, and because I didn’t have port forwarding set up. So I just made the jump and deployed the server on an EC2 instance. Unfortunately, Windows doesn’t use sockets which my client code uses to communicate with the server so Jonathan had to make some adjustments to support Windows and we did not do that in time for the demo. I was still able to demo the communication to the server by running two clients locally. I’ve started writing more code to set up and communicate with the MongoDB database so I’m hoping to have that wrapped up by next week. I am also currently in the middle of integrating the communication between the Arduino and the client software with Nadine and hopefully we will have that done ASAP.

I also want to discuss some of the comments that we got during our midpoint demo. It was mentioned that we should have some way of letting the player know what moves were registered and what moves were not registered. So that in the case the board was somehow knocked over the player would be able to set the board back up. Our team discussed this and we think the best solution would be to also light up the LEDs under the player’s own pieces. That way it tells the player which pieces exactly are not being processed due to an illegal move. I think the GUI feature can also help the player figure out what pieces are committed, but I think lighting up the player’s pieces under the board would also be helpful.

It was also mentioned that we should have a feature that lets the player undo their move, but I don’t think that is really a necessary feature to have. The feature would just be adding a delay in the software before sending the piece to the server allowing the player to remove the latest piece committed within a certain time frame. However, I don’t feel like this is adding anything to the game besides more latency. Realistically if we do implement this feature we would need to give the player at least a few seconds since if a player was unsure about their move then I don’t think they would be able to change in under a second. That means for every move, the player would have to wait a few seconds before their move is sent to the server even if they want to commit their move. Another way we could implement this feature is to have some button to commit a move, but that defeats the purpose of the project because at that point then we don’t even need to sample the video we could just use the frame when the player clicks some sort of button. I think this would make the game feel clunkier and less interactive overall so I don’t think an undo feature should be added.

Change of Plans:

I think I am on track with the schedule.

Deliverables:

  • Deploying server code to EC2
  • Beginning to set up MongoDB
  • Integration with Arduino is ongoing

Plan for Next Week:

I hope to have the MongoDB database set up and to start working on the lobby management code.