Team B1: Ultimate Chess

I spent most of my time this week testing CV on new images from the webcam. The webcam image quality was much lower than that of our chessboard, so I had to experiment with and change parameters for hough_line to discretise the image of the chessboard. The discretisation into squares works now. The first image below is that of the line detection for the webcam image, and the second is the extracted square.

The moves for the tested images are detected correctly 18/20 times. One of the wrong detections is for castling, which Yoorae and I will work on this week.

 

 

. 

 

I also worked on integrating CV with the chess game logic that Yoorae developed. I set up a capstone-integrated repo so Demi can also add her code there. I created a script that lets the user specify the path to 2 images. These images are then cropped by the neural network and CV detects the move. Then this move is validated by Yoorae’s code. It takes the current state of the chessboard (2d array) and the initial_position and final_position of the piece that moved.  If the move is valid, I update the current state of the board.

This week, I plan to refine the script to deal with castling and promotion.  I will also generate more testing metrics for the CV move detection. I am on track with the schedule.

 

.

This week we finished up our individual works and began to focus on integrating our work. Demi recreated the bottom cardboards of the board and installed the LED matrix with the new LED we purchased. She connected the push button to RPi so that user can signal the move is done. Demi also set up the webcam and took sample photos to solidify our CV, and she focused on integration of our project with Stockfish AI. Anoushka also worked on AI and CV integration. So far, we did not have a webcam stand and we have been testing our CV from sample photos we took from our phones. After we set up the webcam, we ran into some issues with CV because the photos from webcam had lower resolution photos. Yoorae and Anoushka spent time on fixing the errors and debugging CV with the new sample photos from the webcam, and testing the new sample photos.

We made a purchase of red chess pieces set to solidify our change detection algorithm for CV. Yoorae wrote change detection code with red piece set.

This week I made an update on Chess Game logic to make testing easier. I added some modules that make changes on the board state, return selected positions, and checks the empty spaces, to help the testing process. I also include test cases on the test bench.

I also focused more on CV this week. I concentrated on detecting the coordinate of the board where a change has been made, while Anoushka focused on the better quality of grid lines through neural net training. The quality of initial sample images were not good enough for detecting changes, since the coordinate of the board center was inconsistent throughout the states. The previous sample photos had inconsistent grid lines, which made change detection impossible. So I took another sample photos at a lab on Wednesday in a more consistent setting with camera held in one place. I edited new sample photos through CV so that the coordinates of each square is aligned.

I also made an update on Hough line threshold Anoushka’s code that get the gridlines from edge detection so that the correct grid lines will be generated from more varying photos.  The most significant obstacle from our CV change detection is inconsistent reflection on our board. Since our physical board is made out of glassy acrylic material, anything that gets in the way between the light source and the board gets reflected on the board (such as user’s hand, camera, user’s hair and etc).

From the photo above, change detection module should output coordinate (4, 4) and (4, 6). However, no matter what method I try, (abs diff, structural similarity, background subtraction method on grey scale, RGB scale, HSV scale) threshold could not separate coordinates without changes from coordinates with changes, because the output difference value of the coordinate with different reflection on the acrylic piece was too high.

This is a huge problem in our CV because reflection on our board is going to be inconsistent throughout the whole game. I updated a working code that computes the difference with limited size of the square (at center) so that it can disregard difference from background reflection. However, this code will not work if the user places a piece away from the center of the square. The other solution will be to remake the board with an unreflecting material, such as silicon. For another solution approach, both me and Anoushka tried to blur out the reflection, but was not achievable.

Next week I will mainly focus on coming up with a solution approach to this reflection problem and to come up with a stable change detection algorithm.

My goal for this week was to have the LED matrix installed. Since the spacing between the LEDs do not match the length of our chessboard grid, I had to cut the strip into 64 LEDs and solder them. So far I have soldered 42 LEDs together. The soldering process is very time consuming and tedious as each LED have 3 connections.

I tested the chess game logic Yoorae wrote. I wrote simple test cases and found that there was an error with checking for vacant squares along the path. It should only check for the squares between the start and end position but was also checking whether the start square was vacant. This issue is now fixed.

Since the LED matrix is not completed, I am behind schedule. Completing the LEDs is my priority, and I am confident to have it done by Wednesday’s demo. For the rest of the week, I will begin working with Anoushka with CV / board integration and camera setup. I hope to get the board completely done next week, so I can focus on integration and testing for the rest of the semester.

 

This week Anoushka began testing CV and Yoorae worked on her Chess logic. Demi worked on soldering the LEDs and installing the LED matrix. She will be soldering next week as well. After talking to the professors, Anoushka and Yoorae discussed integration and figured out what inputs Anoushka needs to provide to Yoorae for her chess logic. Yoorae plans to work on CV with Anoushka next week.  We plan to focus on the demo the first half of the week and then work on integration. We are slightly behind as we haven’t started integration fully yet, but we plan to work on it the second half of next week.

The main risk factor is still CV because we have issues with noise making it hard to detect changes in squares. Our webcam stand arrives next week and is likely to help with this by providing steady images.

This week I spent most of my time working on using CV on actual images of the chessboard that Demi made. I ran into some issues initially due to us not being able to hold the camera steady and straight when taking pictures of the board. The webcam stand will help with some of these problems, but I decided to use a neural network to refine the images so I can test even with images that may not be very good. This also gives us some flexibility with images.

The neural network I used is https://github.com/maciejczyzewski/neural-chessboard. This open-source model takes in chess board images, cleans them up and provides a zoomed in and focused image of the board. I ran into troubles with installation and use initially because the tensorflow and Keras versions were not compatible. I was able to solve this by using a Conda virtual environment and directly installing the requirements of the model.

Since our images weren’t good (tilted, camera moved between images etc), I had to train for 60 epochs to get decent results. I varied my num_epochs between 40 and 70 to find the optimum number for our images. There was not much improvement after 60, so I settled on that value. A sample result is below:

Once this was done, I was finally able to get to testing. This week I tested division of the chessboard into squares and extraction of pieces since this is the most error prone part. All squares were detected properly for 20 trials.  Examples below:

 

 

2.

The next task is testing how well the code can figure out the difference between squares at t-1 and t. The image below shows a sample of the concatenated board at t=0 and t=1. The issue I am running into here is that neither structural similarity nor mean squared error is returning a steady estimate of which squares had changes. This could be due to the movement of the camera in between. I am combating this by using Background subtraction method so that changes in the background are not counted as “structural dissimilarity”.

I plan on working with Yoorae on this for the next 1-2 days. After that, I will begin working more on integration with Yoorae’s chess logic and ensuring I provide her the right inputs.

I am on track with what I wanted to do this week and am hopefully able to resolve the hurdle with square change detection by the beginning of next week with Yoorae. For next week, my deliverable is more test metrics for the actual square change detection. I will also work with Yoorae on integrating her parts with mine.

This week, in a team meeting with Tamal and professor Bryon on Monday, professor has pointed out the edge cases that I have missed for the chess game logic. To update the Chess game logic to take account of every possible edge cases that are possible, I researched the chess rules in depth and I made an update on the struct of board state to return the location of each piece name and the piece color to take account of every possible edge cases in the chess game. The board state struct will also contain an int counter to take account of number of turns that have been played, to check the allowed moves for pawn. This will make the check valid process a bit slower, but it is necessary to correctly process many edge cases in the chess game such as  ‘en passant’ state of the pawns and ‘castling’ state (in which two pieces can move in one turn).

 

I also wrote test benches to check the correctness of my current structure. The current test bench consists of 50 general moves, and each special move per edge case. My logic is currently returning a 100% correctness according to the test bench results.

This week, we continued to work on the three main areas of our project. Creating our custom chessboard, detecting moves with CV, and implementing the chess logic to check for valid moves. We are still working individually but am starting to think about integration on what information needs to be communicated between different modules.

There was one challenge regarding the sample chessboard for CV testing. Working with two chessboards did not work out as well as we expected. The sample chessboard was supposed to be taped together because gluing takes a long time. However, it was difficult to keep the pieces together with tape, and the gaps between pieces made edge detection hard. To mitigate this challenge, we decided to keep the main chessboard in the lab instead of having it in Demi’s apartment for the rest of the semester. Until this week, we mostly worked remotely in our apartments, but from now on we plan to work together in the lab with the main board.

There are no major changes made to our schedule.

This week, I mainly worked on making the case for the chessboard. I chose to use cardboard for the first prototype because cardboard is cheap and easy to cut and drill. I created dxf files using SolidWorks and went to Tech Spark to laser cut them. I also finished gluing the chessboard top, ordered a push button, and connected the LEDs to the RPi and tested controlling the LEDs using the NeoPixel library.

Below is an image of the grids and case glued together with the chessboard on top. The grids are used to separate the light from one LED to other LEDs to ensure that only the correct squares light up. The grids have small holes on the bottom to allow the LED strips to go through. I still have to make the top cover of the electronic case and install the LEDs and push button.

      

Next week’s goal is to have the full 8×8 LED matrix installed under the board. This will require cutting the LED strip into 64 LEDs and soldering them together. I will test the LEDs lighting up correctly given a coordinate. I will also install the push button once it arrives.

This week I spent time gluing and taping the sample chessboard pieces Demi gave me. This took a significant amount of time because I had to remove the wraps and place the pieces properly so there aren’t gaps between them which would make CV hard.

I also tested my algorithm on images of the actual chessboard that Demi sent me. Attached is an image of the results. The first is the original image, the second is with Canny edge detection. The third one contains blue lines which are the slope 0 or infinity lines in the Hough transform. The yellow color on the squares indicates that these are the squares that have been formed by the line. This works as intended.

Another thing I worked on was ensuring that the pieces are detected properly and the contrast is high enough in our board. This worked fine, but I had to ensure that I use the unblurred image for this because otherwise some of the edges don’t remain sharp anymore. I use the blurred for detecting squares so that the image is less noisy, but the pieces are sometimes considered noise. This result is with the unblurred image. All pieces are detected. The actual edges detected don’t matter in our case, we just want to ensure some edges are detected for each piece.

Because Demi has the board I was unable to test on it with actual pieces, but I have been tested on my imperfect sample board. The sample board is harder to work with because the best pieces were picked for the main board so the ones for the sample board aren’t cut as properly.

I wasn’t able to deliver the test metrics on the actual board because I don’t have it yet. I already talked to Byron and Tamal about this and we decided I would give the chess pieces to Demi and ask her to take pictures because she has the board. On Monday, I will give the pieces to Demi. I am also requesting her to bring the board on Monday so I am able to get some time with it on campus and get the pictures I need.

This week, I plan on forming metrics for the actual chessboard on Monday. I am one day behind schedule, but will be caught up on beginning of next week.

After that, I plan to figure how to get the metrics higher if needed and determine if we need any minimum lighting requirements. I will also look into the metric regarding how “inside” a square a piece needs to be to be detected properly.  This is a problem with the corners of the board because the tall pieces often have edges outside of the square due to the fact that we are looking top-down. I will look into ways to mitigate this.