Category: Chris’s Status Reports

For this week I worked on a few final things. I added improvements for the motion planning algorithm to allow it to move better between off board and in board, I added a check for when the manual captured pieces are put on the game board, the next turn it moves the piece to the captured area by automatically detecting where it is. Finally, I improved the automatic detection in order to revert back to this and have manual human confirmation, will make sure the accuracy is high enough tomorrow. Finally, added checks for our parcheesi implementation and made sure it could function properly and move accurately physically on the board.

This week I also updated the algorithm to detect shorter paths instead of minimizing blockers, here is an update on Motion Planner:

Parcheesi Region detection works but is not perfectly consistent with detection and with moving the pieces especially because of the strength of the magnets relative to the piece weight. May need to focus on validation and making sure our other systems and chess/checkers gameplay is smooth rather than trying to make all 3 work perfectly.

CV Detection:

In the past 2 weeks, I worked on many different components of the system and a large chunk of my work was on integration.

We had taken out our code from the different systems (Motor, CV, Software) and combined them into a GitHub Repo in order to proceed with further integration.

You can see the progress and commits here: https://github.com/ChrisBernitsas/FlexyBoard/commits/main/

I integrated the CV pipeline, the GUI, the Pi Bridge, and STM32 into one end to end system.

CV Changes:

I changed the CV system so that it used contours to detect groups of changed pixels rather than a straight up diff which proved more useful for lighting changes and with some fine tuning of threshold levels has proven to be very accurate (need to run tests again to obtain another number, but I do not think it has made a mistake when coupled with the game legal move handling). I also incorporated legal move handling for chess and checkers including special moves. Based on the detected squares changed there is a legal move resolved which uses those changed squares to find which move likely occurred. Accompanying this, there is player 1 move validation so weak CV detections are rejected and retried instead of continuing to player 2 in a corrupted state.

Motor Changes:

For testing: I made multiple scripts to test the accuracy in various scenarios. For example, one of these scripts took one chess piece and moved it to every single one of the 64 squares one by one lifting and removing the magnet each time and then returned to its original position to see if it was accurate enough to stay within the chess square each time (although some minor accuracy issues which we can fine tune, it was accurate enough)

Added percentages to our STM move sequencer, this allows instead of strictly a1 -> b2 or 0,0 -> 7,7, we can add percentages if we want to move pieces outside of the chessboard but within the flexyboard (for capturing pieces and other uses). For integrating with the software, I added a software move in the GUI is converted into commands which is sent to the STM which can be executed. For example:

6,7 -> 8.00%,85.71%
8.00%,85.71% -> 8.00%,82.00%
8.00%,82.00% -> 28.57%,82.00%
28.57%,82.00% -> 5,5

Added a capture area in our software which is 3 rows of 10 squares (2 at the bottom of the board where there is a lot of space and 1 at the top). These 30 squares will be used to store captured pieces and eventually be used to move for promoted pieces. The software now has an inventory which it keeps track of which pieces it moves to these captured pieces and has an order in which the 30 squares are used.

Lastly, a major part this week was the algorithm for moving pieces (especially if there are blockers like knight ‘jumping’ over other pieces). This was done with an A* recursive algorithm which turns the board into a graph with nodes and edges and computes paths given the edge weights and the blockers (if a piece blocks a path). The tie breaker for this algorithm is

1. fewer blockers (fewer pieces in the way)

2. shorter path

3. fewer turns

4. straight movement over diagonal movement when distance is equal

I had added the outer area outside the chessboard (but within the flexyboard) as part of this algorithm for path determination. So this means pieces can now move outside the chess board and back into it if there is a path that is not blocked (or if there are blockers those are moved if it is the most optimal path).  Collision validation was added for both in board and out board paths. The off board inventory is treated as obstacles to move around if a piece there exists.

Promotions have not been tested yet.

 

 

 

ADDITIONAL QUESTION:

As you’ve designed, implemented and debugged your project, what new tools or new knowledge did you find it necessary to learn to be able to accomplish these tasks? What learning strategies did you use to acquire this new knowledge?

Over the course of the project there are many new technologies I had to learn. One of the first things was the Raspberry Pi. I had never really used a Pi extensively before. Had to learn the entire process of setting it up (flashing the SD card with the imager), setting up SSH, installing dependencies, etc… For this, one of the biggest things that helped was when I was eating lunch with my friend and asked him questions about Pis (he had used them extensively before) for like 30 minutes. Was able to ask him questions from time to time after that as well, but the majority after set up was straight forward and trial and error. I had never used an accessory connected to a Pi either obviously so had to go through the process of connecting the camera to the pi and watching a tutorial on how to actually get images and process them. I had read documentation as well about this and this helped a lot for the initial small code in the beginning when I was making sure everything was set up properly.

Communication between the Pi and STM was also something I had to learn. I had taken embedded and was pretty confident working with microcontrollers, but having to format movement commands and send them in a parseable format to the STM was something that I read on the raspberry pi forums combined with old Embedded notes and code that I had done.

There were many other parts; however, one of the bigger parts was the A* algorithm. I had been trying for a while to implement pretty rudimentary methods but nothing seemed to be working. Eventually, I decided to build something up from 15-281. I had forgotten a lot of this but was able to relearn it. In order to do this it was a lot of internet, I had watched a lot of YouTube videos to figure out the best approach I wanted and what would actually suit my requirements and ended up reading a few articles I think GeeksForGeeks proved very useful.

In general another part which I had to learn was integrating and testing. There wasn’t really any external source for this that helped. This was just a lot of manual understanding and planning. I think the learning strategy for this part would be classified as learning by trial and error, for each time a subsystem didn’t work as expected or wasn’t working with another subsystem something had to be changed. So definitely learned a lot in terms of making systems work together and integrating them together after they’ve been built individually.

This week, a major part of it was making sure parts worked together and getting ready for the Demos. I made some improvements to the CV system which had included splitting detection into multiple parts. There is now detection after the camera captures an image on the Pi. This includes a region identification for the entire Flexyboard which shows up on the image in a green grid, there is also detection for the subpart which shows up in a yellow grid (and also lays out the 64 squares). Upon improving this system this week, I had also worked on the communication between the Pi and the STM32. The Pi can now send commands to the STM32 as well as move coordinates such as 0,0 -> 7,7 (this system would be useful for chess and checkers but we would need a more precise variant for parcheesi). The upcoming week I plan to move the CV instead of using the 64 squares for chess and checker to feed output move data detection by where it is proportional to the overall board, this will make this system work better for parcheesi. I will also plan by next week to implement the software algorithm to detect if pieces need to be moved out of the way before making a piece move and then moved back. I plan to do this by detecting how many squares have pieces in the way and calculating the minimal amount of pieces that need to be displaced for a piece to make its move (it could be blocked).

Here is an image of the current output:

Extra Question:

In particular, how will you analyze the anticipated measured results to verify your contribution to the project meets the engineering design requirements or the use case requirements?

For the verification part of this, I am focusing on the computer vision (CV) and board state detection subsystem, along with the Pi to STM32 move-command interface. The main requirement for my subsystem is that it correctly detects board state changes and produces accurate move outputs reliably enough for the full FlexyBoard system to function. In our design requirements, one of the main targets is at least 95% correct move detection accuracy, so my verification testing is centered around measuring whether that threshold is met.

To verify this, I will run repeated trials in checkers first, since that game is currently the most functional, and then expand to chess and Parcheesi as the detection method becomes more generalized. For checkers, I will play full games while recording, for every turn:

• the true move made on the board
• the move output predicted by the CV system
• whether the detected move was correct
• whether the board region and square grid were identified correctly
• any failure type, such as missed detection, incorrect source square, incorrect destination square, or piece visibility issue

From these trials, I will compute the following metrics to help quantify the system’s abilities:

• move detection accuracy = number of correct detected moves / total moves tested
• source-square accuracy
• destination-square accuracy
• board localization success rate
• failure frequency by category

I will run enough total moves across multiple games to create a meaningful sample size rather than relying on one short trial. I will compare the final measured accuracy against the 95% design requirement. If the measured value is below that threshold, I will use the logged failure categories to determine the main causes of error, such as poor contrast between black pieces and dark squares, imperfect region detection, or square-mapping issues.

For the Pi to the STM32 communication portion, I will verify that coordinate commands sent by the Pi are received correctly by the STM32 by testing known coordinate pairs and confirming that the expected command is executed. I will record whether each transmitted move matches the commanded move and compute a command transmission success rate. This will verify that the communication subsystem is working reliably enough for integration into the larger project.

These measured results will be analyzed directly against the design requirements which we laid out int he design doc: if move detection accuracy is at least 95%, region detection is stable, and coordinate commands are transmitted correctly and consistently, then my subsystem can be considered verified for its intended role in the project.

 

This week there were a lot of things that happened. We finished testing with one motor which was quite operational. There was some choppiness which we were able to sort out. However when I connected all 4 motors one of the motors seemed to stop functioning well. I am not fully sure if something was shorted or if it’s a current/resistance issue. Will look more into this over the next couple days. Additionally, there was progress made with the CV detection. There were issues with properly identifying the corners of the board as we were still adjusting the size of the board and issues with identifying the pieces within the board. There was progress made in this domain but still not perfect. I set up a communication layer and state machine for the interaction between pi and STM32 although it is not perfect yet as I am still trying to work on getting the CV detection proper and functioning well.

 

CV Update:

This week I was able to connect to the raspberry pi and get some preliminary code down for motor control, the camera has still not been set up and so the task of making the CV work on the actual physical system is still in play. Also the ML algorithm is pushed back a little now. I do not think this is a major issue as I started working on other parts in later weeks now due to the unset up hardware. I also started planning and working on the 3 boards for the software player who will interact with the hardware. Next week I want to have the CV fully down and working and have most of the ML algorithm done so that it will be functional in checkers and Sorry but not necessarily Chess yet. The week after will be finishing motor control and getting the communication between STM and raspberry pi set up.

This week I worked on the CV implementation. I had taken two pictures of a chess board in different positions with my phone camera and processed those and it appears to detect correct between moves. However, because we don’t have the camera and board fully set up on our flexy board I have not been able to test. Next week, I plan to help setting up the camera and making sure the CV algorithm works fully with our project. This is slightly behind schedule, but I think if I can get this part fully working by the end of next week I can move on to our ML implementation for chess after and it will be smoother as the grounds for testing will already be in place. If we haven’t fully set up the camera system to take pictures, I’ll switch my tasks to work on the software GUIs first. I will also work on any code for the interaction between the camera and the raspberry pi next week to try to get that fully set up.

This week a large majority of my work was on the design document. I worked on Sections III and the corresponding block diagram, Sections VI, VII, VIII (including updating and making our gnatt chart more descriptive and BOM), as well as IX, references, and making smaller edits to other sections where required.

 

While this week the majority was dedicated to the design doc and the finalizing our design choices and components, I also begun working on local testing of our CV algorithm that will be used across all 3 board games. I plan to finalize this next week and then begin the ML portion for the piece classification of chess pieces.

This week in classes I did my peer reviews and listened in on the design presentations outside of class I kept considering the raspberry pi’s CV system and how it would determine the moves based on the game and it’s connection to our STM32. I was previously debating having  an initial baseline from the start of the game that will be told by the user. So they will say “chess” and that’s how the system will know what pieces are where exactly in the beginning as to avoid having to do any ML for the games we are on to have piece classifiers. However after our design presentation we had considered we realized we would need a classifier for pieces for all our games any way because there are positions like in chess here a mere diff won’t tell us exactly which piece was taken if there is more than 1 option for taking. Our CV will be Diff based from start of turn to end of turn which will be determined by a button but also have an image classified. I started working on this implementation. The Pi would communicate to the STM by sending the following to the STM:

HOME

MOVE(from_x, from_y, to_x, to_y) or probably more simply MOVE(to_x, to_y)

STOP

the STM would send back:

RECIEVED, DONE, or ERROR

 

I started considering and drawing out our states and the communication between our two controllers

This week toward the start of the week I went over some of the comments from the first presentation and discussed with my team some of the feedback. We discussed we would have probably 2 games which we want our MVP to be for which would make it easier. We then discussed hardware options that we wanted to build our frame and how they would be controlled and connect to the software. I looked for specific components and the microcontrollers we would need for each components (CV and motor control). I was sick for the second half of this week but started considering the best approach for our CV algorithm.  I considered some options of:

1. preloading and telling the algorithm what game we would be playing where it would split the board upon detection into the number of squares required (8×8 for chess) and then compute an occupancy score for each square and determine which pieces have moved

2. We take an image before and after when the user presses a button and just subtract the images, this way we should have blobs where the pieces moved and can determine what pieces are where.

3. an ML model to detect whether each square is classified or could us a YOLO model approach and detect the actual piece if this was for chess for example

4. marker on top of each piece that had a specific (red/orange etc) color and was easy for the camera to detect.

 

I did two main tasks last week. First I worked on the presentation slides and worked on finalizing those for Sunday’s due date. I then started looking into how we would implement a CV algorithm that would work for a set of boards and be able to communicate with the motors for the magnet. I plan on starting to implement this algorithm next week in actual code with a test setup board before we actually make our own. I also added rough costs for how much each component that we need are in the google drive link posted by Harrison. As this is one of the starting weeks and we had the presentations this week, I believe we are on track as long as we get the components we need and start ordering them next week so they arrive on time.