This week was spent on a lot of debugging related to handshaking and synchronization – we ran into some real-world issues with crosstalk and timing that were highly time consuming to resolve, along with some flaws from how I implemented sequence numbers.
Design/Implementation related bugs:
I was updating the sender’s sequence number when it received new data from the game logic, which caused an issue with packets being duplicated on timeout with the same data but an updated sequence number. I changed sequence number updates to occur upon receiving an ACK instead.
My control FSM would also still resend packets after receiving an acknowledgement, so I added an extra state that stops the timeout counter upon receiving an ACK.
Lastly, the Receiver and its FSM implementations needed some small changes to correctly handle game start/end synchronization. When a player tops out, that game instance sends a game_end signal over the handshake line and enter a GAME OVER state, where they continue to resend that signal until they receive an ACK. The other game would simply return to IDLE upon receiving the signal, but wouldn’t have enough time to send the ACK due to my implementation. I added an explicit GAME_WON state where the instance will remain for a set number of cycles, allowing for several ACKs to be sent before returning to IDLE.
Real world bugs:
Acks and sequence numbers were occasionally being dropped or misinterpreted due to crosstalk causing bitflips. Both were resolved by summing the bits of the field and rounding the sum for more flexible interpretation.
Game start synchronization occurs by both players entering a ready state, where they constantly send ACKs over the handshaking line until they receive one in turn. This was causing an issue where another ACK in flight would be received after the game had started, making the sender/receiver pair have desync’d sequence numbers. I implemented a mechanic to send an ACK upon receiving a packet with the wrong sequence number, effectively dropping a packet but realigning the sequence numbers. This should only occur in the first frame of the game where it is impossible for any lines to be sent, so we shouldn’t lose any garbage lines being sent.
Upon receiving a data packet, the receiver tells the sender to send an ACK in return. This ACK was not being picked up by the receiver on the other end. I suspect this was some sort of timing issue, although everything looked correct and functioned in simulation. This was resolved after much testing by adding a set delay before sending the ACK.
Some other misc issues/changes included adding synchronization chains to the GPIO inputs, and some weirdness with the FPGA keys and getting a proper single-cycle pulse out of them.
With that out of the way we can finish integration and truly test everything together, with multiplayer updating on both boards and game start/end synchronization.