Posted on

Yoyo’s status report for 4/25

This week I worked on system integration with my teammates. On the physical board we added frosted acrylic pieces in the holes on the board for led light diffusion and for cv to not recognize the leds through the hole as pieces.

On computer vision part everything is integrated to the system, we decided to avoid led flashing and add a button in the UI for the users to indicate they are done with a move and the leds will turn off for cv to capture the frame and then turn back on to guide the users for the next move. So there are changes made in the cv code for receiving frame capture requests.

For system integration we switched from hotspot to cmu device to connect everything. we will continue to debug and finalize the whole system especially the UI.

I am on track with schedule and next week we will be doing user testing.

Posted on

Charlie’s Status Report for 4/25

This week, I focused on revising the server architecture and refactoring the state bridge to address limitations from the previous single-channel WebSocket model between the web client and Rust engine. While the earlier design worked for the React frontend, integrating additional hardware components such as CV-based board state detection and LED control required a more robust distributed architecture to support asymmetric clients. The updated design establishes the bridge as the single orchestrator and external gateway, with the engine reduced to pure game logic and stateless analysis. All components—including UI, CV, LED, and coaching—now interact through a unified bridge-owned command/event model, ensuring consistency and eliminating split-brain state. This redesign also ensures that all authoritative moves are broadcast as events, prevents helper operations from mutating live state, and enables CV-detected moves to act as first-class command inputs. I also revised the LED light configuration on the client to better match the physical board design, improving visual alignment between digital and hardware feedback.

In parallel, I continued developing the agent orchestration system, implementing a fast-path execution flow and iterating on decision routing based on board state. I integrated the Go-based coaching service into the new architecture and expanded integration testing across the bridge, client, and agent layers. I also conducted additional testing on the RAG pipeline, fixing issues with context injection and improving prompt constraints for token usage. . For better agent observability, I added a visualization and logging panel to display tool usage and track all agent actions in real time. On the data side, I collected more game position samples for puzzle generation and began revising the puzzle curator agent and state detection logic.

Overall, my progress is mostly on track, with continued focus on stabilizing the new architecture and improving system reliability.

Posted on

Team’s Status Report for 4/25

Unit Tests:

  • CV Pipeline
    • Tested piece detection on single-piece and multi-piece images
    • Verified correct grid mapping after ArUco-based warping
    • Checked FEN consistency across repeated captures
  • Chess Engine
    • Validated legal move generation across edge cases (check, capture, invalid moves)
    • Verified no illegal moves are returned to downstream modules
  • LED System
    • Tested LED addressing and color mapping correctness
    • Verified correct highlighting for legal moves and best move
  • LLM Pipeline
    • Tested that generated suggestions are always legal (engine-bounded)
    • Verified latency under different prompt lengths

We conducted end-to-end system testing under realistic gameplay scenarios.

We found that LLM latency was quite long which made the overall latency of running the whole pipeline longer than we wanted, therefore we decided to use a token-limited setting and was able to boost the latency.

The most significant risks that could jeopardize the success of the project is the Raspberry Pi needing to be reflashed right before the demo, because we have faced some instances were we are forced to reflash and reupload our code. Another risk is our hotspot not working fast enough or at all in the location of the demo, which is needed by all 3 of our subsystems.

There were no changes to the existing design of the system.

Posted on

Claire’s Status Report for 4/25

This week I finished the verification of the LED subsystem. I tested the remaining 3 full games for the move suggestions and the 30 board states for the latency and move suggestion accuracy. These last three games ended up having 99% average accuracy and 185ms average latency, which meets the <200ms metric we were aiming for. I also added tests to tell the latency between when the chess engine send an API request to the LED subsystem to turn on the opponent’s moves and when the LEDs are actually turned on. This is in addition for the 7 full games and 70 board states I collected last week where we had slightly less accuracy at 98% and also around a 185ms average latency. We did not make any design changes after the analysis of these test results since these meet the metric we were aiming for and we thought the experience of playing the game was how we envisioned. One thing we did talk about adding is a button for when the player is done making either their move or the opponent’s move so that the LEDs could be turned off for the CV, which hasn’t been trained with the LEDs lit under the pieces.

My progress is on schedule. Next week, I hope to finish the report and poster to be ready for the demo.

Posted on

Yoyo’s status report for 4/18

This week I mainly focused on integrating the whole subsystem and with the other subsystems.

The cv part now produces results like this which can distinguish all different pieces.  The subsystem now outputs a request to the LED system for the leds to turn on and off when capturing a frame, and a FEN string to the integrate server as a result of board state detection.

I’ve also discovered some small changes that need to be made to the design of the physical board. like the need to raise the leds as close to the board as possible for the light to be seen clearly by the user.

Next week I will continue to address those small changes to improve user experience.

About new knowledge for this project, this is my first time learning about computer vision, I learned not only how to use the code for YOLO models, but also how different things would affect the outcomes of the models, how I should take the photos for the datasets to address different edge cases, how different yolo models work. About learning strategies I learned a lot by reading the official documents of yolo and anylabeling(the tool I used to label datasets), blogs about similar chess board state detection projects, and I also got help from Chatgpt when YOLOX didn’t support mps on macOS much and I had to learn to modify the code for yolox to support mps.

Posted on

Team’s Status Report for 4/18

The most significant risks that could jeopardize the success of the project is the verification and integration of the subsystems. All of the subsystems are integrated in pairs of two like the computer vision with the LED subsystem and the chess engine with the LED subsystem and the computer vision with the chess engine, but we still need to verify that all of the subsystem can all function as one. We are also slightly worried about our hotspot speed, brightness of the room, and height of the table because all of these factors during the demo can affect the experience of our project.

There were no changed to the existing design of the system.

Posted on

Claire’s Status Report for 4/18

This week I integrated the LED subsystem with the computer vision system and the chess engine. I formatted my functions that can show the move suggestions and the opponent’s moves as API function calls so that the integration engine could send requests in real time. I also started the verification process by testing 7 full games and 70 board states for the latency and the accuracy of the move suggestions.

We are on schedule. The next week I hope to finish the verification of the integration of all the subsystems like testing the remaining 3 full games for the move suggestions and the 30 board states for the latency and move suggestion accuracy.

As I’ve designed, implemented and debugged my project a new tool I learned to use was the Raspberry Pi Imager to flash my Pi and upload code onto it. I read Intro to Robotic’s documentation on how to connect to a Raspberry Pi as a learning strategy to acquire this new knowledge.

Posted on

Charlie’s Status Report for 4/18

This week, I rewrote the agent orchestration framework in Go, introducing structured agent state management along with defined skills and tools integrated with the engine. This significantly improved observability and debugging, as the system now uses Prometheus and Grafana along with a dynamic React dashboard to visualize agent execution, parallel node activity, and tool usage in real time. In addition, to improve user interaction beyond speech-to-text input, I developed a virtual avatar using Three.js that maps engine evaluations to animations, providing more intuitive and engaging feedback to the user during gameplay.

During development, I identified limitations in the current embedding strategy for fine-tuning, particularly with Qwen 7B, where dense vectors made it difficult to align outputs with evaluation signals. As a result, I pivoted toward a heuristic-first approach for move explanation generation, incorporating factors such as game phase, pawn structure, center control, and king safety. I also implemented a LoRA fine-tuning pipeline using expert commentary data, but am delaying full training until the heuristic mapping layer is complete.

Throughout this capstone, I learned several new tools and concepts, including agentic frameworks, observability tools like Prometheus and Grafana, and LoRA fine-tuning techniques. I primarily relied on informal learning strategies such as reading Medium articles, technical blogs, and documentation to quickly get up to speed and apply these tools effectively.

Posted on

Charlie’s Status Report for 4/4

This week, I continued working on the LLM agent orchestration pipeline and prompt engineering, focusing on improving reasoning quality and reducing hallucinations in move explanations. I also explored deployment options using Amazon Bedrock and researched LoRA fine-tuning as a potential approach for adapting models to the Xiangqi domain. Through this investigation, I identified a key limitation in standard LLMs: lack of spatial awareness when interpreting raw board representations such as FEN. To address this, I designed a feature engineering approach that converts board states into a structured relational representation, encoding piece interactions, threat maps, palace dynamics, and pawn structures. This effectively transforms the problem from spatial reasoning into a relational reasoning task, which aligns better with transformer-based models.

Additionally, I began prototyping this relational preprocessing pipeline and integrating it into the agent workflow alongside RAG. The goal is to provide the LLM with structured, verifiable context to improve consistency and interpretability of generated explanations. This approach reduces reliance on implicit spatial reasoning and instead supplies explicit constraints derived from the engine, helping mitigate hallucinations. Moving forward, I plan to refine the feature extraction process, validate its impact on explanation accuracy, and continue improving prompt design and integration with the full system pipeline.

I plan to evaluate the LLM orchestration system using  multiple metrics and  structured logging and agent visualization. First, I will measure explanation correctness by checking whether the LLM’s suggested moves and reasoning align with the engine’s validated outputs. I will also track hallucination rate, calculated from the percentage of responses that include invalid moves, incorrect checkmate claims, or inconsistencies with the game state. I will also evaluate consistency by running repeated prompts on the same position and measuring differences in outputs. For performance, time between text and move input and when response is returned  is measured  to ensure we meet the real-time feedback latency. For debugging purpose, I will implement detailed agent logging of intermediate steps, including retrieved context, tool usage, and sub-agent calls, and use this data to compute metrics such as tool-call accuracy and pipeline success rate . Visualization tools will complement this by mapping explanations to board states and candidate moves, enabling faster debugging, but evaluation will remain grounded in measurable benchmarks.

Posted on

Team’s Status Report for 4/4

The most significant risks that could jeopardize the success of the project is still the integration of the subsystems. We did not realize that the computer vision portion may be more complicated than intended and that the output format we chose, an FEN string, may be harder for the other subsystem for interpret than intended. These risks are being managed by building the portions of all 3 subsystems that depend on the CV in conjunction so that any tweaks to make either side of the exchange easier can be done and tested more quickly.

There were no changes to the existing design of the system.

We are planning to run the CV, move suggestion, opponent move generation on 100 different board states and testing the accuracy. Each subsystem during these 100 different boards states will also be measuring the latency and seeing whether they are under our goals of 500ms, 100ms, and 5s respectively. We also plan to run 10 full games to see whether the move deemed “most optimal” is indeed the most optimal and also whether the LLM is able to provide meaningful help during these 10 games.