This week, I continued working on the LLM agent orchestration pipeline and prompt engineering, focusing on improving reasoning quality and reducing hallucinations in move explanations. I also explored deployment options using Amazon Bedrock and researched LoRA fine-tuning as a potential approach for adapting models to the Xiangqi domain. Through this investigation, I identified a key limitation in standard LLMs: lack of spatial awareness when interpreting raw board representations such as FEN. To address this, I designed a feature engineering approach that converts board states into a structured relational representation, encoding piece interactions, threat maps, palace dynamics, and pawn structures. This effectively transforms the problem from spatial reasoning into a relational reasoning task, which aligns better with transformer-based models.
Additionally, I began prototyping this relational preprocessing pipeline and integrating it into the agent workflow alongside RAG. The goal is to provide the LLM with structured, verifiable context to improve consistency and interpretability of generated explanations. This approach reduces reliance on implicit spatial reasoning and instead supplies explicit constraints derived from the engine, helping mitigate hallucinations. Moving forward, I plan to refine the feature extraction process, validate its impact on explanation accuracy, and continue improving prompt design and integration with the full system pipeline.
I plan to evaluate the LLM orchestration system using multiple metrics and structured logging and agent visualization. First, I will measure explanation correctness by checking whether the LLM’s suggested moves and reasoning align with the engine’s validated outputs. I will also track hallucination rate, calculated from the percentage of responses that include invalid moves, incorrect checkmate claims, or inconsistencies with the game state. I will also evaluate consistency by running repeated prompts on the same position and measuring differences in outputs. For performance, time between text and move input and when response is returned is measured to ensure we meet the real-time feedback latency. For debugging purpose, I will implement detailed agent logging of intermediate steps, including retrieved context, tool usage, and sub-agent calls, and use this data to compute metrics such as tool-call accuracy and pipeline success rate . Visualization tools will complement this by mapping explanations to board states and candidate moves, enabling faster debugging, but evaluation will remain grounded in measurable benchmarks.