ChessQA: Evaluating Large Language Models for Chess Understanding
Qianfeng Wen, Zhenwei Tang, Ashton Anderson · 2025
Read ChessQA: Evaluating Large Language Models for Chess Understanding by Qianfeng Wen, Zhenwei Tang, Ashton Anderson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23948.txt.
Description
Qianfeng Wen and two co-authors present ChessQA, a benchmark that tests how deeply language models understand chess, beyond merely picking decent moves. Chess suits evaluation well: its rules are crisp, truth is objective, and skill spans a wide range. Earlier tests were ad hoc and narrow, so they could not show how understanding varies with model size, training recipe, or architecture.
ChessQA sorts its questions into five categories that climb the ladder of player mastery: structure, motifs, short tactics, position judgment, and semantics. That spread separates knowing the rules from calculating tactics and describing high-level ideas. The benchmark is built to evolve, with prompts, answer keys, and construction scripts that refresh as models improve. Across a range of current models, the authors find persistent weaknesses in every category, with error analyses per category. Code, refreshed data, and a public leaderboard will be released to support comparisons.
Catalog record
| CREATOR | Qianfeng Wen, Zhenwei Tang, Ashton Anderson |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23948