ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models
Jincheng Liu, Sijun He, Jingjing Wu, Xiangsen Wang, Yang Chen, Zhaoqi Kuang, Siqi Bao, Yuan Yao · 2025
Read ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models by Jincheng Liu, Sijun He, Jingjing Wu, Xiangsen Wang, Yang Chen, Zhaoqi Kuang, Siqi Bao, Yuan Yao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24239.txt.
Description
Chess punishes sloppy strategy, vague rules, and lost track of the board, which makes it a sharp test of language models. Jincheng Liu and 7 co-authors built ChessArena, a testbed where models play each other under four modes. They tested thirteen models over more than eight hundred games, probing rule knowledge, move choice, and puzzle solving.
No model beat an amateur-level chess engine, and some lost even to random play. A fine-tuned mid-size model from the team did much better, nearing far larger reasoning models. The gap suggests current models lean on pattern matching more than genuine strategic thought, the authors argue.
Catalog record
| CREATOR | Jincheng Liu, Sijun He, Jingjing Wu, Xiangsen Wang, Yang Chen, Zhaoqi Kuang, Siqi Bao, Yuan Yao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-24239