ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models

Jincheng Liu, Sijun He, Jingjing Wu, Xiangsen Wang, Yang Chen, Zhaoqi Kuang, Siqi Bao, Yuan Yao · 2025

Read ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models by Jincheng Liu, Sijun He, Jingjing Wu, Xiangsen Wang, Yang Chen, Zhaoqi Kuang, Siqi Bao, Yuan Yao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24239.txt.

Description

Chess punishes sloppy strategy, vague rules, and lost track of the board, which makes it a sharp test of language models. Jincheng Liu and 7 co-authors built ChessArena, a testbed where models play each other under four modes. They tested thirteen models over more than eight hundred games, probing rule knowledge, move choice, and puzzle solving.

No model beat an amateur-level chess engine, and some lost even to random play. A fine-tuned mid-size model from the team did much better, nearing far larger reasoning models. The gap suggests current models lean on pattern matching more than genuine strategic thought, the authors argue.

Catalog record

CREATORJincheng Liu, Sijun He, Jingjing Wu, Xiangsen Wang, Yang Chen, Zhaoqi Kuang, Siqi Bao, Yuan Yao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24239

Browse the full library · About the Tower of Babel