Chess-World-Model : A 10 10 M-Game Benchmark for Exact State Tracking from Chess Move Sequences
Benjamin Walker, Terry Lyons · 2026
Read Chess-World-Model : A 10 10 M-Game Benchmark for Exact State Tracking from Chess Move Sequences by Benjamin Walker, Terry Lyons (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30100.txt.
Introduction
Benjamin Walker and Terry Lyons introduce Chess-World-Model, a benchmark for exact state tracking from 10 million real chess games. Models must predict the board reached after a sequence of legal moves. A held-out real-game split joins a random-play split testing rule learning, not memorized positions. Recurrent models beat the Transformer at small sizes, real-game scores saturate above 18 million parameters, and the random split separates models at 40 million. Weaker state-transition designs lose ground on the random split, showing it exposes failures scale alone would hide.
Catalog record
| CREATOR | Benjamin Walker, Terry Lyons |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56169 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30100
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Chess-World-Model : A 10 10 M-Game Benchmark for Exact State Tracking from Chess Move Sequences Benjamin Walker, Terry Lyons Abstract World models require state tracking, which is the ability to maintain a correct latent state across action sequences. Existing benchmarks are often synthetic or language-based, limiting their value as tests of structured state updates in realistic domains. We introduce Chess-World-Model , a large-scale state-tracking benchmark built from 10 10 million real chess games, where models predict the exact board state reached after a sequence of legal moves. Alongside a held-out real-game split, we include an out-of-distribution split from uniformly random legal play, which tests whether models learn the transition rules rather than shortcuts from common human positions. Prior theoretical and empirical work has shown that Transformers struggle to state-track, while input-dependent linear RNNs require expressive state-transition matrices to do so. We therefore benchmark a causal Transformer, block-diagonal SLiCE, Mamba-3, and Gated DeltaNet with negative eigenvalues under a matched interface and training protocol. The recurrent models strongly outperform the Transformer at 3 3 and 8 8 million parameters. Real-game performance saturates above 18 18 million parameters, but the random-uniform split remains discriminative up to 40 40 million, exposing failures otherwise hidden by scale. Additionally, ablations show that less expressive state-transition mechanisms reduce performance on the out-of-distribution split for all three recurrent models. Together, these results establish Chess-World-Model as a practical large-scale benchmark for state tracking that exposes failures model scale would otherwise conceal. 1 Introduction 1.1 Motivation A world model is only as useful as the state it maintains. As actions and observations accumulate, the model must update this state, preserve information over long horizons, and remain consistent when trajectories depart from familiar training patterns. This requirement appears in classical model-based reinforcement learning and predictive state representations ( 36 ; 24 ) , in neural world models that learn latent dynamics for control ( 13 ; 14 ) , and in recent joint-embedding methods that learn representations by predicting images and videos in latent space ( 20 ; 1 ; 2 ) . In all of these settings, prediction is not merely a one-step pattern-matching problem. It depends on whether the model can maintain and update a coherent representation of the world. Despite this central role, exact state maintenance remains difficult to evaluate. Many world-model tasks couple state tracking to perception, reward prediction, planning, or language understanding, making failures hard to localise. Conversely, many formal state-tracking benchmarks isolate the computation cleanly but rely on synthetic tasks such as parity, modular arithmetic, regular languages, or permutation composition ( 6 ; 3 ; 11 ; 26 ) . These tasks have been crucial for exposing architectural limitations. However, because they are often small or highly regular, increasing model scale can obscure state-tracking failures by allowing models to exploit task-specific shortcuts or memorise large parts of the problem structure ( 6 ; 21 ) . They therefore leave open a complementary empirical question. Can modern sequence models maintain a structured latent state in a realistic deterministic action domain? Chess provides a natural testbed for this question. The task is not to play chess, but to act as a learned game engine. Given only a sequence of legal moves, the model must reconstruct the exact state reached by the game. This state includes both the board configuration and the auxiliary variables needed to specify the position. The transition rules are known exactly, and every prediction can be checked against the rules of chess without learned simulators or subjective judgments. Chess also provides a principled axis of distribution shift. Human games occupy a structured and strategically meaningful subset of legal play, just as observations of the physical world occupy only a small subset of all dynamically possible trajectories. However, a world model should not merely interpolate within familiar trajectories. It should infer the underlying dynamics well enough to remain consistent under distribution change. Uniformly random legal play tests this distinction directly. It produces positions that are valid under the same transition rules, but that depart sharply from human regularities. Strong performance on held-out human games may therefore reflect memorised openings, common tactical motifs, or likely piece configurations. Strong performance under random legal play requires robust rule-consistent state updates. 1.2 Contributions We introduce Chess-World-Model , a large-scale benchmark for exact state tracking from chess move sequences. The benchmark is constructed from 10 M real games and evaluates models on both held-out real games and a random-uniform split generated by uniformly random legal play. Given a sequence of moves, models must predict the corresponding sequence of game states, including piece locations and auxiliary state variables. This sequence-to-sequence formulation allows every prediction to be evaluated exactly, while the random-uniform split tests whether models have learned the transition rules of chess rather than shortcuts from the distribution of human games. Recent theoretical and empirical work has shown that Transformers and input-dependent linear RNNs with diagonal state-transition matrices, such as Mamba and Mamba-2, struggle with state-tracking tasks ( 27 ; 26 ; 41 ) . By contrast, richer recurrent transition mechanisms, such as input-dependent state-transition matrices with negative eigenvalues, block-diagonal structure, or diagonal-plus-low-rank structure, are theoretically more expressive and empirically improve state-tracking performance ( 11 ; 8 ; 41 ; 33 ; 19 ) . Motivated by these results, we benchmark a causal Transformer against three recent input-dependent linear RNNs with expressive state transitions: block-diagonal SLiCE, Mamba-3, and Gated DeltaNet with negative eigenvalues ( 41 ; 11 ; 19 ) . All models are trained under a shared prediction interface and matched training protocol across four architecture scales, \mathrm{d}128/\mathrm{l}1 , \mathrm{d}256/\mathrm{l}2 , \mathrm{d}384/\mathrm{l}4 , and \mathrm{d}512/\mathrm{l}6 , where \mathrm{d} denotes hidden dimension and \mathrm{l} denotes number of layers. These correspond to approximately 3 , 8 , 18 , and 38 million parameters, respectively. Our experiments yield three main findings. First, the recurrent models substantially outperform the Transformer at the two smaller scales, \mathrm{d}128/\mathrm{l}1 and \mathrm{d}256/\mathrm{l}2 . Second, held-out real-game performance largely saturates by \mathrm{d}384/\mathrm{l}4 , whereas the random-uniform split remains discriminative through the largest scale, \mathrm{d}512/\mathrm{l}6 . Third, ablations show that reducing the expressivity of the state-transition mechanism reduces random-uniform performance among the recurrent models. Together, these results establish Chess-World-Model as a practical benchmark for state tracking that exposes failures hidden by scale and in-distribution evaluation. It provides a foundation for developing expressive and efficient sequence models whose update mechanisms support reliable state tracking, a necessary ingredient for learned world models. Code for constructing Chess-World-Model from the public Lichess Open Database, generating the random-uniform test split, training the evaluated models, and computing the reported metrics is available at https://github.com/Benjamin-Walker/Chess-World-Model . 1.3 Related work Chess is a natural domain for studying state tracking in sequence models, since move sequences induce an exactly defined latent board state. 37 introduced chess as a testbed for language-model state tracking, training Transformer language models to predict move sequences and evaluating legal move prediction together with probes of the internal board state. Related interpretability work has studied whether chess models trained on game transcripts learn internal representations of board state and player skill ( 17 ) . 4 introduced the PGN-to-FEN benchmark, which tests whether pretrained large language models can reconstruct the exact final board state from