Transformers Linearly Represent Highly Structured World Models

Roman Kniazev, Nathanaël Fijalkow · 2026

Read Transformers Linearly Represent Highly Structured World Models by Roman Kniazev, Nathanaël Fijalkow (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18847.txt.

Introduction

Roman Kniazev and Nathanaël Fijalkow probe whether a transformer trained on Sudoku traces builds an internal world model. Their eight-layer model does not track the board cell by cell, as a human analyst might expect. Instead it organizes knowledge around rows, columns, and boxes, the units Sudoku constraints act on. A small set of final-layer neurons forms a naked-single circuit, each spotting a cell with exactly one possible digit and promoting it. The shape of the world model, they argue, follows the task structure.

Catalog record

CREATORRoman Kniazev, Nathanaël Fijalkow
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43156
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18847

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Transformers Linearly Represent Highly Structured World Models Roman Kniazev, Nathanaël Fijalkow Abstract Do transformers, when trained on sequential reasoning traces, build internal models of the underlying task? And if so, does the structure of those internal representations mirror the structure of the domain? We train an 8-layer transformer on Sudoku solving traces and perform a mechanistic analysis of its internal computation. We establish two results. First, the model builds a substructure world model : it does not represent the board state cell by cell, as a human analyst would expect, but organizes information around the rows, columns, and boxes that Sudoku’s constraints act on. Second, we identify a naked-single circuit : a small set of dedicated neurons in the final MLP layer, each individually detecting when exactly one digit remains possible for a specific cell, and reliably promoting that digit. These findings show that the geometry of an emergent world model is shaped by the constraint algebra of the domain, not its surface presentation, and that the resulting decision circuit is sparse, monosemantic, and fully interpretable. More broadly, they demonstrate that mechanistic interpretability tools can recover an end-to-end algorithmic account of how a transformer solves a combinatorial reasoning task. 1 Introduction Context. Do transformers, when trained on sequential reasoning traces, build internal world models of the underlying environment? Li et al. 2023 and Nanda et al. 2023 showed that the answer is yes for Othello: a GPT-2-style model trained only on game transcripts spontaneously develops a representation of board occupancy (black / white / empty per cell), despite receiving no direct supervision on board state. This finding sits within the programme of mechanistic interpretability Elhage et al. 2021 ; Elhage et al. 2022 , which aims to reverse-engineer the algorithms learned by neural networks. A central working hypothesis in this programme is the linear representation hypothesis Park et al. 2024 ; Mikolov et al. 2013 : that high-level concepts are encoded as linear directions in activation space, and can therefore be read out by simple linear probes Alain and Bengio 2016 ; Belinkov 2022 . Understanding when and how world models emerge under this hypothesis, and what geometry they take, is important for building reliable, inspectable AI systems. Background: linear world models and how to find them. Nanda et al. 2023 built on Li et al. 2023 ’s result by establishing that the Othello world model is linear : the occupancy of each cell is recoverable by a simple logistic regression probe on the residual stream, without any nonlinear transformation. They further showed, using activation patching , that these linear directions are causally used by the model to predict the next move, not merely a side effect of training. The same paradigm has since revealed world models in chess-playing transformers ( Karvonen 2024 ) and maze-solving models ( Ivanitskiy et al. 2024 ) . In all of these cases, the unit of representation matches the most natural human decomposition of the state space: one feature per cell. Our question: does the geometry reflect the task’s constraint algebra? We ask whether this phenomenon extends to a qualitatively different class of task: combinatorial constraint-satisfaction . Our domain is Sudoku : a one-player puzzle on a 9\times 9 grid whose fundamental difficulty is constraint propagation across 27 overlapping substructures (9 rows, 9 columns, 9 boxes), each requiring every digit 1 – 9 to appear exactly once. Unlike Othello, where the state is naturally decomposed by cell, Sudoku has an explicit algebraic structure: the validity of any placement is determined by the digit’s presence in three substructures, not in any single cell. This raises a sharp question: does the model represent the board state per cell (as in Othello) or per substructure (as the constraints demand)? Following Giannoulis et al. 2026 and Shah et al. 2024 , we train a causal transformer on Norvig-style solving traces with backtracking ( Norvig 2006 ) , achieving 98.4% per-cell and 97.5% per-grid accuracy on a held-out test set of 150,000 puzzles. We then apply Nanda et al. 2023 ’s probe-and-patch methodology to the residual stream. Findings and contributions. Our contribution is new findings about the geometry of emergent world models and the circuits that read them, obtained by applying the probe-and-patch methodology of Nanda et al. 2023 to a transformer trained on Sudoku following Giannoulis et al. 2026 . 1. Substructure world model ( Section 3 ). Prior world-model work has consistently found representations whose geometry matches the surface decomposition of the task: one feature per cell. We show this is not a universal property: when the task has an explicit constraint algebra, the model’s representation reflects that algebra instead. Linear probes for “is digit d present in substructure S ?” achieve perfect exact match accuracy across all 243 (substructure, digit) pairs in mid-layers, while per-cell probes plateau at 80% accuracy. The representation forms by mid-layers and tracks the current board state throughout generation, remaining geometrically stable across the entire solving trace. Causal patching confirms these directions are actively used. 2. Substructure-aligned attention ( Section 4 ). We identify that mid-layer attention heads organize along Sudoku’s substructures: heads concentrate their attention on individual rows, columns, or boxes (or coherent groups of three), and their OV circuits apply a symmetric mechanism: digits already present in the attended substructure have their logits suppressed, while absent digits are promoted. This shows that the transformer represents Sudoku’s rules through localized constraint checkers operating directly on the substructure-organized residual stream. 3. Naked-single circuit ( Section 5 ). We identify a small set of dedicated neurons in the final MLP layer that detect when exactly one digit remains possible for a specific cell (a naked single ). Each neuron is specific and monosemantic. Moreover, the residual stream before the final MLP already strongly promotes the correct digit, so mid-layers encode the answer and the final MLP amplifies it. The circuit is sparse and provides a fully mechanistic account of how the model handles the most common forced placement. Code is available at the link , and the dataset and a model checkpoint at Hugging Face . 2 Preliminaries Sudoku. A Sudoku puzzle is defined on a 9\times 9 grid of 81 cells. Each cell (r,c) , for r,c\in\{1,\ldots,9\} , is to be filled with a digit d\in\{1,\ldots,9\} . The grid is partitioned into 27 substructures : 9 rows, 9 columns, and 9 boxes . Row r is \mathsf{Row}_{r}=\{(r,c):c\in\{1,\ldots,9\}\} ; column c is \mathsf{Col}_{c}=\{(r,c):r\in\{1,\ldots,9\}\} ; box \mathsf{Box}_{k} for k=3(i-1)+j ( i,j\in\{1,2,3\} ) is the 3\times 3 block \{(3i-2,3j-2),\ldots,(3i,3j)\} . Every cell (r,c) belongs to exactly three substructures: \mathsf{Row}_{r} , \mathsf{Col}_{c} , and one box \mathsf{Box}_{k} . Three horizontally adjacent boxes sharing the same rows form a band ; three vertically adjacent boxes sharing the same columns form a stack . There are 3 bands and 3 stacks, each covering 27 cells. A placement of digit d in cell (r,c) is valid if d does not already appear in \mathsf{Row}_{r} , \mathsf{Col}_{c} , or \mathsf{Box}_{k} . The candidate set \mathsf{cand}(r,c)=\{1,\ldots,9\}\setminus(\text{digits in }\mathsf{Row}_{r}\cup\mathsf{Col}_{c}\cup\mathsf{Box}_{k}) collects all valid digits for an empty cell. A puzzle is given as an initial partial assignment (the clues ), and the task is to complete it uniquely so that every digit appears

Read the complete text