What Do World Models Learn in RL? Probing Latent Representations in Learned Environment Simulators

Xinyu Zhang · 2026

Read What Do World Models Learn in RL? Probing Latent Representations in Learned Environment Simulators by Xinyu Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21546.txt.

Introduction

Zhang probes what world models in reinforcement learning store inside. Such models learn to mimic an environment, which makes training cheap on samples. Probes, causal tests, and attention maps are applied to two models: a token transformer and a diffusion model. Both trained on Atari Breakout and Pong. Game facts like object positions and scores could be read out with simple linear probes. Shifting hidden states along probe directions changed predictions as hoped. So the stored codes are used, not just present.

Catalog record

CREATORXinyu Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE12113
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21546

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What Do World Models Learn in RL? Probing Latent Representations in Learned Environment Simulators Xinyu Zhang Abstract World models learn to simulate environment dynamics from experience, enabling sample-efficient reinforcement learning. But what do these models actually represent internally? We apply interpretability techniques—including linear and nonlinear probing, causal interventions, and attention analysis—to two architecturally distinct world models: IRIS (discrete token transformer) and DIAMOND (continuous diffusion UNet), trained on Atari Breakout and Pong. Using linear probes, we find that both models develop linearly decodable representations of game state variables (object positions, scores), with MLP probes yielding only marginally higher R 2 R^{2} , confirming that these representations are approximately linear. Causal interventions—shifting hidden states along probe-derived directions—produce correlated changes in model predictions, providing evidence that representations are functionally used rather than merely correlated. Analysis of IRIS attention heads reveals spatial specialization: specific heads attend preferentially to tokens overlapping with game objects. Multi-baseline token ablation experiments consistently identify object-containing tokens as disproportionately important. Our findings provide interpretability evidence that learned world models develop structured, approximately linear internal representations of environment state across two games and two architectures. 1 Introduction World models—learned simulators of environment dynamics—have become a cornerstone of sample-efficient reinforcement learning ( Ha & Schmidhuber 2018 ; Hafner et al. 2023 ) . Recent advances such as IRIS ( Micheli et al. 2023 ) and DIAMOND ( Alonso et al. 2024 ) achieve strong performance on the Atari 100k benchmark by learning to predict future observations entirely from experience. Yet despite their empirical success, a fundamental question remains: what do world models represent internally? This question connects to the broader “linear representation hypothesis” emerging in mechanistic interpretability research ( Park et al. 2023 ; Elhage et al. 2022 ) . Li et al. 2023 showed that a transformer trained to predict Othello moves develops an emergent linear representation of the board state, despite never being trained on it directly. Nanda et al. 2023 extended this to chess. If sequence models trained on game transcripts develop world representations, do world models—which are explicitly trained to predict future observations—develop even richer ones? We apply probing classifiers ( Alain & Bengio 2017 ; Belinkov 2022 ) and causal interventions to two architecturally distinct world models—IRIS ( Micheli et al. 2023 ) (VQ-VAE + transformer) and DIAMOND ( Alonso et al. 2024 ) (UNet diffusion)—on Breakout and Pong. We contribute: (1) linear and MLP probing showing approximately linear game state representations ( \Delta\leq 0.06 ); (2) causal interventions confirming representations are functionally used ( r>0.95 ); (3) per-head spatial attention specialization; and (4) multi-baseline token ablation with consistent importance rankings ( \rho>0.9 ). 2 Method 2.1 Models and Ground Truth IRIS tokenizes 64\times 64 observations into 16 discrete tokens (VQ-VAE, codebook 512, 4\times 4 grid) then predicts sequences with a GPT-2 transformer (10 layers, 4 heads, dim 256). DIAMOND uses a UNet denoiser (4 stages, 64 channels) with EDM preconditioning. We probe on Breakout ( ball_x , ball_y , player_x , score) and Pong ( ball_x , ball_y , player_y , enemy_y ), with ground truth from Atari RAM ( Anand et al. 2019 ) . 2.2 Probing Protocol We extract frozen representations from all layers ( N{=}10{,}000 frames per game): IRIS VQ-VAE encoder/embedding + 10 transformer layers; DIAMOND conv input, 4 encoder/decoder stages, bottleneck, norm output. For each (layer, property) pair, we train Ridge regression ( \alpha{=}1.0 ) and 2-layer MLP probes (256 \to 128 \to 1, ReLU, Adam), both with 5-fold CV R^{2} . The selectivity gap \Delta=R^{2}_{\text{MLP}}-R^{2}_{\text{linear}} measures nonlinear structure. Controls: raw pixels, random model, shuffled labels. 2.3 Causal Intervention Protocol To move beyond correlation, we perform activation patching along probe directions ( Geiger et al. 2021 ) : modify hidden states \mathbf{h}^{\prime}=\mathbf{h}+\alpha\cdot\hat{\mathbf{w}} (where \hat{\mathbf{w}} is the normalized Ridge probe weight) and measure the resulting change in next-token logits (KL divergence, token change rate). A positive correlation between |\alpha| and prediction change indicates the probe direction is functionally used . 2.4 Attention Analysis and Token Ablation For IRIS’s 40 attention heads, we compute attention entropy and per-head spatial selectivity (mean attention to each of 16 token positions, forming 4\times 4 maps). For token ablation, we replace each spatial token under three baselines (zero, mean, random codebook entry) and measure prediction disruption (KL divergence). Consistency across baselines (Spearman \rho ) indicates robust importance rankings. 3 Results 3.1 Linear Representations Across Games Figure 1: Probe R^{2} across layers (in network data-flow order) for IRIS (left) and DIAMOND (right) on Breakout (top) and Pong (bottom). Each line tracks one game-state property; shaded bands show \pm 1 std over 5-fold CV. IRIS representations are flat across transformer layers, while DIAMOND shows a peaked inverted-V centered on the UNet bottleneck. Note: y -axis includes negative R^{2} values, revealing that DIAMOND’s early encoder layers are worse than a constant predictor for ball position. Figure 1 shows probe R^{2} across all layers for both games. IRIS. Ball position encoding is stable across all transformer layers in both games: for Breakout, R^{2} for ball_x ranges from 0.84\pm 0.01 (layer 0) to 0.85\pm 0.006 (layer 5), a span of only 0.01 . Paddle position and score are near-perfectly encoded ( R^{2}>0.99 ) at every layer. The VQ-VAE encoder already achieves R^{2}=0.83 for ball-x, and the transformer barely improves this ( +0.02 ). DIAMOND. Early encoder stages (0–2) show negative R^{2} for ball position (as low as -1.45 ), while the deepest encoder stage begins recovering ( R^{2}=0.78 ); the bottleneck achieves peak linear R^{2}=0.81\pm 0.01 for ball_x , and decoder stages decline—an inverted-V pattern suggesting the bottleneck compresses information maximally. MLP probes recover substantially more from decoder layers ( R^{2}=0.91 at dec_2 vs. -0.27 linear), indicating these layers encode ball position nonlinearly via skip connections. Cross-game consistency. Pong shows the same architectural signatures: flat IRIS profiles and peaked DIAMOND bottleneck (Figure 1 , bottom row), indicating these patterns are architecture-dependent rather than game-specific. However, both models track ball position significantly better in Pong ( R^{2}\geq 0.95 ) than Breakout ( R^{2}\leq 0.85 ), likely because Pong’s visual scene is simpler (no bricks, fewer objects). Interestingly, DIAMOND’s V-shape in ball tracking is much less pronounced in Pong (dec_2 R^{2}=0.95 vs. -0.27 in Breakout), suggesting this pattern is game-dependent. Table 1: Best-layer R^{2} (mean \pm std over 5-fold CV) for Breakout. Both linear and MLP probes are shown; the small selectivity gap ( \Delta ) confirms approximately linear representations. Representation ball_x ball_y player_x score Random model -1.21 -1.22 -1.14 -1.18 Shuffled labels -0.51 -0.49 -0.53 -0.52 Raw pixels -1.31 -0.48 0.9989\pm.0006 0.9998\pm.0001 IRIS (Linear) \mathbf{0.85}\pm.006 0.58\pm.03 \mathbf{0.9994}\pm.0001 1.0000\pm.0000 IRIS (MLP) 0.91\pm.005 0.59\pm.03 0.9987\pm.0002 0.9999\pm.0000 \Delta_{\text{IRIS}} +0.06 +0.01 -0.0007 -0.0001 DIAMOND (Linear) 0.81\pm.01 \mathbf{0.57}\pm.05 1.0000\pm.0000 \mathbf{1.0000}\pm.0000 DIAMOND (MLP) 0.91\pm.005 0.63\pm.05 0.9994\pm.0002 0.9998\pm.0001 \Delta_{\text{DIAMOND}} +0.10 +0.06 -0.0006 -0.0002 Table 1 shows that MLP probes yield only marginal improvements over linear probes for IRIS ( |\Delta|\leq 0.06 ). DIAMOND shows a larger gap for ball position ( \Delta=0.10 for ball_x ), driven by decoder layers where skip connections mix information nonlinearly; at the bottleneck itself, the gap is small ( \Delta=0.04 ). Both models dramatically outperform baselines, with raw pixels failing on ball position ( R^{2}=-1.31 ), showing that world model representations extract non-trivial structure. For Pong,

Read the complete text