Symmetry in language statistics shapes the geometry of model representations

Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri · 2026

Read Symmetry in language statistics shapes the geometry of model representations by Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15029.txt.

Description

Karkada and colleagues explain why language model representations show such clean geometry. Months arrange into a circle, historical years trace a smooth one-dimensional manifold, and city coordinates can be read out with a linear probe. The team first shows that language statistics carry translation symmetry: how often two months co-occur depends only on the gap between them. They prove that symmetry governs the shapes in high-dimensional word embedding models and derive the manifold shapes analytically.

Those predictions match large text embedding models and large language models in tests. The shapes also survive at moderate dimension when statistics are perturbed, for instance by deleting every sentence where two months co-occur. The team proves that robustness follows when co-occurrence numbers answer to a hidden latent variable, tracing representational manifolds back to statistical symmetries of language itself.

Catalog record

CREATORDhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE104 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-15029

Browse the full library · About the Tower of Babel