Symmetry in language statistics shapes the geometry of model representations
Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri · 2026
Read Symmetry in language statistics shapes the geometry of model representations by Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15029.txt.
Description
Karkada and colleagues explain why language model representations show such clean geometry. Months arrange into a circle, historical years trace a smooth one-dimensional manifold, and city coordinates can be read out with a linear probe. The team first shows that language statistics carry translation symmetry: how often two months co-occur depends only on the gap between them. They prove that symmetry governs the shapes in high-dimensional word embedding models and derive the manifold shapes analytically.
Those predictions match large text embedding models and large language models in tests. The shapes also survive at moderate dimension when statistics are perturbed, for instance by deleting every sentence where two months co-occur. The team proves that robustness follows when co-occurrence numbers answer to a hidden latent variable, tracing representational manifolds back to statistical symmetries of language itself.
Catalog record
| CREATOR | Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 104 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-15029