Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models
Alexander Guha · 2026
Read Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models by Alexander Guha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02765.txt.
Description
Guha asks how many features a transformer model can hold. Model dimension is a basic design choice. Features are thought to live as directions in latent space, nearly at right angles to each other. Guha counts how many such directions fit. The embedding matrix serves as a usable proxy. In the pairwise cosine similarity spread, a boundary splits real token relations from stray similarity. That boundary estimates the accepted deviation from perfect orthogonality.
Dozens of open-source models fall into two classes. Some keep the tight structure. Others lack it. The standard Johnson-Lindenstrauss lemma greatly underestimates how well trained representations pack. A revised formula depends on the ratio of vectors to dimensions. It cuts prediction error by two orders of magnitude, with no extra parameters. Representational capacity is an upper bound on usable directions. Capacity is sharply sensitive to the deviation parameter. Larger models favor tighter orthogonality over capacity.
Catalog record
| CREATOR | Alexander Guha |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.02765