Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models

Alexander Guha · 2026

Read Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models by Alexander Guha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02765.txt.

Description

Guha asks how many features a transformer model can hold. Model dimension is a basic design choice. Features are thought to live as directions in latent space, nearly at right angles to each other. Guha counts how many such directions fit. The embedding matrix serves as a usable proxy. In the pairwise cosine similarity spread, a boundary splits real token relations from stray similarity. That boundary estimates the accepted deviation from perfect orthogonality.

Dozens of open-source models fall into two classes. Some keep the tight structure. Others lack it. The standard Johnson-Lindenstrauss lemma greatly underestimates how well trained representations pack. A revised formula depends on the ratio of vectors to dimensions. It cuts prediction error by two orders of magnitude, with no extra parameters. Representational capacity is an upper bound on usable directions. Capacity is sharply sensitive to the deviation parameter. Larger models favor tighter orthogonality over capacity.

Catalog record

CREATORAlexander Guha
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02765

Browse the full library · About the Tower of Babel