Less is More: Local Intrinsic Dimensions of Contextual Language Models
Benjamin Matthias Ruppik Julius von Rohrscheidt Carel van Niekerk Michael Heck, Renato Vukovic Shutong Feng Hsien-chin Lin Nurul Lubis, Bastian Rieck Marcus Zibrowius Milica Gašić · 2025
Read Less is More: Local Intrinsic Dimensions of Contextual Language Models by Benjamin Matthias Ruppik Julius von Rohrscheidt Carel van Niekerk Michael Heck, Renato Vukovic Shutong Feng Hsien-chin Lin Nurul Lubis, Bastian Rieck Marcus Zibrowius Milica Gašić (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01034.txt.
Description
Benjamin Ruppik and a large team use local intrinsic dimension to read the inner geometry of contextual language models. Basic questions, such as how fine tuning changes a model, usually need heavy testing. The shape of latent embeddings offers a lighter lens on training and adaptation.
Local dimensions of the space are measured and tracked through training and fine tuning. Their mean predicts when training capacity runs out in a dialogue tracking task, when overfitting sets in on an emotion task, and when grokking occurs on an arithmetic task. Drops in mean local dimension tend to come before gains. The heuristic gives users a practical signal when they tune a model.
Catalog record
| CREATOR | Benjamin Matthias Ruppik Julius von Rohrscheidt Carel van Niekerk Michael Heck, Renato Vukovic Shutong Feng Hsien-chin Lin Nurul Lubis, Bastian Rieck Marcus Zibrowius Milica Gašić |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01034