Less is More: Local Intrinsic Dimensions of Contextual Language Models

Benjamin Matthias Ruppik Julius von Rohrscheidt Carel van Niekerk Michael Heck, Renato Vukovic Shutong Feng Hsien-chin Lin Nurul Lubis, Bastian Rieck Marcus Zibrowius Milica Gašić · 2025

Read Less is More: Local Intrinsic Dimensions of Contextual Language Models by Benjamin Matthias Ruppik Julius von Rohrscheidt Carel van Niekerk Michael Heck, Renato Vukovic Shutong Feng Hsien-chin Lin Nurul Lubis, Bastian Rieck Marcus Zibrowius Milica Gašić (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01034.txt.

Description

Benjamin Ruppik and a large team use local intrinsic dimension to read the inner geometry of contextual language models. Basic questions, such as how fine tuning changes a model, usually need heavy testing. The shape of latent embeddings offers a lighter lens on training and adaptation.

Local dimensions of the space are measured and tracked through training and fine tuning. Their mean predicts when training capacity runs out in a dialogue tracking task, when overfitting sets in on an emotion task, and when grokking occurs on an arithmetic task. Drops in mean local dimension tend to come before gains. The heuristic gives users a practical signal when they tune a model.

Catalog record

CREATORBenjamin Matthias Ruppik Julius von Rohrscheidt Carel van Niekerk Michael Heck, Renato Vukovic Shutong Feng Hsien-chin Lin Nurul Lubis, Bastian Rieck Marcus Zibrowius Milica Gašić
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01034

Browse the full library · About the Tower of Babel