Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story

Vladislav Pedashenko, Laida Kushnareva, Yana Khassan Nibal, Eduard Tulchinskii, Kristian Kuznetsov, Vladislav Zharchinskii, Yury Maximov, Irina Piontkovskaya · 2025

Read Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story by Vladislav Pedashenko, Laida Kushnareva, Yana Khassan Nibal, Eduard Tulchinskii, Kristian Kuznetsov, Vladislav Zharchinskii, Yury Maximov, Irina Piontkovskaya (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15210.txt.

Introduction

Vladislav Pedashenko, Laida Kushnareva, and six co-authors measure the intrinsic dimension of texts, a geometric score for how complex a passage is inside a language model's representation space. That score is known to track training behavior and data structure, but what in the writing itself drives it was unclear. The team tests texts from academic abstracts to creative stories.

Three findings stand out. The score carries information that entropy-style measures miss, once length is accounted for. It also sorts genres reliably: scientific prose scores lowest, encyclopedic writing sits in the middle, and creative or opinion writing scores highest across every model tested. Linguistic features and sparse autoencoder analysis help explain what the geometry is responding to.

Catalog record

CREATORVladislav Pedashenko, Laida Kushnareva, Yana Khassan Nibal, Eduard Tulchinskii, Kristian Kuznetsov, Vladislav Zharchinskii, Yury Maximov, Irina Piontkovskaya
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-15210

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story Vladislav Pedashenko, Laida Kushnareva, Yana Khassan Nibal, Eduard Tulchinskii, Kristian Kuznetsov, Vladislav Zharchinskii, Yury Maximov, Irina Piontkovskaya Abstract Intrinsic dimension (ID) is an important tool in modern LLM analysis, informing studies of training dynamics, scaling behavior, and dataset structure, yet its textual determinants remain underexplored. We provide the first comprehensive study grounding ID in interpretable text properties through cross-encoder analysis, linguistic features, and sparse autoencoders (SAEs). In this work, we establish three key findings. First, ID is complementary to entropy-based metrics: after controlling for length, the two are uncorrelated, with ID capturing geometric complexity orthogonal to prediction quality. Second, ID exhibits robust genre stratification: scientific prose shows low ID ( ∼ \sim 8), encyclopedic content medium ID ( ∼ \sim 9), and creative/opinion writing high ID ( ∼ \sim 10.5) across all models tested. This reveals that contemporary LLMs find scientific text "representationally simple" while fiction requires additional degrees of freedom. Third, using SAEs, we identify causal features: scientific signals (formal tone, report templates, statistics) reduce ID; humanized signals (personalization, emotion, narrative) increase it. Steering experiments confirm these effects are causal. Thus, for contemporary models, scientific writing appears comparatively “easy,” whereas fiction, opinion, and affect add representational degrees of freedom. Our multi-faceted analysis provides practical guidance for the proper use of ID and the sound interpretation of ID-based results. 1 Introduction Quantifying data complexity is one of the key pillars supporting modern ML and LLM theory. Early information–theoretic views (e.g., the Information Bottleneck) tie learning efficiency to compressibility Tishby and Zaslavsky 2015 , while the manifold perspective motivates a geometric view of representations. In NLP, complexity assessments have evolved from data–intrinsic measures (e.g., gzip ratio, lexical or syntactic indicators) to model–based ones that depend on how a model organizes and predicts text. The latter divide into predictive metrics (entropy, cross-entropy, perplexity) and geometric metrics (anisotropy, intrinsic dimension). While entropy-like measures are ubiquitous in training and evaluation, geometric complexity remains comparatively underexplored. We focus on intrinsic dimension (ID) as a geometric measure of degrees of freedom in embedding space. We establish a conceptual separation between ID and prediction-based entropy: ID depends solely on the geometry of hidden representations, whereas entropy arises only after unembedding and reflects alignment with the vocabulary. Hence, the two quantities are not substitutes. We begin by showing empirically that, after controlling for text length, ID and (cross-)entropy are essentially uncorrelated; we then substantiate this separation with a formal argument. In this paper, we provide a comprehensive, semantics-grounded account of ID for text. First, we characterize how ID varies across domains and styles, showing that scientific/informational prose is consistently low-ID, while opinionated and fictional writing is high-ID, indicating increased representational degrees of freedom from the model’s perspective. Second, we link ID to observable linguistic regularities: higher lexical diversity and lower cross-sentence repetition associate with higher ID, whereas syntactic indicators are comparatively uninformative for well-formed text. Third, using sparse autoencoders and feature steering, we connect ID to interpretable semantic axes (e.g., genre, narrative, personalization, emotional tone) and probe their causal effects on generation. Together, these results position ID as a complementary, geometry-based lens on textual complexity with practical implications for evaluation and training: specifically, the need to balance low-ID (encyclopedic/scientific) and high-ID (personal/narrative) data to faithfully assess model capabilities. The paper is organized as follows: Section 2 formalizes ID and its relation to predictive metrics; Section 3 details our SAE-based analysis methodology; Section 5 presents experimental observations. Finally, Section 6 summarizes our findings and discusses implications. 1.1 Related Work Intrinsic Dimension in Neural Networks. Intrinsic dimension (ID) analysis provides a geometric framework for understanding and improving large language models (LLMs). Early work by Li et al. 2018 showed that good solutions lie in low-dimensional subspaces of the parameter space. Extending this idea to NLP, Aghajanyan et al. 2021 demonstrated that task-specific variation in pretrained models is confined to a low-dimensional subspace, inspiring low-rank adaptation methods such as LoRA Hu et al. 2022 , which assume that effective fine-tuning occurs within compact intrinsic subspaces. Recent studies have shifted from parameter to representation space, using nonlinear estimators to measure ID in embeddings. While Havrilla and Liao 2024 link ID to model scaling, Razzhigaev et al. 2024 demonstrate that it may reflect training saturation and dynamics. Viswanathan et al. 2025 confirm previous observations and show that higher ID correlates with higher entropy and loss. ID has also been used for architecture search He et al. 2023 , providing an efficient geometric signal for model evaluation. Further, Arnold 2025 relate ID to memorization capacity, while Lee et al. 2025 show that nonlinear ID captures semantic compositionality, unlike linear measures such as PCA. Persistent Homology Dimension (PHD) in Neural Networks. Intrinsic dimension can be estimated via Persistent Homology Dimension (PHD), which combines local and global properties of point cloud Schweinhart 2021 . Birdal et al. 2021 link it to generalization property of neural networks, while Tulchinskii et al. 2023a show that texts generated by early GPT models exhibit lower PHD than human-written texts, enabling simple AI text detection. However, this distinction diminishes for newer models Kuznetsov et al. 2024 . Building on this, Kushnareva et al. 2024 demonstrate that local changes in intrinsic dimensionality can detect boundaries between human and machine-generated text segments. Other Intrinsic Dimension Approaches. Several studies explore ID-based analysis for downstream tasks. Tsukagoshi and Sasano 2025 show that substantial dimensionality reduction of task-specific embeddings is possible with minimal performance loss across classification, clustering, and retrieval tasks, using estimators such as TwoNN and metrics like Isoscore. Yin et al. 2024 apply Local Intrinsic Dimension (LID) to hallucination detection, showing that high and highly variable LID signals fabricated or unsupported content, whereas truthful generations exhibit smoother, low-variance LID profiles. Ruppik et al. 2025 further argue that intrinsic dimensionality varies locally across embedding space: common tokens occupy simple, low-LID neighborhoods, while rare or domain-specific terms require higher-dimensional representations. Layer-wise analysis reveals that lower layers exhibit uniformly low LID, whereas higher layers show increased LID and variance, reflecting growing specialization and semantic disambiguation. This rapidly expanding body of research demonstrates that intrinsic dimension in neural networks is not merely a mathematical curiosity, but rather a practical tool that offers pathways toward more interpretable and capable models and opens new applications in downstream tasks. 2 Intrinsic Dimension Intrinsic dimension (ID) measures a dataset’s degrees of freedom: for linear spaces and smooth manifolds it matches the usual dimension, and extensions to metric spaces aim to remain robust to noise and recoverable from finite samples. A rich line of work has produced estimators of intrinsic dimension with complementary strengths. TwoNN estimates ID from ratios of first/second neighbor distances via a simple log–log fit Facco et al. 2017 ; MLE fits a local Poisson model to k -NN distances; TLE leverages extreme-value theory in tight neighborhoods to reduce variance Amsaleg et al. 2022 . Beyond distance methods, PHDim uses persistent homology to capture multi-scale connectivity/holes and is comparatively stable under text domain shift and addition of noise to embeddings Tulchinskii et al. 2023b . The formal definitions are given in App. A.1 . 2.1 Relation between ID and model-prediction entropy Intrinsic dimension (ID) is computed from the geometry

Read the complete text