Manifold Dimension Estimation: An Empirical Study

Bi, Zelong, Pierre Lafaye de Micheaux · 2025

Read Manifold Dimension Estimation: An Empirical Study by Bi, Zelong, Pierre Lafaye de Micheaux (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15517.txt.

Introduction

Zelong Bi and Pierre Lafaye de Micheaux survey ways to guess the true dimension of data. Much high dimensional data may lie near a surface of far fewer dimensions. Guessing that count helps users exploit the shape of their data. Past work on such guesses is scattered, with no shared test of the tools.

The pair review the theory that is often skipped and pick eight tools to test. In controlled runs they vary noise, curve, and sample size one at a time. They also compare the tools on fake and real data sets. A principled way to tune each tool for each data set is part of the study. Their results guide which tool to pick when.

Catalog record

CREATORBi, Zelong, Pierre Lafaye de Micheaux
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE128 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15517

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Manifold Dimension Estimation: An Empirical Study Bi, Zelong, Pierre Lafaye de Micheaux Abstract The manifold hypothesis suggests that high-dimensional data often lie on or near a low-dimensional manifold. Estimating the dimension of this manifold is essential for leveraging its structure, yet existing work on dimension estimation is fragmented and lacks systematic evaluation. This article provides a comprehensive survey for both researchers and practitioners. We review often-overlooked theoretical foundations and present eight representative estimators. Through controlled experiments, we analyze how individual factors—such as noise, curvature, and sample size—affect performance. We also compare the estimators on diverse synthetic and real-world datasets, introducing a principled approach to dataset-specific hyperparameter tuning. Our results offer practical guidance and suggest that, for a problem of this generality, simpler methods often perform better. Keywords: manifold hypothesis, intrinsic dimension, nonlinear dimension reduction 1 Introduction Scientists and engineers are increasingly training and testing models on high-dimensional datasets ( Zhou 2021 ) . Yet, the success of modern machine learning algorithms appears to defy the curse of dimensionality, which posits that the sample size required to reliably recover underlying structures grows exponentially with the data dimension ( Keogh and Mueen 2011 ) . One possible explanation is the manifold hypothesis , which assumes that despite sitting within a high-dimensional space \mathbb{R}^{p} , the data actually lie on or near a lower d -dimensional manifold ( Aghajanyan et al. 2020 ; Meilă and Zhang 2024 ) , as illustrated in Figure 1 . Under this assumption, at least locally or approximately, the dataset can be described using only d free parameters corresponding to the dimension of the manifold — thus greatly simplifying the problem. Figure 1 : Data points in \mathbb{R}^{3} lie on (left) or close to (right) a 2-dimensional manifold (a cylinder). A natural task that arises from the manifold hypothesis is the manifold learning problem , which aims to recover the underlying manifold from the dataset. This task is also referred to as nonlinear dimension reduction ; see Balasubramanian and Schwartz 2002 and Coifman and Lafon 2006 for classical methods, and Floryan and Graham 2022 and Alberti et al. 2024 for more recent, data-driven approaches. The most important output of manifold learning is often a low-dimensional representation of the data. This reduced representation is particularly useful because many classical statistical methods do not scale well in high dimensions and can be more effectively applied to the lower-dimensional coordinates. While the integration of manifold learning with traditional statistical analysis remains relatively underexplored, there has been some work in this direction, including regression on manifolds ( Aswani et al. 2011 ; Cheng and Wu 2013 ) and factor models for time series ( Chan et al. 2017 ) . As high-dimensional data continues to challenge standard statistical techniques, we believe that combining manifold learning with classical methods offers a promising and underutilized alternative. Among the many properties of the underlying manifold, its dimension is arguably the most important, yet it is often assumed to be known and specified in advance by manifold learning algorithms ( Meilă and Zhang 2024 ) . As a result, researchers have long been interested in the subproblem of manifold dimension estimation , and numerous methods have been proposed. But existing work is often fragmented, with researchers approaching the problem from domain-specific perspectives with no unified treatment. Moreover, only limited systematic evaluation of existing techniques have been conducted (such as Campadelli et al. 2015 and Camastra and Staiano 2016 , but both are now dated and limited in scope 1 1 1 During the final stage of writing this article, we became aware of the arXiv preprint by Binnie et al. 2025 , which also surveys dimension estimators. However, our work was conducted independently and approaches the problem from different angles, offering unique insights and experimental rigor. ), making it difficult for practitioners to make informed choices This article offers a comprehensive survey of the manifold dimension estimation problem and is organized as follows. Section 2 introduces essential theoretical background in an accessible language, often unfamiliar to practitioners and typically omitted from previous surveys. We notably cover manifolds and their constructions, sampling procedures, and the statistical difficulty of the dimension estimation problem. Section 3 details the mathematical construction of eight estimators of the manifold dimension. We chose these specific estimators with care, considering top performers identified in prior work, as well as promising more recent proposals. Section 4 presents our empirical evaluation, including the experimental setup, results, and discussion. We benchmark the performance of these estimators under varying conditions, including among things neighborhood size, curvature, and noise level. While earlier comparative studies provided valuable insights, they did not typically control for these factors individually. We also compare the eight estimators on 18 synthetic data sets and on 3 real-world datasets with different geometry, distribution and noise characteristics. Building on insights from our controlled experiments, we develop an algorithm to tune hyperparameters per dataset for the estimators, addressing a common flaw in prior studies where hyperparameters were fixed across datasets. This leads to more equitable comparisons and more reliable conclusions. Finally, Section 5 concludes with practical recommendations for practitioners and outlines directions for future research. We conduct our study using both Python and R , and all code and datasets used in our experiments are publicly available. 2 Theoretical Background In this section, we provide a concise yet comprehensive overview of the theoretical foundations relevant to manifold dimension estimation. We begin with a review of manifolds and their associated structures, then discuss the problem of sampling from manifolds. Finally, we formulate the manifold dimension estimation problem and outline its statistical challenges. Manifolds—and the mathematical constructions built upon them—can become quite abstract when developed in full generality. Such abstraction, however, is unnecessary for our purposes. Throughout, we adopt the most direct and practical definitions, while avoiding oversimplification. For readers interested in more advanced treatments, we refer to standard references in differential geometry, such as Lee 2003 and Lee 2018 . 2.1 Manifold and Dimension In this article, we view a d -dimensional manifold M simply as a smooth 2 2 2 Formally, a manifold is smooth if it can be fully described by a collection of (local) coordinate maps (called chart functions), and whenever two chart functions describe the same region (i.e., their domains overlap), the change of variables between their coordinate systems is infinitely differentiable. d -dimensional object, assumed to be sitting inside some higher-dimensional Euclidean space \mathbb{R}^{p} , called the ambient space . In that case, the Euclidean structure of \mathbb{R}^{p} naturally induces a way to measure distances and angles along M , leading to what is formally called a Riemannian manifold. Intuitively, we can think of M as a geometric object with a flat or curved smooth surface, that when you zoom in closely enough, looks like \mathbb{R}^{d} . When d=1 , a manifold is a curve (e.g., a circle or a helix) and locally resembles a segment of a straight line. When d=2 , it appears as a surface (e.g., a sphere or a torus) and locally resembles a plane. See Figure 2 for an illustration of some classical examples. A manifold therefore generalizes the idea of a smooth curve or surface to any dimension

Read the complete text