Information-Theoretic Quality Metric of Low-Dimensional Embeddings

Sebastián Gutiérrez-Bernal, Hector Medel Cobaxin, Abiel Galindo González · 2025

Read Information-Theoretic Quality Metric of Low-Dimensional Embeddings by Sebastián Gutiérrez-Bernal, Hector Medel Cobaxin, Abiel Galindo González (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23981.txt.

Introduction

Sebastián Gutiérrez-Bernal, Hector Medel Cobaxin, and Abiel Galindo González propose a new yardstick for dimension reduction, the maps that squeeze high-dimensional data into pictures or compact features. Standard scores judge such embeddings by distorted distances or warped local geometry. Neither directly measures how much information the projection throws away, which is what matters in information-sensitive uses such as early-warning indicators.

Their Entropy Rank Preservation Measure works neighborhood by neighborhood, comparing the Shannon entropy of the singular-value spectrum of each patch before and after projection, summarized through the stable rank. Validated on a financial time series and a standard manifold, the measure correlated only weakly with distance-based scores but strongly on average with a geometric criterion, while disagreeing in specific local regimes. The authors conclude their metric complements existing ones by flagging exactly where severe information loss hides.

Catalog record

CREATORSebastián Gutiérrez-Bernal, Hector Medel Cobaxin, Abiel Galindo González
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23981

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Information-Theoretic Quality Metric of Low-Dimensional Embeddings Sebastián Gutiérrez-Bernal, Hector Medel Cobaxin, Abiel Galindo González Abstract In this work we study the quality of low-dimensional embeddings from an explicitly information-theoretic perspective. We begin by noting that classical evaluation metrics such as stress, rank-based neighborhood criteria, or Local Procrustes quantify distortions in distances or in local geometries, but do not directly assess how much information is preserved when projecting high-dimensional data onto a lower-dimensional space. To address this limitation, we introduce the Entropy Rank Preservation Measure (ERPM), a local metric based on the Shannon entropy of the singular-value spectrum of neighborhood matrices and on the stable rank, which quantifies changes in uncertainty between the original representation and its reduced projection, providing neighborhood-level indicators and a global summary statistic. To validate the results of the metric, we compare its outcomes with the Mean Relative Rank Error (MRRE), which is distance-based, and with Local Procrustes, which is based on geometric properties, using a financial time series and a manifold commonly studied in the literature. We observe that distance-based criteria exhibit very low correlation with geometric and spectral measures, while ERPM and Local Procrustes show strong average correlation but display significant discrepancies in local regimes, leading to the conclusion that ERPM complements existing metrics by identifying neighborhoods with severe information loss, thereby enabling a more comprehensive assessment of embeddings, particularly in information-sensitive applications such as the construction of early-warning indicators. keywords Manifold Learning, dimensionality reduction, embedding quality, entropy, singular values 1 Introduction In high-dimensional spaces, data geometry deteriorates, implying that pairwise distances become less informative, neighborhoods degenerate, and many models see their performance collapse as the ambient dimension D grows, a phenomenon often described as a manifestation of the curse of dimensionality 2 . In consequence, both statistical estimation and learning algorithms, that rely on Euclidean distances or local neighborhood structure, may be affected. This motivates the search for representations in which the relevant structure of the data becomes more regular and accessible. With these considerations in mind, various studies and proposals have been developed 8 , mainly assuming that observations x\in\mathbb{R}^{D} are concentrated near a low-dimensional manifold \mathcal{M}\subset\mathbb{R}^{D} of intrinsic dimension d\ll D , and to construct a mapping f:\mathbb{R}^{D}\to\mathbb{R}^{d} such that preserves salient structural properties of \mathcal{M} , including connectivity, curvature and intrinsic distances. This manifold-learning perspective underlies a broad family of nonlinear dimensionality-reduction methods, including geodesic-based approaches such as Isomap 19 , locally linear methods such as LLE and its variants 13 ; 21 , and spectral or kernel methods such as Laplacian Eigenmaps and Kernel PCA 1 ; 16 . Therefore, it is extremely necessary to evaluate the quality of these low-dimensional representations. A variety of criteria have been proposed for this evaluation 8 , such as global measures like Kruskal’s stress 11 , Sammon’s stress 15 , and Residual Variance 19 , which quantify distortions of pairwise distances across the entire dataset. On the other hand, local criteria focus on neighborhood relations and assess changes in nearest neighbors, rank orders, and local shape through indices such as Trustworthiness, Continuity 20 , Mean Relative Rank Error 12 , and Local Procrustes 7 . However, these existing measures are defined purely in geometric terms and therefore quantify only how distances, ranks or local shapes change under the mappings, without having one that directly assesses how much information is lost or preserved when the projection is generated. This omission is particularly problematic when dimensionality reduction is used as a preprocessing step for sensitive tasks, such as reconstructing attractors from time series embeddings 18 ; 6 ; 3 or building early-warning indicators for regime shifts in dynamical and financial systems 9 . In such settings, the informational content carried by the embedding is as critical as its geometric fidelity, in recent work on stable rank and geometry-preserving embeddings 5 suggests that spectral properties of the data matrix can play a central role in capturing this content. This work is motivated precisely by that gap. Our goal is to complement existing geometric criteria with a quality measure that incorporates an explicitly informational point of view on the effect of dimensionality reduction. By linking local spectral structure to entropy, we aim to quantify not only how the manifold is geometrically distorted, but also how the uncertainty and effective number of spectral directions change when passing from the original representation to its low-dimensional embedding. 2 Case of study: Information-based manifold learning for early warnings The motivation of this study arises from Huang’s et al. 9 work, where a Manifold Learning methodology based on information criteria is proposed. This method takes a financial time series to reconstruct its phase space through Takens’ theorem 18 , building the delay vectors X_{t}=(x_{t},x_{t-\tau},x_{t-2\tau},\dots,x_{t-(m-1)\tau}), (1) choosing the delay \tau as the first minimum of the auto mutual information function 6 and the embedding dimension m through Cao’s method 3 , obtaining a matrix X\in\mathbb{R}^{n\times m} that represents the reconstructed phase space in high dimension. Their idea centers in an information-theoretic perspective, in which instead of describing neighborhoods through Euclidean distances, each reconstructed state X_{i} is modeled as a discrete probability distribution p_{i} , estimated through kernel density, where the similarity between states is measured with the symmetrized Kullback–Leibler divergence \displaystyle h(P_{i},P_{j})=\sum_{t=1}^{m}p_{i}^{(t)}\log\frac{p_{i}^{(t)}}{p_{j}^{(t)}}+\sum_{t=1}^{m}p_{j}^{(t)}\log\frac{p_{j}^{(t)}}{p_{i}^{(t)}}, (2) whose values form the matrix H that replaces the classical geometric neighborhoods. From this, they construct a Manifold Learning method analogous to LLE 13 , but with weights defined by the information metric, in which the low-dimensional embedding Y is obtained by minimizing \displaystyle\min_{Y}\sum_{i=1}^{n}\left\lVert Y_{i}-\sum_{j=1}^{n}h_{ij}Y_{j}\right\rVert^{2}, (3) which reduces to an eigenvalue problem over M=(I-H)^{\top}(I-H) , whose d eigenvectors associated with the smallest eigenvalues form the reduced coordinates. Finally, in the methodology they quantify the quality of the embedding using the Procrustes measure to compare the high-dimensional representation against the reduced one, using datasets from a stochastic simulation and the financial series of CSI800 and S&P 500, where they show that their information-based method has much lower Procrustes error compared with other relevant methods such as Isomap, LLE, or KPCA. One of the things we highlight most about their work is its direct application, because once the manifold is reduced in dimensions, through clustering techniques and using Hidden Markov models, they detect regions associated with pre-crisis dynamics, which allows the reduced representation to be used as an early-warning tool. Furthermore, we consider their analysis of associating the curvature of the manifold with a global indicator of the underlying attractor’s rigidity or resilience to be extremely relevant. However, we observe two relevant methodological limitations. First, the Procrustes metric measures global geometric preservation but is not sensitive to local distortions. Second, although the method is information-based, no metric was introduced to evaluate how well the embedding preserves divergence structure in the original space. Consequently, the comparative evaluation does not fully examine the method along the most relevant theoretical dimensions. This leads us to propose a local metric that can explicitly characterize aspects associated with the loss or preservation of information. 3 Background and Related Work 3.1 Manifold Learning Methods Manifold learning methods are commonly categorized according to the type of geometric structure they are designed to preserve. In this work, we adopt the view that, despite differences in implementation, these methods share the

Read the complete text