Topological Metric for Unsupervised Embedding Quality Evaluation

Aleksei Shestov, Anton Klenitskiy, Daria Denisova, Amurkhan Dzagkoev, Daniil Petrovich, Andrey Savchenko, Maksim Makarenko · 2025

Read Topological Metric for Unsupervised Embedding Quality Evaluation by Aleksei Shestov, Anton Klenitskiy, Daria Denisova, Amurkhan Dzagkoev, Daniil Petrovich, Andrey Savchenko, Maksim Makarenko (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15285.txt.

Introduction

Aleksei Shestov and six coauthors propose a way to grade learned embeddings without any labels. Modern representation learning trains on huge unlabeled corpora, and its embeddings generalize well. Yet judging their quality normally needs labeled downstream tasks, which unsupervised work tries to avoid.

Their metric, called Persistence, comes from persistent homology, a topology tool that tracks shapes across scales. It quantifies the geometric structure and topological richness of an embedding space in a fully unsupervised way. Unlike metrics that assume linear separability or simple covariance, it captures global, multi scale organization. Across diverse domains, Persistence matched downstream performance as well as or better than existing unsupervised metrics. It reached top tier correlations, making it a reliable guide for choosing models and hyperparameters without labels.

Catalog record

CREATORAleksei Shestov, Anton Klenitskiy, Daria Denisova, Amurkhan Dzagkoev, Daniil Petrovich, Andrey Savchenko, Maksim Makarenko
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-15285

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Topological Metric for Unsupervised Embedding Quality Evaluation Aleksei Shestov, Anton Klenitskiy, Daria Denisova, Amurkhan Dzagkoev, Daniil Petrovich, Andrey Savchenko, Maksim Makarenko Abstract Modern representation learning increasingly relies on unsupervised and self-supervised methods trained on large-scale unlabeled data. While these approaches achieve impressive generalization across tasks and domains, evaluating embedding quality without labels remains an open challenge. In this work, we propose Persistence , a topology-aware metric based on persistent homology that quantifies the geometric structure and topological richness of embedding spaces in a fully unsupervised manner. Unlike metrics that assume linear separability or rely on covariance structure, Persistence captures global and multi-scale organization. Empirical results across diverse domains show that Persistence consistently achieves top-tier correlations with downstream performance, outperforming existing unsupervised metrics and enabling reliable model and hyperparameter selection. 1 Introduction Modern representation learning increasingly relies on unsupervised and self-supervised methods that leverage large-scale unlabeled data through proxy or contrastive objectives [ 5 , 19 , 13 , 18 , 23 , 14 ] . In parallel, zero-shot models have shown strong generalization to unseen domains and tasks [ 3 , 22 , 27 ] . As these approaches operate without labels, a key challenge is to assess embedding quality and guide model selection without supervised validation [ 1 , 31 ] . Reliable label-free metrics are thus essential for scalable and autonomous learning, including applications in recommender systems and information retrieval. Embeddings must preserve the geometric structure of the data manifold to perform well on unseen tasks. Intuitively, collapsing distinct regions or losing multi-scale relationships limits downstream performance. Conversely, embeddings retaining intrinsic geometric complexity better support diverse tasks. This motivates evaluating embeddings by the richness of their geometric structure. Measuring geometric structure in a principled, model-agnostic way is challenging. Existing metrics rely on simplifying assumptions: RankMe [ 11 ] assumes linear subspaces, \alpha -ReQ [ 1 ] and NESum [ 17 ] use covariance models that fail on multimodal data, and \mu_{0} -incoherence [ 30 ] presumes linear separability. SelfCluster [ 30 ] suits contrastive setups but poorly generalizes beyond them. This highlights the need for more holistic, model-agnostic evaluation. Topological methods naturally address these limitations by capturing global and multi-scale data structure through connected components, loops, and cavities that persist across scales [ 4 , 10 ] . Intuitively, embeddings that preserve rich data structure exhibit long-lived topological features. Persistent homology, the core tool of Topological Data Analysis (TDA), quantifies such features via persistence diagrams [ 12 , 26 ] and has been successfully applied in biology [ 25 ] , materials science [ 20 ] , and neuroscience [ 29 ] . However, its use for evaluating unsupervised embeddings remains largely unexplored. To address this gap, we introduce Persistence 1 1 1 CODE: https://anonymous.4open.science/r/topo_metrics-94D6/ , a topological metric for unsupervised embedding quality evaluation. Persistence constructs a Vietoris–Rips complex over the embedding point cloud and computes persistence diagrams for homology groups H_{0} and H_{1} . The total persistence, defined as the sum of topological feature lifetimes, quantifies the geometric richness of the embedding space (see Section 2 ). Unlike prior methods, Persistence makes no assumptions about embedding geometry, downstream tasks, or objectives, enabling a model-agnostic evaluation across domains. In extensive benchmarks covering financial analytics, user behavioral modeling (UBM), and collaborative filtering, Persistence achieves the highest or near-highest correlations across all configurations, with mean Spearman correlations of up to 0.84 in UBM and 0.76 in collaborative filtering. In the model selection per epoch setting, it reaches Spearman 0.61 and quality 0.71, outperforming competing metrics such as RankMe, \alpha -ReQ, and \mu_{0} -incoherence. Overall, Persistence consistently aligns with supervised performance, generalizes across different domains and tasks, and establishes a new state of the art in geometry-aware, label-free embedding evaluation. 2 Proposed Approach Figure 1 : Vietoris–Rips filtration and persistence barcode. (a) Evolution of the complex as the radius r grows, showing transitions from isolated points to a connected, contractible structure. (b) Complex at r=0.12 with metric spheres, illustrating two-point and three-point H_{0} deaths. (c) Corresponding persistence barcode with H_{0} and H_{1} intervals across scales. Topology-Aware Metric for Embedding Spaces. We introduce a topology-aware metric based on persistent homology [ 12 ] to quantify the intrinsic structure of embedding spaces. Figure 1 illustrates the computation of persistent homology for a set of embedding vectors. Given a point cloud of embeddings X_{n}=\{x_{1},x_{2},\ldots,x_{n}\}\subset\mathbb{R}^{d} , we construct a Vietoris–Rips filtration to track how topological features emerge and vanish as the connectivity radius r increases. As r grows, isolated points begin to connect via edges, forming larger connected components and transient one-dimensional loops (Fig. 1 a). At a specific scale, for instance r=0.12 , we visualize the resulting simplicial complex together with metric spheres centered at the embedding points. When two spheres first intersect, their corresponding points are joined by an edge, representing a two-point death of an H_{0} feature (a component merge). When three spheres intersect simultaneously, the corresponding points form a filled triangle (a 2-simplex), leading to a three-point death, which fills a loop and terminates an H_{1} feature (Fig. 1 b). The persistence barcode summarizes these topological transitions across scales: each horizontal bar corresponds to a topological feature that appears at its birth value b_{i} and disappears at its death value d_{i} - either by merging with another component ( H_{0} ) or being filled by higher-dimensional simplices ( H_{1} ) (Fig. 1 c). Finally, the total persistence in homology dimension k quantifies the overall topological complexity as the normalized sum of feature lifetimes: \text{Persistence}_{k}(X_{n})=\frac{\sum_{(b_{i},d_{i})\in H_{k}}(d_{i}-b_{i})}{\max_{i,j}\|x_{i}-x_{j}\|}, (1) where H_{k} denotes the set of persistence pairs (b_{i},d_{i}) in dimension k , and the denominator normalizes by the maximum pairwise distance between embeddings in X_{n} . Linking Persistence to Intrinsic Dimension. The core intuition behind the proposed approach is that the geometry of an embedding determines how much of the original data manifold it preserves. When embeddings maintain rich topological structure, connected components and loops persist over larger connectivity scales, leading to higher total persistence. This persistence serves as a proxy for the intrinsic dimension of the embedding manifold. Let X_{n}=\{x_{1},x_{2},\ldots,x_{n}\}\subset\mathbb{R}^{d} denote a sample of n points drawn uniformly from a smooth d -dimensional manifold M with volume \mathrm{Vol}(M) and diameter \mathrm{Diam}(M) . The expected zero-dimensional persistence then scales as \mathbb{E}[\mathrm{Persistence}_{0}(X_{n})]=\alpha(d)\,n^{1-\frac{1}{d}}\frac{\mathrm{Vol}(M)^{1/d}}{\mathrm{Diam}(M)}+o\!\left(n^{1-\frac{1}{d}}\right), (2) where \alpha(d) is a dimension-dependent constant and o(\cdot) denotes a higher-order remainder term [ 32 ] . Higher intrinsic dimension d results in more gradual topological merging, reflected in larger persistence values. Conversely, embeddings with dimension lower than that of the data manifold collapse regions of M , reducing the number of distinguishable events and degrading classification accuracy. Thus, high total persistence indicates that the embedding retains sufficient intrinsic dimensionality to preserve discriminative structure, explaining its observed correlation with improved downstream performance. 3 Experiments 3.1 Experimental Setup Domains and Datasets . We evaluate Persistence across three application families that jointly span structured behavioral data, event-sequence modeling, and user–item interactions: financial analytics, user behavioral modeling (UBM), and recommendation systems. This setup ensures a diverse evaluation covering both domain-specific prediction tasks and general-purpose interaction modeling. In the financial analytics family, we use two real-world banking datasets for gender and age group prediction [ 9 , 8 ] . The UBM family is represented by

Read the complete text