Shrink the longest: improving latent space isotropy with simplicial geometry
Sergej Kudrjashov, Olesya Karpik, Eduard Klyshinsky · 2025
Read Shrink the longest: improving latent space isotropy with simplicial geometry by Sergej Kudrjashov, Olesya Karpik, Eduard Klyshinsky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05502.txt.
Introduction
Sergei Kudriashov and co-authors tackle a shape problem in language model embeddings in a 2025 paper. Word and sentence vectors from transformer models tend to bunch into a narrow cone, which blurs meaning differences. Spreading them out evenly, called isotropy, improves later task scores, but past fixes cost speed or need much data.
Their method borrows from simplicial geometry, the study of simplex shapes. During fine-tuning it rewards vector sets whose barcodes, a summary from topology, carry high persistent entropy. In plain terms, it shrinks the longest spread directions to even out the space. Tests show better downstream results with lower cone bunching. The gain comes from geometry already present in the data, not from retraining the model.
Catalog record
| CREATOR | Sergej Kudrjashov, Olesya Karpik, Eduard Klyshinsky |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-05502
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Shrink the longest: improving latent space isotropy with simplicial geometry Sergej Kudrjashov, Olesya Karpik, Eduard Klyshinsky Abstract Although transformer-based models have been dominating the field of deep learning, various studies of their embedding space have shown that they suffer from "representation degeneration problem" — embeddings tend to be distributed in a narrow cone, making the latent space highly anisotropic. Increasing the isotropy has shown to improve performance in downstream tasks both in static and contextual language models. However, most of approaches either add inference overhead or require substantial amount of data for model reparametrization. We propose a novel regularization technique based on simplicial geometry to improve the isotropy of latent representations. The core idea of our method is based on maximizing the persistent entropy of barcodes obtained using Vietoris-Rips filtration from contextual embeddings in the underlying latent space. We demonstrate that the method leads to an increase in downstream performance while significantly lowering the anisotropy during fine-tuning by exploiting existing geometric structures instead of reparametrization. The code is avaliable at github.com/Xenomirant/Shrink-the-longest 1 Introduction Since the first introduction in Vaswani et al. [ 20 ] , transformers have become a dominant SotA architecture in a variety of fields and applications. However, their impressive effectiveness and black-box nature made researchers wonder about their internal structure and seek reasons for their effectiveness and problems inherent to the class of models. Ethayarajh in [ 6 ] notes that contextual word representations in ELMO, BERT and GPT-2 are highly anisotropic. Moreover, the extent of anisotropy grows with the number of layers [ 17 ] . In other words, embeddings are not uniformly distributed in latent space, but clustered in a narrow cone, which limits the informational expressiveness of the space, making it impractical to be considered euclidean anymore. As [ 7 ] point out, it can be partly attributed to static embeddings being tied during pretraining [ 16 ] , which pushes the frequent tokens in opposite direction to all the others during pretraining. This is empirically justified by Liang et al. [ 13 ] finding high correlation between embedding L2 norm and frequency in the pretraining corpora. However, it is far from explaining the whole story. Kawin Ethayarajh shows that less than 5% of the variance in the word’s contextual representations could be explained by the geometry of static embeddings [ 6 ] . Despite the continuing debate on sources of anisotropy, preserving isotropy has shown to be a favorable property both theoretically and practically [ 21 , 17 , 9 , 13 ] – not only in semantic similarity tasks, which seem to benefit the most from richer informational capacity of contextualized embeddings, but in terms of model convergence as well. It also helps to mitigate the "stolen probability effect" [ 3 ] – a consequence of low embedding norm combined with overall positive cosine distance between embeddings, which results in a failure to attribute high probability to certain token embeddings in any context. Further investigation by Cai et al. [ 2 ] has shown that while being globally anisotropic, the latent consists of various clusters and manifolds that demonstrate significantly larger isotropy within themselves. This line of research was continued in [ 17 ] finding that most clusters are biased towards structural information, and e.g. verb tense representations are distributed in different subspaces. Thus, the clustering of the space is expressive and must be preserved, while we seek for methods to improve overall isotropy. Our contribution is the following. We address the problem by designing an additional regularization loss function inspired by recent results in computational topology [ 11 ] that preserves the representational geometry of the latent space, while improving the isotropy by maximizing entropy of distances between clusters. Thus we can optimize for isotropy while keeping the internal clustering structure intact, which differs from most of previously proposed methods. Our method is to our knowledge the first one to leverage the differentiability of persistent barcodes to mitigate the anisotropy of the latent space. It should be noted that although we utilize it for pretrained language models, the method is in fact model-agnostic, and can be applied to any vector-based representational model. The structure of the paper is as follows. In section 2 we review the recent methods proposed for anisotropy mitigation. In section 3 describe the method and give a brief overview of results in computational topology we took inspiration from. Section 4 presents an experimental setup, while section 5 is dedicated to the results we have obtained. Section 6 concludes the paper. 2 Related work To define and compute anisotropy we leverage the approach from Razzhigaev et al. in [ 18 ] and define it by exploiting a contextual embeddings’ matrix decomposition. Let X\in\mathbb{R^{N\times D}} represent the matrix of embeddings, where \sigma_{1},\dots,\sigma_{k} are its singular values, N is the number of samples and D is the hidden size. Then the k-th anisotropy score of X is given by: anisotropy_{k}(X)=\frac{\sigma_{k}^{2}}{\sum_{i=1}^{\min(\mathbb{N},\mathbb{D})}\sigma_{i}^{2}} Latest research has shown that anisotropy is a common trait in transformer-based architectures, where the overall contribution is split between attention [ 8 ] , LayerNorm [ 7 , 23 ] and various other factors. There have been multiple methods proposed recently to address these peculiarities of transformers’ latent space. Subsequent works of Gao et al. [ 7 ] and Zhang et al. [ 25 ] introduce explicit regularization of cosine distance between normalized static embeddings. However not only its computation is dependent on the vocabulary size, which appears to be costly for each gradient update, but it can hardly be directly reapplied to subsequent layers making it irrelevant to account for anisotropy induced by each of further layers. Another line of research was taken by Wang et al. [ 21 ] . They propose to reparametrize the initial embeddings matrix by a slow-decaying prior distribution over its singular values during training, resulting in substantially smoother decrease in isotropy dimension-wise. There have been also various attempts to adopt generative approaches for the task. Li et al. [ 12 ] utilized normalizing flows to map embeddings in the transformer’s latent space to an isotropic Gaussian. Zhang et al. [ 25 ] explore the effect of using VAE with isotropic Gaussian posterior noting that it significantly improves the classification performance and robustness to input perturbations. However, despite their utility in developing higher isotropy, all these approaches seem to have a flaw that prevents them from being fully utilized during fine-tuning of already pretrained models. They reparametrize the whole internal space of the model, completely changing its geometry, which can result in suboptimal results when the number of supervised data is small. This view is supported by results found in Ding et al. [ 5 ] . The authors evaluate aforementioned isotropy calibration methods on various GLUE tasks finding that none of them produce consistent improvements over the uncalibrated models across tasks, domains and architectures. They attribute it to the effectiveness of representations already found in local subspaces, which are utilized during fine-tuning. 3 Method Topological data analysis has recently found lots of use cases deep learning both due to its ability to represent global features, characteristic of the dataset, and due to recent results in differentiability of persistent homology [ 11 ]