Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not apply
Vivi Nastase, Paola Merlo · 2025
Read Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not apply by Vivi Nastase, Paola Merlo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01606.txt.
Introduction
Transformer models place words with similar meanings near each other in an embedding space, and users often assume the reverse for sentences: near means alike. Vivi Nastase and Paola Merlo test whether that geometry holds for sentence embeddings. They compute sentence vectors three ways, by averaging word vectors, by taking a special classification token, and by taking a random token's vector.
The pair then asks whether distances between these variants predict their performance on linguistic tasks, and whether distant variants still encode the same information in the same way. Their results show the geometry of the sentence embedding space is not predictive of relative task performance. Nastase and Merlo warn that the cosine measure, the standard closeness score, need not apply as a guide to what sentence embeddings know.
Catalog record
| CREATOR | Vivi Nastase, Paola Merlo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-01606
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not apply Vivi Nastase, Paola Merlo Abstract Transformer models learn to encode and decode an input text, and produce contextual token embeddings as a side-effect. The mapping from language into the embedding space maps words expressing similar concepts onto points that are close in the space. In practice, the reverse implication is also assumed: words corresponding to close points in this space are similar or related, those that are further are not. Does closeness in the embedding space extend to shared properties for sentence embeddings? We present an investigation of sentence embeddings and show that the geometry of their embedding space is not predictive of their relative performances on a variety of tasks. We compute sentence embeddings in three ways: as averaged token embeddings, as the embedding of the special [CLS] token, and as the embedding of a random token from the sentence. We explore whether there is a correlation between the distance between sentence embedding variations and their performance on linguistic tasks, and whether despite their distances, they do encode the same information in the same manner. The results show that the cosine similarity – which treats dimensions shallowly – captures (shallow) commonalities or differences between sentence embeddings, which are not predictive of their performance on specific tasks. Linguistic information is rather encoded in weighted combinations of different dimensions, which are not reflected in the geometry of the sentence embedding space. 1 Introduction Projecting words and larger pieces of text into an n -dimensional space allows us to map linguistic objects into a well-defined mathematical space, with specific metrics and operations. Building this projection relies on equating word similarity in language with closeness between their corresponding vectors in the embedding space, that is, the embedding space is smooth Bengio et al. 2013 . The smoothness of the embedding space comes with several assumptions: similar or related words or sentences will be projected to points that are close in the space, and words or sentences corresponding to points that are close in the space are similar or related. Distance or similarity metrics in this space are the basis for the functioning of all current LLMs and their applications, and understanding the topology of the embedding space can bring insights both into the successes and also the failures of these models. Analysis of the embedding spaces of the tokens through similarity measures have revealed that the spaces of many LLMs are anisotropic, with most words appearing in a narrow cone in this space, thus making distance metrics less informative Timkey and van Schijndel 2021 ; Cai et al. 2021 . These analyses are shallow: each dimension of these vectors is treated independently of the other dimensions. The dimensions may encode some information at a shallow level – e.g. length, or extreme word frequencies within the sentence Nikolaev and Padó 2023b ; sentence structure through the words’ relative positions Manning et al. 2020 – but they do not correspond to linguistic features such as phrase type, or semantic role. The interplay among embedding dimensions is complex, as each can contribute to various linguistic features in different measures Bengio et al. 2013 ; Elhage et al. 2022 . This implies that the level at which words and sentence embeddings share features is not at a level that the cosine metric can detect. This explanation may also shed light on the apparent contradiction between the word embedding space being anisotropic Mimno and Thompson 2017 ; Timkey and van Schijndel 2021 ; Cai et al. 2021 while word embeddings still leading to good results on a variety of NLP tasks (e.g. Mercatali and Freitas 2021 ; Bao et al. 2019 ; Chen et al. 2019 ). We present an investigation of sentence embeddings and show that the geometry of their embedding space is not predictive of their relative performances on a variety of tasks. We consider four pretrained models from the BERT family . For each model, we study and compare three variations of sentence representations: the averaged token embeddings, the embedding of the special [CLS] token, and a random token embedding. While the vocabulary is finite, the sentences in a language are not. We therefore study the embedding space of sentences by comparing variations of representations for the same sentence, which should (theoretically) be very close as they should contain the same information. For this analysis, we start with the commonly used method –the cosine similarity– and establish how close these representation variations are on a dataset of sentences extracted from the ParaCrawl corpus Bañón et al. 2020 (Section 2.5 ). In the next step, we test these sentence representations on the FlashHolmes benchmark Waldis et al. 2024 , which contains morphological, syntactic, semantic, discourse and reasoning tasks, and test whether embeddings that are close in the embedding space lead to similar performance on linguistic tasks (Section 3 ). Finally, we mine for sentence structure information, and test whether this kind of information is encoded consistently across the three representation variations of a sentence (Section 4 ). The results show that closeness in the embedding space is not predictive of closeness of performance. In particular, for RoBERTa, all representation variations of a sentence are close in the embedding space, but their performance on the FlashHolmes tasks are very different. For Electra and DeBERTa, the S CLS representations are almost orthogonal to the S AVG representations, while having very close performance on the FlashHolmes tasks, and on deeper probing for syntactic structure. These seemingly contradictory results between distance in the embedding space and performance on various tasks support the view that the geometry of the embedding space is not a good proxy for investigating shared information among sentence embeddings. We discover linguistic features encoded through deeper combinations of weighted dimensions. Our work sheds lights on the embedding space, how linguistic information is encoded in embeddings and that cosine similarity is a shallow metric that does not provide information about (deep) shared features among words or sentences. 2 Sentence representation comparisons in the embedding space We investigate the embedding space of sentences, and whether the common assumption – that close points correspond to textual units that are similar – is true for sentence representations. 2.1 Sentence representations Averaged token embeddings: S AVG The representation obtained by averaging a sentence’s tokens (without the special [CLS] and [SEP] tokens) is frequently used as the sentence’s embedding ( Nikolaev and Padó 2023a ) . This representation benefits from the fact that the learning signal for transformer models is stronger at the token level, with a much weaker objective at the sentence level – next sentence prediction ( Devlin et al. 2019 ; Liu et al. 2019 ) , sentence order prediction ( Lan et al. 2019 ) . The embedding of the special [CLS] token: S CLS This representation is most commonly used after fine-tuning for specific tasks such as story continuation ( Ippolito et al. 2020 ) , sentence similarity ( Reimers and Gurevych 2019 ) , alignment to semantic features Opitz and Frank 2022 . The embedding of a random token: S {}_{T_{rand}} Using this as the sentence’s representation can reveal how much