Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings

V. S. Raghu Parupudi [email protected] · 2026

Read Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings by V. S. Raghu Parupudi [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29571.txt.

Introduction

Parupudi settles a practical question about text embeddings, the number vectors that represent text for search and matching. Cosine similarity is the standard way to compare two embeddings. Scattered reports claim other metrics do better, without pinning down when or why. His study tests nineteen parameter-free similarity metrics across nineteen encoders, from compact sentence transformers to large language models, on seven datasets.

The deciding factor is geometry. When an encoder spreads variance evenly across directions, cosine similarity wins and no rival helps by a usable margin. When variance piles into a few dominant directions, a property called anisotropy, rank-based metrics beat cosine by a clear relative margin. One number, the share of variance in the single most dominant dimension, predicts the gain strongly across all encoders. Removing those dominant directions makes cosine recover, which points to concentration as the cause.

Catalog record

CREATORV. S. Raghu Parupudi [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29571

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings V. S. Raghu Parupudi [email protected] Abstract The standard way to compare two text embeddings is cosine similarity. Scattered studies report that a different metric does better, but they never pin down the geometric condition that decides when, or the reason why. We settle both with a comprehensive empirical study: nineteen parameter-free similarity metrics on nineteen encoders, from compact sentence transformers up to seven-billion-parameter large language models, across seven datasets. The answer is geometric. When an encoder spreads its variance evenly across directions, cosine similarity is the best parameter-free choice and no other metric helps by a usable margin. When the variance concentrates into a few dominant directions, a property known as anisotropy, rank-based and L 1 L_{1} -type metrics beat cosine by a clear margin. The absolute gain is modest, but because cosine starts low on these encoders it is a sizable relative improvement, around twenty percent on average and largest where cosine is weakest. What decides this is the geometry of the embedding space, not how the model was trained. When the two point different ways, the metric follows the geometry. One number captures the concentration, namely the fraction of variance held by the single most dominant dimension, and it predicts how much the alternatives help across all nineteen encoders, with a rank correlation of 0.86 0.86 and a linear correlation of 0.95 0.95 . To test concentration as the cause rather than a correlate, we project out the dominant directions: cosine recovers and the advantage of the other metrics nearly vanishes, but only on the encoders that were anisotropic to begin with. The effect is directional, not magnitude based, since it survives normalizing every vector to unit length, and the one metric that relies on length is the worst of the set. Among parameter-free metrics, then, cosine is the right tool wherever an encoder is well spread, which includes the fine-tuned embedders commonly deployed for retrieval and similarity, and we give a one-number diagnostic for when it is not. 1 Introduction Text embeddings turn sentences into vectors, with the idea that similar sentences end up as nearby vectors. Using them requires a way to measure how near two vectors are. Search, retrieval, clustering, and paraphrase detection all come down to this one measurement, and the usual choice is cosine similarity ( Reimers & Gurevych 2019 ) . People have questioned whether cosine is the right default. One line of work shows that embeddings from language models are anisotropic, which means the vectors bunch up in a narrow part of the space instead of spreading out ( Ethayarajh 2019 ; Gao et al. 2021 ) . When the vectors are all crowded together, even unrelated sentences look similar to cosine, so it loses its edge. Other papers suggest swapping cosine for a different simple metric. Some find that Euclidean distance retrieves better in certain settings ( Tessari et al. 2025 ) . Others reach for set-overlap measures like the Tanimoto coefficient from chemistry ( Bajusz et al. 2015 ) . Each of these comes from a narrow setup, and together they do not add up to a clear rule. No one has said when a non-cosine metric actually helps, or what about the encoder decides it. That is the gap we fill. We do not propose a new metric. We ask a plain question instead. For a given encoder and task, can any simple metric beat cosine, and if so, what controls how much it helps? To answer it we vary the two things that matter. One is the encoder, and we use nineteen of them, from small sentence transformers up to seven-billion-parameter language models, including multilingual ones. The other is the metric, and we use nineteen parameter-free ones taken from the literature on comparing high-dimensional vectors. We test every encoder and metric on seven datasets covering semantic similarity, paraphrase detection, and inference. We treat the rest of the paper as a small investigation. We start with what we see, ask why it happens, list the explanations that could account for it, and knock them out one by one until the real one is left. What we see is a clean split. On some encoders cosine is hard to beat, and the so-called alternatives turn out to be cosine in disguise. On the others, rank-based and L_{1} -type metrics beat cosine by about 0.05 in Spearman correlation, which is a real margin on these tasks. The split lines up almost perfectly with how the encoder was trained. Models trained with a cosine objective sit on the cosine-wins side, and plain language models sit on the other side. It would be easy to stop there and say training is the answer. The reason we do not is that training is not what the metric responds to. The metric responds to the shape of the embedding space, and training is just the usual way an encoder ends up with one shape or the other. Two encoders make this clear, because in them shape and training disagree. E5-Mistral is trained with a cosine objective, yet its space is crowded, and the alternatives help it. Multilingual BERT is a plain language model, yet its space is well spread, and the alternatives do not help it. Both follow their geometry, not their training label. So we investigate, looking for some property of the encoder, the task, or the metric that decides the outcome, and we rule out the easy explanations first. Size is not it, since two models of the same size land on opposite sides. Pooling is not it, since models that pool the same way spread across the whole range. The task is not it either, because within every task type the split still falls along geometry, with crowded encoders gaining and well-spread ones not. Weak cosine is not it, which we show by holding cosine’s own quality fixed and watching the effect survive. Vector length is not it, since the result barely moves when we rescale every vector to length one. What is left is how the variance is spread across directions. The encoders where alternatives help are the ones that pack most of their variance into a few directions, and a single measure of that packing predicts the gain across all nineteen encoders with a rank correlation of 0.86 and a linear correlation of 0.95 . A correlation is not proof, so we test the cause by removing it. If those few crowded directions are what break cosine, taking them out should let cosine recover and remove any reason for the other metrics. That is what happens. We project the top directions out, cosine recovers most of what it lost on the crowded encoders, the advantage of the alternatives nearly vanishes, and on encoders that were never crowded the same step changes little. This is not a clean causal proof, since removing the top directions also removes whatever else lives in them, but the result is what we would expect if the crowded directions are the cause. The contribution is the diagnosis, not a metric. The choice of similarity function matters only when an embedding space is packed into a few

Read the complete text