Semantic Shift: the Fundamental Challenge in Text Embedding and Retrieval

Hang Gao, Dimitris N. Metaxas · 2026

Read Semantic Shift: the Fundamental Challenge in Text Embedding and Retrieval by Hang Gao, Dimitris N. Metaxas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21437.txt.

Introduction

Gao and co-authors argue that pooling, not text length alone, drives a key flaw in text embeddings. When a model pools diverse sentences into one vector, meaning gets diluted and vectors crowd together. They prove this effect in a unified framework and define semantic shift, the drift of meaning inside a text. Controlled tests show semantic shift best predicts severe embedding crowding. Retrieval tests add that anisotropy hurts mainly when strong semantic shift causes it.

Catalog record

CREATORHang Gao, Dimitris N. Metaxas
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82430
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21437

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semantic Shift: the Fundamental Challenge in Text Embedding and Retrieval Hang Gao, Dimitris N. Metaxas Abstract Transformer-based embedding models rely on pooling to map variable-length text into a single vector, enabling efficient similarity search but also inducing well-known geometric pathologies such as anisotropy and length-induced embedding collapse. Existing accounts largely describe what these pathologies look like, yet provide limited insight into when and why they harm downstream retrieval. In this work, we argue that the missing causal factor is semantic shift : the intrinsic, structured evolution and dispersion of semantics within a text. We first present a theoretical analysis of semantic smoothing in Transformer embeddings: as the semantic diversity among constituent sentences increases, the pooled representation necessarily shifts away from every individual sentence embedding, yielding a smoothed and less discriminative vector. Building on this foundation, we formalize semantic shift as a computable measure integrating local semantic evolution and global semantic dispersion. Through controlled experiments across corpora and multiple embedding models, we show that semantic shift aligns closely with the severity of embedding concentration and predicts retrieval degradation, whereas text length alone does not. Overall, semantic shift offers a unified and actionable lens for understanding embedding collapse and for diagnosing when anisotropy becomes harmful. 1 Introduction Text embeddings have become indispensable for retrieval, question answering, clustering, and a wide range of semantic processing tasks. Classic distributional methods (e.g. Word2Vec ( 41 ) , GloVe ( 46 ) , and fastText ( 6 ) ) have been largely superseded by Transformer-based Pretrained Language Models (PLM) such as BERT ( 14 ) and its variants ( 37 ) , as well as GPT-style models ( 49 ) , which produce context-sensitive representations that substantially improve semantic matching. Despite their empirical success, a growing body of work has revealed that embedding spaces exhibit non-trivial geometric pathologies . A widely discussed phenomenon is anisotropy , where embeddings concentrate into a narrow cone rather than being uniformly distributed ( 19 ; 16 ) . A series of post-processing and normalization techniques have been proposed to mitigate such issues, e.g. removing dominant directions ( 43 ; 2 ; 50 ) , whitening ( 56 ; 26 ) , or flow-based transformations ( 34 ) . However, recent analyzes suggest that global concentration metrics can be misleading and do not reliably predict semantic quality or downstream performance ( 58 ; 17 ; 1 ) . Recent work has identified and formalized length-induced embedding collapse, where embeddings of longer texts exhibit reduced variance and become increasingly difficult to distinguish ( 66 ) . They attribute this effect to the attention mechanism: as input length grows, the attention matrix exhibits a stronger low-pass filtering behavior, accelerating the suppression of high-frequency semantic variations and consequently driving long-text embeddings toward increasingly similar representations. These observations are important but incomplete: they characterize what embedding spaces look like, not why such structures arise or when they actually harm downstream performance. A striking paradox illustrates this gap. When we embed the same corpus using different models, the resulting Mean Pairwise Distance (MPD) ( 16 ; 1 ) , a common measure of concentration/anisotropy, can vary dramatically. Figure 1 illustrates how the MPD of sentence embeddings evolves on two corpora, ArXiv ( 12 ) and Alice’s Adventures in Wonderland ( 48 ; 7 ) , under several widely used embedding models bge-large ( 62 ) , e5-large ( 61 ) , and all-mpnet ( 54 ) . In this experiment, texts are segmented into sentences, all sentences are embedded, and the MPD is computed incrementally over the first 1, 2, …, n sentences. As shown in Figure 1 , the MPD converges to a stable value once n becomes sufficiently large. However, the converged MPD differs drastically across models: bge-large stabilizes around 0.6 , e5-large around 0.2 , and all-mpnet around 0.8 . Despite these large discrepancies in embedding concentration, these models exhibit broadly comparable performance in practical downstream tasks. This observation makes it difficult to attribute degraded performance solely to anisotropy (i.e., embedding concentration), and it cannot be explained by length-induced embedding collapse either, since all models embed texts of identical length. Together, these findings raise a central question. If neither embedding concentration, anisotropy, nor length collapse can account for the behavior observed in Figure 1 , what factors—beyond model-specific effects—fundamentally drive embedding concentration and, more importantly, lead to difficulties in embedding-based retrieval? Figure 1: Mean Pairwise Distance (MPD) curves for three embedding models across two corpora. The x -axis is the number of sentences; the y -axis is MPD. In this paper, we argue that the fundamental factor is semantic shift : the intrinsic, structured evolution of semantics within a text. Natural language exhibits strong local coherence – adjacent sentences tend to be semantically similar, but this similarity naturally decays as one moves through the text. Over long ranges, the accumulated change can be substantial. This process resembles a "telephone game": each step preserves local information, yet the meaning at the end can differ markedly from the beginning. To substantiate this claim, we first provide a theoretical explanation for semantic smoothing in Transformer embeddings. We show that because Transformer encoders necessarily aggregate token-level representations through mean pooling or attention pooling, the resulting text embedding is effectively a convex combination of its constituent token/sentence embeddings. We further prove that as the pairwise semantic diversity among tokens/sentences increases, the aggregated embedding inevitably moves farther away from every individual token/sentence. This mathematically explains why embeddings for multi-sentence texts tend to under-represent any specific semantic component, shifting toward a compromise direction. This smoothing effect directly connects semantic diversity to length collapse and anisotropy. Building on this theoretical foundation, we introduce a formal definition of semantic shift that captures both local semantic evolution and global semantic dispersion. Using controlled experiments on synthetic concatenation patterns, we show that semantic shift, not text length, predicts the severity of embedding concentration. Furthermore, in the retrieval experiments, we observe that anisotropy becomes much more harmful when induced by strong semantic shifts, whereas the harm caused by anisotropy solely based on length is much less significant. 2 Semantic Smoothing in Transformer-Based Embedding Models 2.1 Token-Level Pooling and Its Sentence-Level Interpretation Transformer encoders construct text embeddings by aggregating contextualized token representations through a fixed pooling mechanism. In this section, we show that any pooling-based embedding model inevitably smooths and dilutes the semantics of a multi-sentence text, and that the extent of this dilution grows monotonically with the semantic diversity of the constituent sentences. This provides a theoretical foundation for understanding anisotropy, length collapse, and their connection to semantic shift. Let an input text be tokenized as (x_{1},\dots,x_{N}) , and let a Transformer encoder produce contextualized token embeddings h_{1},h_{2},\dots,h_{N}\in\mathbb{R}^{d}. (1) To obtain a fixed-length text embedding, all widely used models apply a pooling operator: z=\mathrm{Pool}(h_{1},\dots,h_{N}). (2) Two pooling mechanisms dominate practice: Mean pooling. Used in Sentence-BERT ( 51 ) , SimCSE ( 20 ) , E5 ( 61 ) , BGE ( 62 ) , and many other embedding models: z=\frac{1}{N}\sum_{t=1}^{N}h_{t}. (3) Attention-weighted pooling. Used in vanilla BERT ( 14 ; 11 ; 16 ) , where the [CLS] token representation after self-attention can be expressed as: h_{\mathrm{CLS}}=\sum_{t=1}^{N}\alpha_{t}\,v_{t}, (4) with attention-derived weights \alpha_{t}\geq 0 and \sum_{t}\alpha_{t}=1

Read the complete text