Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias
Elias Schuhmacher Andrianos Michail Juri Opitz Rico Sennrich Simon Clematide Department of Computational LinguisticsUniversity of Zurich <firstname>.<lastname>@uzh.ch · 2026
Read Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias by Elias Schuhmacher Andrianos Michail Juri Opitz Rico Sennrich Simon Clematide Department of Computational LinguisticsUniversity of Zurich <firstname>.<lastname>@uzh.ch (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16934.txt.
Introduction
Elias Schuhmacher, Andrianos Michail, Juri Opitz, Rico Sennrich, and Simon Clematide test if long texts get fair play in search codes. Search codes pack a text in one code line. Each part of the text should show in that line. Their 2026 preprint builds a swap test to gauge bias in the pack. Long texts with many parts are the test ground.
Top search codes show clear place and tongue bias. Parts near the start count more than late parts. Some tongues pack worse than the rest. Such bias can hide key facts in long files. Fair packs matter for law, care, and news search.
Catalog record
| CREATOR | Elias Schuhmacher Andrianos Michail Juri Opitz Rico Sennrich Simon Clematide Department of Computational LinguisticsUniversity of Zurich <firstname>.<lastname>@uzh.ch |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-16934
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias Elias Schuhmacher Andrianos Michail Juri Opitz Rico Sennrich Simon Clematide Department of Computational LinguisticsUniversity of Zurich <firstname>.<lastname>@uzh.ch Abstract To be discoverable in an embedding-based search process, each part of a document should be reflected in its embedding representation. To quantify any potential reflection biases, we introduce a permutation-based evaluation framework. With this, we observe that state-of-the-art embedding models exhibit systematic positional and language biases when documents are longer and consist of multiple segments. Specifically, early segments and segments in higher-resource languages like English are over-represented, while later segments and segments in lower-resource languages are marginalized. In our further analysis, we find that the positional bias stems from front-loaded attention distributions in pooling-token embeddings, where early tokens receive more attention. To mitigate this issue, we introduce an inference-time attention calibration method that redistributes attention more evenly across document positions, increasing discoverabiltiy of later segments. Our evaluation framework and attention calibration is available at github.com/impresso/fair-sentence-transformers 1 Introduction Text embedding models serve as the backbone of search engines and retrieval modules in Retrieval Augmented Generation (RAG) and agentic systems. These models map documents and queries into a shared vector space where cosine similarity determines discoverability—any distortion during embedding directly affects which information can be retrieved. Previous work has shown that models exhibit a pronounced positional bias —prioritizing information based on where it appears ( 15 ; 13 ; 4 ; 11 ; 8 ; 10 ; 6 ; 2 ; 9 ) . Longer documents often place distinct information in different sections—legal documents put definitions in appendices, technical manuals include troubleshooting after introductory content. When embedding models prioritize early content, key information in later positions becomes invisible to semantic search. Consider newspaper pages with multiple independent articles, sometimes in different languages. If the model prioritizes early content, later articles become undiscoverable. Structural conventions and placement determine which information is matched, often at the expense of relevance. Figure 1: We demonstrate that standard document encoders generate skewed embeddings that underrepresent later segments when processing multi-segment documents, such as newspaper pages. We show that an inference-time attention calibration method yields embeddings that are positionally fairer. Figure 1 illustrates this problem: due to positional bias, the representation of a multi-segment document is dominated by the first segment, making information from later segments invisible. We formalize this as a threat to Information Representation Fairness : when a document composed of multiple segments is embedded, each segment should contribute equally to the resulting vector representation, regardless of its position or language. Our analysis focuses on a general search scenario in which a user searches for document-level information using one of its segments. In this scenario, a fair document embedding should be equally similar to all of its constituent segments, regardless of segment position or source language. Making the first steps towards fairer embedding models, our contributions are: (1) an evaluation framework that uses a comparable corpus to quantify positional biases in long-context embedding models; (2) a mixed-language extension that reveals source language bias as an additional dimension of unfairness; (3) an analysis of the attention patterns of embedding models that use the <s> -token as the pooling token and identify attention based causes of positional bias; (4) an inference-time attention calibration method that produces positionally fairer embeddings. 2 Related Work Positional Bias in Long-Context Models Early evidence of long-context limitations of transformer-based models was provided by 10 , who introduced a passkey retrieval challenge to test decoder language models and showed that these models often fail to retrieve the correct key when it is buried deep in the input. Similarly, 8 observe a “lost in the middle” phenomenon stating that decoder models struggle to use information placed in the middle of their input context as opposed to information placed at the start or end. 6 showed that embeddings of encoder models are more strongly influenced by perturbations at the start of a text than by identical perturbations in the middle or at the end. 13 empirically demonstrated that, when performing question answering retrieval, embedding models exhibit higher performance degradation when the related content is placed later within the context. Attention Calibration in Decoder-Based Language Models 7 proposed a method to counteract the U-shaped positional bias in decoder-based language models, attributing the “lost in the middle” phenomenon ( 8 ) to a corresponding U-shaped pattern in decoder self-attention. This method estimates positional bias by performing one inference per position while shifting a fixed dummy segment across all (e.g., 20) positions, thereby obtaining a baseline attention distribution that reflects positional bias. The estimated bias is then subtracted from the observed segment attention to produce calibrated attention, which is used during a “positionally fair” inference. 3 Methods We introduce the notation of segments and documents used in this work and define the two concepts: positional fairness and information retention . Notation Let \Sigma be a finite vocabulary (alphabet) of tokens, and let \Sigma^{*} denote its Kleene closure–the set of all finite token sequences over \Sigma . We define a segment s\in\Sigma^{*} : a logically coherent unit of text of arbitrary length. Each segment is treated as an indivisible unit of information. Further, we define a segment set S=\{s_{1},\ldots,s_{n}\} : an unordered collection of n segments, with s_{i}\neq s_{h} for i\neq h . Finally, we define a document D\in\Sigma^{*} : a concatenation of multiple segments. Given a segment set S , we construct n! distinct documents by permuting segment order. We employ this permutation setup to eliminate the segment content as a confounding factor in our analysis. Given a segment set S of size n\geq 1 and a specific ordering j out of all n! distinct orderings, document D_{j} is the concatenation (with whitespace) of the n segments in S following this ordering. We denote the segment at position i within document D_{j} as s_{i}^{D_{j}} , and use 1-indexed positions. Figure 2: Monolingual experiment instances (n,\mathbf{L}) , where n varies across rows, and \mathbf{L} varies across columns. Left y-axes show (i) average representation in the global document embedding (mGTE and jina-v3), and (ii) average information retention (jina-v3) per segment position. (Gray bars) show average token length per segment position. We consider text embedding models with \mathscr{L} transformer layers, following the functional form: F_{\theta}:\Sigma^{*}\to\mathbb{R}^{d} . We write \mathbf{e}_{s_{i}^{D_{j}}}^{iso}=F_{\theta}({s_{i}^{D_{j}}}) to denote the embedding of segment i from document D_{j} . Note that this is the embedding of the segment isolated on its own as standalone text, i.e., without contextual information from other segments in D_{j} . Similarly, \mathbf{e}_{D_{j}}=F_{\theta}({D_{j}}) the embedding of a document D_{j} . For mean-pooled models, we write \mathbf{e}_{s_{i}^{D_{j}}}^{ctx} to denote the contextualized segment embedding of the segment at position i within document D_{j} : \mathbf{e}_{s_{i}^{D_{j}}}^{ctx}=\frac{1}{|P(s_{i}^{D_{j}})|}\sum_{p\in P(s_{i}^{D_{j}})}\mathbf{h}_{p}^{(\mathscr{L})}, with \mathbf{h}_{p}^{(\mathscr{L})} the contextual token representation at the final layer \mathscr{L} of embedding model F_{\theta} for the token at position index p , P(s_{i}^{D_{j}}) the set of token index positions belonging to segment s_{i}^{D_{j}} within document D_{j} , and |P(s_{i}^{D_{j}})| the number of tokens in segment s_{i}^{D_{j}} . As segment-wise mean pooling requires meaningful token-level representations, this operation is only applicable for embedding models preserving token-level semantics during fine-tuning, such as mean-pooled models. Figure 3: Preferential treatment of English segments: later