Attention Calibration for Position-Fair Dense Information Retrieval

Andrianos Michail Elias Schuhmacher Juri Opitz Simon Clematide Rico Sennrich Department of Computational LinguisticsUniversity of Zurich [email protected] · 2026

Read Attention Calibration for Position-Fair Dense Information Retrieval by Andrianos Michail Elias Schuhmacher Juri Opitz Simon Clematide Rico Sennrich Department of Computational LinguisticsUniversity of Zurich [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02737.txt.

Introduction

Michail and co-authors fix a position bias in dense text retrieval. Dense retrieval packs a passage into one vector. Early content dominates that vector. Relevant spans near the end get short shrift. A prior method equalized the pooling token's attention across segments. It used fixed strength and a fixed self-attention mass. Its effect on retrieval was never tested. The new work adds control.

A strength coefficient blends raw and fully equalized attention. A lean implementation cuts calibration memory from gigabytes to under a megabyte. A variant keeps the pooling token's self-attention mass intact. Tests span three embedding models and two pooling schemes. Moderate calibration beats full equalization as a trade-off. On a benchmark across 10 languages and 31 domains, one setting cuts position sensitivity in all 16 test combinations. Gains reach 43 percent in relative terms. Retrieval quality also improves slightly. Calibration runs at indexing time, so queries stay fast.

Catalog record

CREATORAndrianos Michail Elias Schuhmacher Juri Opitz Simon Clematide Rico Sennrich Department of Computational LinguisticsUniversity of Zurich [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02737

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention Calibration for Position-Fair Dense Information Retrieval Andrianos Michail Elias Schuhmacher Juri Opitz Simon Clematide Rico Sennrich Department of Computational LinguisticsUniversity of Zurich [email protected] Abstract Dense retrieval models exhibit positional bias: retrieval effectiveness degrades when relevant information appears later in a passage ( 6 ) . We ask whether this bias can be reduced at inference time, without retraining and without sacrificing overall retrieval effectiveness. To this end, we adapt inference-time attention calibration ( 5 ) to downstream retrieval and extend it with a strength coefficient λ \lambda that interpolates between the original and fully calibrated attention distributions. Across three embedding models on SQuAD-PosQ and FineWeb-PosQ, we examine how basket size, calibrated layer set, and strength affect the trade-off between positional fairness and retrieval effectiveness, finding that partial calibration frequently outperforms full calibration. A single configuration ( 𝔅 = 128 \mathfrak{B}=128 , λ = 0.5 \lambda=0.5 , 50% layer depth) improves the harmonic mean of nDCG@10 across positional groups on FineWeb-PosQ for all three models without per-model tuning, and applies to both <s> -pooled and last-token-pooled architectures. This default configuration transfers without modification to PosIR, which spans 10 languages and 31 domains, reducing the Position Sensitivity Index in all 16 length-quartile × \times model × \times retrieval-setting combinations, while preserving or improving aggregate nDCG@10. We release our extended codebase at github.com/impresso/fair-sentence-transformers Figure 1: Attention calibration with strength coefficient \lambda . Top: The pooling token’s original, front-loaded attention. Bottom: The calibrated distribution after basket-level equalization. The coefficient \lambda\in[0,1] linearly interpolates between them. 1 Introduction Dense retrieval models are central to modern information retrieval systems, mapping queries and documents into a shared embedding space to enable efficient semantic matching. However, recent work by 6 reports that such models exhibit positional bias : retrieval effectiveness degrades when relevant information appears later in a passage. Using two position-aware benchmarks, they show that dense embedding models are sensitive to answer position. 5 investigate positional imbalance at the representation level and propose an inference-time attention calibration method that redistributes attention mass more evenly across document segments. While this approach reduces positional skew in representation profiles, its effect on downstream retrieval has not been examined, and their method offers no control over the strength of the redistribution. In this work, we extend attention calibration with a tunable strength coefficient \lambda and study when it mitigates positional bias in dense retrieval without sacrificing retrieval effectiveness. Figure 1 illustrates the calibration procedure and our strength coefficient. We characterize the effects of the calibration parameters—basket size \mathfrak{B} , calibrated layer set \mathfrak{L}^{C} , and strength \lambda —on the trade-off between positional consistency and retrieval quality. We use this analysis to identify a model-agnostic default configuration that applies across embedding models with different pooling mechanisms ( <s> -pooling and last-token pooling). We then validate this configuration at scale on the Position Aware IR Benchmark (PosIR), covering 10 languages, 31 domains, and two embedding models in both multilingual and cross-lingual retrieval settings. We make the following contributions: • Calibration strength coefficient: We extend the attention calibration method of 5 with a strength coefficient \lambda\in[0,1] that linearly interpolates between the original and calibrated attention distributions, enabling controlled exploration of the trade-off between positional redistribution and retrieval effectiveness. We also test whether attention calibration is specific to <s> -pooled models, where early-token over-attention may be expected from the pooling-token position, and show that the same intervention applies to last-token-pooled models. • Systematic study of calibration parameters: We analyze how basket size \mathfrak{B} , calibrated layer set \mathfrak{L}^{C} , and strength \lambda jointly control the balance between positional consistency and retrieval effectiveness, identifying cases where calibration improves or degrades downstream performance. • Model-agnostic default configuration: We identify a modest partial-calibration setting ( \lambda=0.5 , 50% of layers, \mathfrak{B}=128 ) that improves the harmonic mean of nDCG@10 across positional groups and reduces positional sensitivity on FineWeb-PosQ across all three evaluated embedding models without per-model tuning. • Large-scale generalization evidence: We validate the identified configuration on PosIR across 10 languages, 31 domains, and two embedding models, showing consistent reductions in positional sensitivity with preserved or improved retrieval effectiveness in both multilingual and cross-lingual (x \rightarrow en) settings. 2 Related Work Position Bias in Dense Retrieval. 6 introduced two position-aware evaluation sets to study how the location of relevant information within a passage affects dense retrieval effectiveness. SQuAD-PosQ repurposes the SQuAD v2 test set by removing unanswerable questions, yielding 92,749 examples. Retrieval is performed over the full corpus and questions are grouped by the character position where the answer begins in the reference passage: [0–100], [100–200], [200–300], [300–400], [400–500], and [500–3,120]. nDCG@10 decreases as the answer position moves later in the passage. FineWeb-PosQ addresses the short average passage length of SQuAD ( \approx 117 words) and potential training data leakage. The authors sample 13,902 passages of 500–1,024 words from FineWeb-edu, divide each into beginning, middle, and end segments, and generate section-specific questions with GPT-4o-mini, resulting in 25,775 questions. Retrieval performance again degrades for the middle and end segments. Position Aware IR Benchmark ( 7 ) extends this line of work to multilingual and cross-lingual retrieval. It comprises 310 datasets spanning 10 languages and 31 domains, with relevance tied to precise reference spans. Its central methodological feature is a length-controlled bucketing strategy that groups queries by the length of the positive document (Q1: 512, Q2: 1024, Q3: 1536, Q4: 2048 tokens) and analyzes positional effects within each bucket, isolating position bias from length-induced degradation. Results across ten state-of-the-art embedding models show that position bias is pervasive and intensifies with document length. Figure 2: Evaluation segments and attention calibration baskets for SQuAD-PosQ (a) and FineWeb-PosQ (b). Baskets partition the token sequence independently of evaluation groupings; \mathfrak{B} determines the size of baskets and the granularity of redistribution. Attention Calibration 5 analyze how information from different document positions contributes to the final embedding representation. Using a permutation-based evaluation framework, they show that multilingual encoder-based embedding models exhibit systematic positional imbalance: early segments are over-represented in the document embedding, while later segments are progressively less reflected. They attribute this effect to front-loaded attention distributions of the pooling token, whose query assigns disproportionate attention mass to early key tokens across transformer layers. However, their analysis focuses on representation profiles rather than downstream retrieval, and it remains open whether the same attention-based mechanism applies across pooling architectures. To mitigate this imbalance, they propose an inference-time attention calibration that redistributes the pooling token’s attention more evenly across document positions. The method partitions the key sequence into contiguous baskets of size \mathfrak{B} , isolating the document representation token (e.g., <s> or </s> ) in its own basket to avoid redistributing special-token attention across content tokens, and enforces equal total attention mass per basket while preserving relative attention weights within each basket. For a token at position k assigned to basket b(k) , the calibrated attention weight is: a^{\mathrm{cal}}_{k}=\frac{a_{k}}{\sum_{j\in b(k)}a_{j}}\cdot\frac{1}{n_{\mathrm{baskets}}}, where a_{k} is the original post-softmax attention weight and n_{\mathrm{baskets}} is the number of baskets. Calibration is applied only to the pooling token’s query row in selected transformer layers and requires no retraining. While this method was shown to flatten representation profiles in a controlled permutation setting, its effect on downstream retrieval has not been examined. 3 Method

Read the complete text