PosIR : Position-Aware Heterogeneous Information Retrieval Benchmark
Ziyang Zeng, Dun Zhang, Xu Sun, Yudong Zhou, Yuqing Yang · 2026
Read PosIR : Position-Aware Heterogeneous Information Retrieval Benchmark by Ziyang Zeng, Dun Zhang, Xu Sun, Yudong Zhou, Yuqing Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08363.txt.
Introduction
Ziyang Zeng and four co-authors present PosIR, a benchmark for position-aware information retrieval. Their 2026 arXiv paper notes that dense retrieval models perform strongly but face an untested weakness. How sensitive they are to where relevant evidence sits has barely been measured. Existing benchmarks use position-agnostic relevance labels. That mixes long-context difficulty with bias against certain evidence locations. PosIR separates the two so builders can see if a retriever favors text at the start or end. The distinction matters for long documents in search and question answering.
Catalog record
| CREATOR | Ziyang Zeng, Dun Zhang, Xu Sun, Yudong Zhou, Yuqing Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 111 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08363
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PosIR : Position-Aware Heterogeneous Information Retrieval Benchmark Ziyang Zeng, Dun Zhang, Xu Sun, Yudong Zhou, Yuqing Yang Abstract While dense retrieval models have achieved remarkable success, rigorous evaluation of their sensitivity to the position of relevant information (i.e., position bias) remains largely unexplored. Existing benchmarks typically employ position-agnostic relevance labels, conflating the challenge of processing long contexts with the bias against specific evidence locations. To address this challenge, we introduce PosIR ( Pos ition-Aware I nformation R etrieval), a comprehensive benchmark designed to diagnose position bias in diverse retrieval scenarios. PosIR comprises 310 datasets spanning 10 languages and 31 domains, constructed through a rigorous pipeline that ties relevance to precise reference spans, enabling the strict disentanglement of document length from information position. Extensive experiments with 10 state-of-the-art embedding models reveal that: (1) Performance on PosIR in long-context settings correlates poorly with the MMTEB benchmark, exposing limitations in current short-text benchmarks; (2) Position bias is pervasive and intensifies with document length, with most models exhibiting primacy bias while certain models show unexpected recency bias; (3) Gradient-based saliency analysis further uncovers the distinct internal attention mechanisms driving these positional preferences. In summary, PosIR serves as a valuable diagnostic framework to foster the development of position-robust retrieval systems. 1 1 1 https://github.com/Ziyang1060/PosIR 1 Introduction The rapid advancement of dense retrieval has largely outpaced the evolution of rigorous evaluation protocols. While modern embedding models demonstrate impressive capabilities, characterizing their fine-grained behaviors requires benchmarks designed with higher diagnostic precision. In recent years, the Information Retrieval (IR) community has benefited from a series of milestone benchmarks. Early contributions like MS MARCO 18 establish baselines for monolingual passage retrieval. Recognizing the linguistic diversity of the web, datasets such as MIRACL 35 extend the evaluation to multilingual scenarios. Additionally, benchmarks like BEIR 25 , MMTEB 8 and AIR-Bench 3 aggregat diverse tasks to test general-purpose embeddings. These efforts have been instrumental in advancing the field. Figure 1: The four-stage data generation pipeline of PosIR . However, a critical dimension remains largely unexplored in existing benchmarks: position bias in information retrieval 11 . Specifically, current benchmarks generally operate on a position-agnostic assumption, providing relevance labels without specifying the location of the evidence. This conflation creates a diagnostic blind spot: when a retrieval model fails on a document, it is unclear whether the failure stems from the sheer volume of context (capacity limitation) or from bias against the location of the relevant span (positional sensitivity). Although 5 ; 33 have observed primacy bias 2 2 2 Primacy bias refers to models disproportionately focusing on the beginning of documents while overlooking relevant content appearing later. in embedding-based retrieval models, their experimental setups lack sufficient granularity to disentangle document length from information position . Without a controlled evaluation that orthogonalizes these variables, position bias cannot be strictly isolated from the model’s capacity to handle long contexts. Furthermore, the extent to which such bias varies across multilingual and cross-lingual retrieval settings remains an open question. To address these gaps, we introduce PosIR ( Pos ition-Aware I nformation R etrieval), a comprehensive benchmark designed to rigorously evaluate and analyze position bias, which is characterized by three distinctive features: 1) Position-Aware : PosIR goes beyond coarse-grained document labels by associating each query with a precise reference span. By verifying relevance through strict span-based contrast, we enable the first systematic quantitative analysis of how the physical location of information impacts retrieval performance. 2) Heterogeneous : PosIR is designed to test robustness across diverse linguistic and semantic landscapes. It covers 31 domains and 10 languages (English, Chinese, and 8 translated languages), resulting in 310 datasets. This heterogeneity ensures that our findings on position bias are universal and not artifacts of a specific language or topic. 3) Length-Diverse : PosIR adopts a length-aware sampling strategy that spans documents from short passages to long texts. 3 3 3 Up to 2048 tokens for native English and Chinese, with longer sequences for translated languages. This design enables the disentanglement of length and position factors, and exposes critical model behaviors, such as performance degradation in long-context settings, which are often overlooked by short-text benchmarks. Using PosIR , we conduct extensive experiments on 10 popular IR models, yielding valuable insights into their performance characteristics: • Benchmarking Discrepancy : We find that retrieval performance on MMTEB correlates poorly with long-context retrieval performance on PosIR . Models that excel in short-context retrieval tasks often degrade significantly as document length increases. • Prevalence of Bias : Our evaluation reveals that position bias is pervasive and becomes more pronounced as document length increases, across both multilingual and cross-lingual retrieval settings. While most models exhibit primacy bias , favoring information at the beginning of documents, we identify unexpected recency bias in certain models. • Mechanistic Origins : Going beyond performance metrics, we employ gradient-based saliency analysis to uncover the internal mechanisms driving these biases. We identify two distinct attention behaviors that directly correlate with embedding models’ positional preferences over input tokens. 2 Benchmark Construction The entire data generation pipeline of PosIR consists of four stages: 1) Bilingual corpora preparation, 2) Position-aware candidate generation, 3) Quality control, and 4) Multilingual translation. For detailed information of the models appearing in this paper, please refer to Table 12 . 2.1 Bilingual Corpora Preparation As illustrated in Figure 1 , the first stage involves preparing high-quality bilingual corpora across diverse domains. Specifically, we leverage IndustryCorpus2 4 4 4 https://huggingface.co/datasets/BAAI/IndustryCorpus2 , an open-source and carefully curated English-Chinese dataset that covers 31 industry categories, broadly representing mainstream real-world application domains. We exclude the original Others category to maintain domain specificity and introduce FineWeb to represent the general domain: for English, we use the fineweb sampled 10BT subset 5 5 5 https://huggingface.co/datasets/HuggingFaceFW/fineweb/tree/main/sample/10BT , while for Chinese we adopt Fineweb-Edu-Chinese-V2.1 6 6 6 https://huggingface.co/datasets/opencsg/Fineweb-Edu-Chinese-V2.1 . After this adjustment, the resulting corpus still consists of 31 domains. Then, we adopt a length-aware sampling strategy that moderately over-samples shorter documents, which are more prevalent in real-world corpora. Specifically, using the Qwen3 tokenizer 29 , we partition documents into eight length buckets with an interval of 256 tokens, covering lengths from 0 to 2048 tokens. We apply a sampling ratio of 3:3:3:3:2:2:2:2, such that documents with lengths between 0–1024 tokens are sampled at 1.5 \times the rate of those between 1024–2048 tokens, while maintaining equal sampling within each length range to guarantee document length diversity. Following this strategy, we sample approximately 70k documents for each domain from both the English and Chinese corpora. Formally, for a given domain, we denote the English and Chinese corpora as \mathcal{D}_{en}=\{d_{i}\}_{i=1}^{n_{en}} and \mathcal{D}_{zh}=\{d_{i}\}_{i=1}^{n_{zh}} , containing n_{en} and n_{zh} documents, respectively. More generally, we use \mathcal{D}=\{d_{i}\}_{i=1}^{n} to denote a corpus in an arbitrary language, where n denotes the number of documents in that corpus. 2.2 Position-Aware Candidate Generation A retrieval dataset typically consists of three components: a document corpus, a set of queries, and relevance judgments (i.e., qrels). 7 7 7 In TREC parlance, qrels refer to relevance judgments that indicate whether a document is relevant to a given query. After preparing the corpus in the initial stage, the candidate generation stage constructs the remaining two components of the dataset—queries and qrels—while explicitly accounting for the positional nature of relevance. As shown in