T-Retrievability: A Topic-Focused Approach to Measure Fair Document Exposure in Information Retrieval Conference: Proceedings of the 34th ACM International Conference on Information and Knowledge Management; November 10–14, 2025; Seoul, Republic of Korea Proceedings of the 34th ACM International Con

Xuejun Chang, Zaiqiao Meng, Debasis Ganguly · 2025

Read T-Retrievability: A Topic-Focused Approach to Measure Fair Document Exposure in Information Retrieval Conference: Proceedings of the 34th ACM International Conference on Information and Knowledge Management; November 10–14, 2025; Seoul, Republic of Korea Proceedings of the 34th ACM International Con by Xuejun Chang, Zaiqiao Meng, Debasis Ganguly (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-21704.txt.

Introduction

Xuejun Chang and co-authors refine a fairness score for search systems, in work from 2025. The score, called retrievability, gauges how likely a document is to surface in top ranks. A flat spread of scores across a collection hints at fair exposure for all. Past work read a skewed spread as bias in the model.

The team argues a whole-set skew may just mirror topic mix, not bias. Their topic-focused score first rates groups of like documents, then merges those local rates. Tests on neural rankers showed new patterns of exposure. The authors offer the local score as a truer gauge of fair access.

Catalog record

CREATORXuejun Chang, Zaiqiao Meng, Debasis Ganguly
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-21704

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

T-Retrievability: A Topic-Focused Approach to Measure Fair Document Exposure in Information Retrieval Conference: Proceedings of the 34th ACM International Conference on Information and Knowledge Management; November 10–14, 2025; Seoul, Republic of Korea Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM ’25), November 10–14, 2025, Seoul, Republic of Korea DOI: 10.1145/3746252.3760820 ISBN: 979-8-4007-2040-6/2025/11 CCS: Information systems Retrieval models and ranking Xuejun Chang, Zaiqiao Meng, Debasis Ganguly Abstract . Retrievability of a document is a collection-based statistic that measures its expected (reciprocal) rank of being retrieved within a specific rank cut-off. A collection with uniformly distributed retrievability scores across documents is an indicator of fair document exposure. While retrievability scores have been used to quantify the fairness of exposure for a collection, in our work, we use the distribution of retrievability scores to measure the exposure bias of retrieval models. We hypothesise that an uneven distribution of retrievability scores across the entire collection may not accurately reflect exposure bias but rather indicate variations in topical relevance. As a solution, we propose a topic-focused localised retrievability measure, which we call T-Retrievability (topic-retrievability), which first computes retrievability scores over multiple groups of topically-related documents, and then aggregates these localised values to obtain the collection-level statistics. Our analysis using this proposed T-Retrievability measure uncovers new insights into the exposure characteristics of various neural ranking models. The findings suggest that this localised measure provides a more nuanced understanding of exposure fairness, offering a more reliable approach for assessing document accessibility in IR systems. † † cc-license: by 1. Introduction Supervised information retrieval (IR) models based on pre-trained transformer architectures, such as ColBERT ( Khattab and Zaharia 2020 ) and Mono-T5 ( Nogueira et al. 2020 ) , have demonstrated superior relevance ranking performance compared to their unsupervised counterparts, such as BM25 ( Robertson et al. 1994 ) . However, this paradigm shift necessitates the consideration of additional dimensions of effectiveness, including those of efficiency ( Bruch et al. 2024 ; Chang et al. 2024 ) , explainability ( Verma and Ganguly 2019 ; Pandian et al. 2024 ; Sen et al. 2020 ) , and exposure fairness ( Singh and Joachims 2018 ; Heuss et al. 2022 ; Patro et al. 2022 ) . In this study, we specifically examine the exposure fairness criterion of search systems, which has significant societal implications. For instance, ensuring a balanced distribution of topically relevant documents from diverse sources and ideological viewpoints is essential for preserving the integrity of information ecosystems ( Patro et al. 2022 ) , fostering an informed and unbiased public opinion, and supporting democratic decision-making ( Kulshrestha et al. 2017 ; Krause et al. 2024 ; Abolghasemi et al. 2024 ) . The data-driven learning of relevance in neural models makes it challenging to identify and mitigate latent biases present in the training data, which can manifest as algorithmic preferences leading to exposure biases ( Patro et al. 2022 ; Liu et al. 2023 ; Musto et al. 2021 ) . These exposure biases are expected ranks of retrieved documents computed over a large set of queries that several users may execute on a collection. In the case of queries with similar information needs and overlapping sets of relevant documents, an IR model that aims to ensure exposure fairness should distribute the rankings of these documents more evenly by shuffling their positions across queries ( Diaz et al. 2020 ; Ekstrand et al. 2023 ) . Assessing whether IR models achieve this fairness objective is crucial for understanding and addressing potential biases in retrieval systems. We restrict the scope of our investigation of exposure bias to only document collections without categorical meta-data attributes, e.g., gender, polarity etc., as studied in ( Rekabsaz et al. 2021 ) . Previous research has introduced the concept of retrievability as a measure of document accessibility ( Azzopardi and Vinay 2008 ; Wilkie and Azzopardi 2014 ; Wilkie and Azzopardi 2015 ; Wilkie and Azzopardi 2016 ) . This collection-based statistic quantifies a document’s expected (reciprocal) rank of retrieval within a predefined rank cut-off, averaged over a sufficiently large and topically diverse set of queries. While retrievability analysis has traditionally been used to assess document accessibility across different collections ( Wilkie and Azzopardi 2014 ) , more recent studies have applied it to examine exposure bias across various IR models ( Abolghasemi et al. 2023 ) . However, two key considerations must be taken into account when using retrievability for such analyses. First, most existing studies, including those evaluating exposure fairness in IR models ( Abolghasemi et al. 2023 ) , rely on synthetic queries generated via word n -gram sampling. Synthetic queries used in prior research predominantly consist of unigrams or bigrams, whereas real-world user queries tend to be longer on average. Second, non-uniform retrievability values often arise due to variations in relevance priors, where certain high-quality documents are inherently more likely to be relevant than others. In such cases, disparities in retrievability should be accounted for by aggregating locally computed retrievability measures over groups of similar information needs. The contributions of this paper are two-fold: a) addressing the limitation of synthetic queries by conducting retrievability analysis on a large set of real queries (MS MARCO dev set), aligning with real-world information needs; and b) proposing a topic-focused localized retrievability measure, named T-Retrievability (topical retrievability), to conduct more fine-grained analysis of document accessibility of various ranking models. 2. Proposed Measure for Exposure Fairness Background on retrievability As per the originally proposed definition of retrievability ( Azzopardi and Vinay 2008 ) , its value for a document D\in\pazocal{C} (a collection of documents) indicates the likelihood of the document to be retrieved within a specified cut-off rank. As such, the retrievability value of a document depends on the retrieval model used to induce the ranking itself, and a cut-off rank is usually set to a small number, such as 10, indicating the size of a typical search results page ( Kelly and Azzopardi 2015 ) . More formally, r(D,\pazocal{C},\pazocal{Q},\theta,k)=\frac{1}{|\pazocal{Q}|}\sum_{Q\in\pazocal{Q}}\mathbb{I}[\rho(D;Q,\theta)\leq k] , where D is a document of the collection \pazocal{C} , \pazocal{Q} denotes a set of queries, \theta:D\times Q\mapsto\mathbb{R} denotes an IR model that assigns scores to documents thereby inducing a rank for D denoted as \rho(D;Q,\theta) , which is a function of both a particular query Q and the IR model \theta . The parameter k denotes a cut-off rank and \mathbb{I}[X] denotes an indicator function which is 1 if the condition X is true, otherwise 0. Removing dependence on rank cut-off In this paper, we work with a slightly modified version of the retrievability measure, where we remove the dependence of the cut-off rank k . More specifically, instead of computing the likelihood of a document to be retrieved within a top- k , we rather compute the expected rank of a document ( Sinha et al. 2023 ) . In particular, to interpret retrievability as a score, i.e., the higher the better, we work with reciprocals of ranks instead of the ranks themselves. Formally, (1) r(D,\pazocal{C},\pazocal{Q},\theta)=\frac{1}{|\pazocal{Q}|}\sum_{Q\in\pazocal{Q}}\frac{1}{\log(1+\rho(D;Q,\theta))}, As a practical consideration, we restrict the expected rank computation to the top-100 set for each query. The advantage of Equation 1 is that it is a rank-based

Read the complete text