Towards Robust Offline Evaluation: A Causal and Information Theoretic Framework for Debiasing Ranking Systems
Seyedeh Baharan Khatami, Sayan Chakraborty, Ruomeng Xu, Babak Salimi · 2025
Read Towards Robust Offline Evaluation: A Causal and Information Theoretic Framework for Debiasing Ranking Systems by Seyedeh Baharan Khatami, Sayan Chakraborty, Ruomeng Xu, Babak Salimi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03997.txt.
Introduction
Seyedeh Baharan Khatami and co-workers rebuilt offline ranking review on causal grounds in a 2025 paper. Search and pick teams judge new rankers on logged clicks before risking live tests, but logs are biased: popular, top-placed. Conforming items get clicked for reasons unrelated to quality, and the missing clicks are not missing at random. Their setup reweights the logged data to mimic random exposure, converting the bias form into a missing-at-random problem that standard metrics can handle honestly.
Black-box tuning tunes the reweighting, guided by neural estimates of information-theoretic quantities. The causal formulation is system-agnostic, applying to any ranker whose outputs can be rescored. Tests validate fairer, more generalizable offline checks.
Catalog record
| CREATOR | Seyedeh Baharan Khatami, Sayan Chakraborty, Ruomeng Xu, Babak Salimi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-03997
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Robust Offline Evaluation: A Causal and Information Theoretic Framework for Debiasing Ranking Systems Seyedeh Baharan Khatami, Sayan Chakraborty, Ruomeng Xu, Babak Salimi Abstract . Evaluating retrieval-ranking systems is crucial for developing high-performing models. While online A/B testing is the gold standard, its high cost and risks to user experience require effective offline methods. However, relying on historical interaction data introduces biases—such as selection, exposure, conformity, and position biases—that distort evaluation metrics, driven by the Missing-Not-At-Random (MNAR) nature of user interactions and favoring popular or frequently exposed items over true user preferences. We propose a novel framework for robust offline evaluation of retrieval-ranking systems, transforming MNAR data into Missing-At-Random (MAR) through reweighting combined with black-box optimization, guided by neural estimation of information-theoretic metrics. Our contributions include (1) a causal formulation for addressing offline evaluation biases, (2) a system-agnostic debiasing framework, and (3) empirical validation of its effectiveness. This framework enables more accurate, fair, and generalizable evaluations, enhancing model assessment before deployment. DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 1. INTRODUCTION Recommender systems (RS) help users navigate information overload by providing personalized suggestions, benefiting both users and providers. Companies refine RS models through iterative improvements, underscoring the need for robust evaluation. While A/B testing remains the gold standard, it is costly, slow, and risks degrading user experience. Offline evaluation using historical data offers a more efficient alternative but suffers from biases due to its observational nature, including selection ( Yang et al., 2018 ; Chen et al., 2023 ) , exposure ( Chen et al., 2023 ) , popularity ( Chen et al., 2020 ; Krishnan et al., 2018 ; Chen et al., 2023 ) , and position biases ( Collins et al., 2018 ; Chen et al., 2023 ) . Implicit feedback only captures positive interactions, leading to Missing-Not-At-Random (MNAR) data, where engagement skews toward frequently surfaced items, making user preferences harder to infer. Ignoring these biases in offline evaluation can reinforce the long-tail effect, propagate biases from prior models, and misalign results with A/B tests, increasing the risk of poor model selection ( Marlin and Zemel, 2009 ; Schnabel et al., 2016 ; Steck, 2010 ) . Existing debiasing approaches have limitations: some assume Missing-At-Random (MAR) interactions, which is unrealistic ( He and McAuley, 2016 ; Hsieh et al., 2017 ; Lim et al., 2015 ) ; others address single biases like position ( Oosterhuis and de Rijke, 2021 ) or popularity bias ( Krishnan et al., 2018 ) but lack generalizability. Many methods require clean, unbiased data, which is often impractical ( Hu et al., 2008 ; Lee et al., 2021 ) . To bridge these gaps, our framework allows for specifying a bias attribute—such as exposure, popularity, or temporal bias—and debiases the evaluation data accordingly by transforming MNAR data into Missing-At-Random (MAR) data, ensuring a more reliable assessment of recommendation quality. Unlike methods that require access to a clean, bias-free dataset, our approach operates effectively without such data but can leverage it when available for further debiasing. Additionally, our framework is generalizable, system-agnostic, and adaptable across diverse ranking systems, making it suitable for a wide range of recommendation scenarios. By leveraging a causal formulation and an information-theoretic perspective, our method corrects for biases inherent in evaluation data, leading to offline metrics that more accurately reflect true user preferences and system performance. Our contributions are threefold: • Causal Problem Formulation: A theoretical foundation leveraging information-theoretic principles to mitigate biases in offline evaluations. • Mutual Information-Based Framework: A general system-agnostic approach that minimizes dependence between observed interactions and a given biasing factor. • Empirical Validation: Evaluation on both public and company internal offsite real-time recommendation system data, demonstrating effectiveness as a minimal-effort debiasing prerequisite for ranking systems. 2. RELATED WORK Evaluating retrieval-ranking systems while mitigating biases in historical interaction data has been extensively studied. Inverse Propensity Scoring (IPS) ( Yang et al., 2018 ; Marlin and Zemel, 2009 ; Lee et al., 2021 ; Hu et al., 2008 ) is a widely used method for correcting selection bias but suffers from high variance. To improve stability, doubly robust estimators ( Wang et al., 2019 ) . Adversarial learning techniques ( Xu et al., 2020 ; Wu et al., 2021 ) aim to identify and mitigate bias through adversarial training, while causal inference methods ( Wang et al., 2021 ; Zhang et al., 2022 ; Wang et al., 2023 ; Zhang et al., 2021 ) address confounding factors by leveraging do-calculus and backdoor adjustment. Recent works explore invariant learning to disentangle user preferences from bias ( Zhang et al., 2021 ; Zheng et al., 2022 ) , but these approaches struggle with accuracy and stability. Knowledge distillation methods ( Bai et al., 2024 ) fuse invariant and variant information to improve generalization. Adaptive model selection strategies ( Wang et al., 2025 ) dynamically switch between biased and debiased models depending on test conditions. Despite these advancements, most methods require unbiased data supervision, which is costly and challenging in real-world settings. Our proposed framework addresses this limitation by introducing a resampling-based evaluation method using conditional mutual information to systematically mitigate the entanglement between bias and user preferences, offering a scalable and generalizable solution. 3. PROBLEM FORMULATION X^{nr} X^{r} E C Figure 1 . Causal DAG of Random Variables: X^{r} represents relevant user-specific item features, X^{nr} represents non-relevant biasing factor feature influencing exposure, E is exposure, and C is click. This section introduces the notation and theoretical formulation of the problem. Our dataset is defined as \mathcal{D}=\{(U_{i},I_{i},X_{i},E_{i},C_{i})\;|\;i=1,2,\dots,N\} , where U_{i} and I_{i} are user and item IDs for the i -th interaction. The feature vector X_{i}=(X^{r}_{i},X^{nr}_{i}) consists of user-specific item features relevant to the user’s preferences X^{r}_{i} and exposure-related features X^{nr}_{i} , which may not reflect user preferences. E_{i} denotes whether I_{i} was exposed to U_{i} , and C_{i} is the observed interaction (e.g., a click). The goal is to reduce the impact of X^{nr} on exposure, as it biases interactions ( C ). We aim to debias with respect to X^{nr} and illustrate this with four examples, though the approach extends to other exposure-related biases as well. • Popularity Bias: Let X^{nr} represent item popularity scores. High popularity may skew exposure, overexposing few items and underexposing others, limiting visibility. • Sensitive Attribute Bias: Let X^{nr} represent a sensitive attribute (e.g., gender or race) that biases exposure, leading to unequal visibility of items associated with certain groups. • Staleness Bias: Let X^{nr} denote the timestamp an item was introduced. Systems may favor items added earlier, giving them higher exposure probabilities. This staleness bias is a key case study in our experiments, discussed in detail in the evaluation section. The definition of X^{nr} varies by application and is typically determined using domain knowledge from system designers. We observe samples from the joint distribution P(X,C) , denoted as the observed distribution P_{\mathbf{o}}(X,C) . If X^{nr} can be partitioned into m groups \{X_{1}^{nr},X_{2}^{nr},\dots,X_{m}^{nr}\} , an ideal ranking system should satisfy P_{\mathbf{o}}(X^{r},C\mid X_{k}^{nr})=P_{\mathbf{o}}(X^{r},C\mid X_{l}^{nr}) for all k,l\in\{1,2,\dots,m\} . However, this equality often fails due to various systematic factors or design choices in ranking systems. In general, a biased ranking system