Unbiased Learning to Rank with Query-Level Click Propensity Estimation: Beyond Pointwise Observation and Relevance Conference: Companion Proceedings of the ACM Web Conference 2025; April 28-May 2, 2025; Sydney, NSW, Australia Companion Proceedings of the ACM Web Conference 2025 (WWW Companion ’25),

Lulu Yu, Keping Bi, Jiafeng Guo, Shihao Liu, Dawei Yin, Xueqi Cheng · 2025

Read Unbiased Learning to Rank with Query-Level Click Propensity Estimation: Beyond Pointwise Observation and Relevance Conference: Companion Proceedings of the ACM Web Conference 2025; April 28-May 2, 2025; Sydney, NSW, Australia Companion Proceedings of the ACM Web Conference 2025 (WWW Companion ’25), by Lulu Yu, Keping Bi, Jiafeng Guo, Shihao Liu, Dawei Yin, Xueqi Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11414.txt.

Introduction

Six authors improve unbiased learning to rank for search in this 2025 paper. Ranking models learn from clicks, but clicks are biased: users must see a result before they can click it, and past debiasing methods assume an observed relevant result is always clicked.

In truth, users often stop after one or two clicks even when more good results were seen, a pattern the authors call relevance saturation bias. Their model estimates click propensity for a whole result list, and their DualIPW method corrects both list level and position level bias at once. Theory shows the method learns an unbiased ranker, and tests on a large real dataset beat prior methods clearly.

Catalog record

CREATORLulu Yu, Keping Bi, Jiafeng Guo, Shihao Liu, Dawei Yin, Xueqi Cheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11414

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unbiased Learning to Rank with Query-Level Click Propensity Estimation: Beyond Pointwise Observation and Relevance Conference: Companion Proceedings of the ACM Web Conference 2025; April 28-May 2, 2025; Sydney, NSW, Australia Companion Proceedings of the ACM Web Conference 2025 (WWW Companion ’25), April 28-May 2, 2025, Sydney, NSW, Australia DOI: 10.1145/3701716.3715458 ISBN: 979-8-4007-1331-6/2025/04 CCS: Information systems Learning to rank Lulu Yu, Keping Bi, Jiafeng Guo, Shihao Liu, Dawei Yin, Xueqi Cheng Abstract . Most existing unbiased learning-to-rank (ULTR) approaches are based on the user examination hypothesis, which assumes that users will click a result only if it is both relevant and observed (typically modeled by position). However, in real-world scenarios, users often click only one or two results after examining multiple relevant options, due to limited patience or because their information needs have already been satisfied. Motivated by this, we propose a query-level click propensity model to capture the probability that users will click on different result lists, allowing for non-zero probabilities that users may not click on an observed relevant result. We hypothesize that this propensity increases when more potentially relevant results are present, and refer to this user behavior as relevance saturation bias. Our method introduces a Dual Inverse Propensity Weighting (DualIPW) mechanism—combining query-level and position-level IPW—to address both relevance saturation and position bias. Through theoretical derivation, we prove that DualIPW can learn an unbiased ranking model. Experiments on the real-world Baidu-ULTR dataset demonstrate that our approach significantly outperforms state-of-the-art ULTR baselines. The code and dataset information can be found at https://github.com/Trustworthy-Information-Access/DualIPW . † † cc-license: by 1. Introduction Most existing unbiased learning-to-rank (ULTR) studies focus on mitigating position bias ( Joachims et al. 2017 ; Ai et al. 2018 ; Luo et al. 2024 ; Zhang et al. 2023 ) and are based on position-based click modeling ( Richardson et al. 2007 ) . Position-based modeling (PBM) assumes that users click a result when they examine the result and it is relevant. The observation probability (i.e., click propensity) depends on the position of the result on the search result page (SERP). Grounded on PBM, a major thread of ULTR research is to estimate the propensity of each position on the SERP and conduct inverse propensity weighting (IPW) on the click ( Joachims et al. 2017 ; Ai et al. 2018 ; Luo et al. 2024 ) . Among them, a popular approach is the dual learning algorithm (DLA) ( Ai et al. 2018 ) that adaptively learns relevance and click propensity by IPW and inverse relevance weighting simultaneously. Despite its efficacy on synthetic click data, DLA does not perform as well on the Baidu-ULTR dataset ( Zou et al. 2022 ) , real-world large-scale click data ( Hager et al. 2024 ) . There are several potential reasons for this. First, users may have more complex behaviors PBM cannot capture ( Zou et al. 2022 ) . Second, search engines have strong logging policies that rank high-relevance results to higher positions, leading to inaccurate propensity estimation from confounded relevance and observation ( Hager et al. 2024 ) . Third, severe false negative issues exist in the result list. Most results are relevant but users only click one or two of them ( Yu et al. 2023 ) . Existing work has made efforts to address these challenges. To capture complex user behaviors, Chen et al. 2021 propose the Interactional Observation-Based Model (IOBM), which estimates a position’s click propensity by incorporating clicks from other positions in the result list. To mitigate issues arising from strong logging policies, Zhang et al. 2023 introduce gradient reversal (GradRev) and observation dropout (Drop) to disentangle relevance and observation. Luo et al. 2024 propose an unconfounded propensity estimation (UPE) approach with backdoor adjustment. To address the false negative problem, Wang et al. 2021 argue that non-clicks with higher observation probability are more likely to be true negatives, and therefore assign higher weights to pairs formed by a click and such non-clicks. However, this approach implies that higher-position results are more likely to be true negatives, which contradicts the fact that strong logging policies tend to rank more relevant results higher. Since most of these methods are evaluated on synthetic data, their performance on real-world click data remains unclear. Targeting effective ULTR with real-world click data, we challenge the examination hypothesis in PBM. In our analysis of the Baidu-ULTR dataset, we find that among query sessions with clicks, only about 1.2% involve more than two clicks, approximately two-thirds have a single click, and one-third have exactly two clicks. Given users’ limited patience and the fact that a single relevant result typically satisfies their information needs, we hypothesize that users may have a non-zero probability of not clicking a relevant result, even after examining it. As the number of potentially relevant results in a list increases, users are less likely to leave the session without clicking. This behavior reflects a query-level click propensity influenced by the quality of the ranking list, which we refer to as relevance saturation bias. Accordingly, we propose a Dual Inverse Propensity Weighting mechanism (DualIPW) that incorporates both query-level and position-level click propensity. When the query-level propensity is larger, the ranking list of the query has a smaller inverse weight, which reduces the importance of sessions with potentially more false negatives. Hence, inaccurate contrastive learning between clicks and non-clicks will be mitigated. We theoretically prove that DualIPW can learn an unbiased ranking model. We compare DualIPW with state-of-the-art baselines, including GradRev, Drop, UPE, and IOBM, on the Baidu-ULTR dataset and demonstrate significant improvements over them. 2. Preliminary 2.1. Dual Learning Algorithm (DLA) DLA ( Ai et al. 2018 ) is an effective method for mitigating position bias based on the user examination hypothesis ( Richardson et al. 2007 ) : (1) p(c_{d}=1)=p(r_{d}=1)\cdot p(o_{d}=1)\text{,} where c_{d} , r_{d} , and o_{d} represent whether a document d is clicked, relevant, and observed. DLA treats the estimation of relevance and propensity as a dual problem. Given a query q and its document list \pi_{q} , DLA alternates the learning of an unbiased ranking model f and an unbiased propensity model g by inverse propensity weighting (IPW) and inverse relevance weighting (IRW): (2) l_{IPW}(f,q)=\!\!\!\!\!\!\!\!\sum_{d\in\pi_{q},c_{d}=1}\!\!\!\!\frac{l(f(x_{d}),c_{d})}{p(o_{d}=1)}\text{, }l_{IRW}(g,q)=\!\!\!\!\!\!\!\!\sum_{d\in\pi_{q},c_{d}=1}\!\!\!\!\frac{l(g(k_{d}),c_{d})}{p(r_{d}=1)}\text{,} where k_{d} , x_{d} , and l denote the document at position k , features of d regarding q , and the loss function, respectively. 2.2. Pilot Study 2.2.1. The Relationship Between Click Positions and Relevance Saturation (RS) Bias: A more severe relevance saturation (RS) bias leads to a higher number of false negatives (relevant but non-clicked results), which in turn can degrade model performance due to inaccurate labels. To explore how RS bias varies across query sessions, we divide the Baidu-ULTR query sessions into subsets and evaluate models trained on each subset. Since we have limited query-level information, we use click positions within a query as the partition criterion. For simplicity, we focus on single-click sessions, which account for two-thirds of the sessions with clicks. Fig. 1 shows the average nDCG@10 for models trained on single-click sessions at different positions, using results from 5 random seeds. We focus primarily on the first four positions, as they constitute about 92% of all the single-click sessions. The results show that sessions with

Read the complete text