Off-Policy Evaluation for Ranking Policies under Deterministic Logging Policies
Koichi Tanaka, Kazuki Kawamura, Takanori Muroi, Yusuke Narita, Yuki Sasamoto, Kei Tateno, Takuma Udagawa, Wei-Wei Du, Yuta Saito · 2026
Read Off-Policy Evaluation for Ranking Policies under Deterministic Logging Policies by Koichi Tanaka, Kazuki Kawamura, Takanori Muroi, Yusuke Narita, Yuki Sasamoto, Kei Tateno, Takuma Udagawa, Wei-Wei Du, Yuta Saito (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21485.txt.
Introduction
Tanaka and co-authors tackle off-policy evaluation for ranking systems, where a new ranking policy must be judged from logged data only. Standard estimators need the logging policy to be random enough, and they fail badly when logging is fully deterministic. Their Click-based Inverse Propensity Score instead leans on the randomness in user clicks as the importance weight. Theory covers the bias and variance of the new estimators. In synthetic and real-world tests, bias dropped sharply versus strong baselines under deterministic logging.
Catalog record
| CREATOR | Koichi Tanaka, Kazuki Kawamura, Takanori Muroi, Yusuke Narita, Yuki Sasamoto, Kei Tateno, Takuma Udagawa, Wei-Wei Du, Yuta Saito |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64805 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.21485
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Off-Policy Evaluation for Ranking Policies under Deterministic Logging Policies Koichi Tanaka, Kazuki Kawamura, Takanori Muroi, Yusuke Narita, Yuki Sasamoto, Kei Tateno, Takuma Udagawa, Wei-Wei Du, Yuta Saito Abstract Off-Policy Evaluation (OPE) is an important practical problem in algorithmic ranking systems, where the goal is to estimate the expected performance of a new ranking policy using only offline logged data collected under a different, logging policy. Existing estimators, such as the ranking-wise and position-wise inverse propensity score (IPS) estimators, require the data collection policy to be sufficiently stochastic and suffer from severe bias when the logging policy is fully deterministic. In this paper, we propose novel estimators, Click-based Inverse Propensity Score (CIPS) , exploiting the intrinsic stochasticity of user click behavior to address this challenge. Unlike existing methods that rely on the stochasticity of the logging policy, our approach uses click probability as a new form of importance weighting, enabling low-bias OPE even under deterministic logging policies where existing methods incur substantial bias. We provide theoretical analyses of the bias and variance properties of the proposed estimators and show, through synthetic and real-world experiments, that our estimators achieve significantly lower bias compared to strong baselines, for a range of experimental settings with completely deterministic logging policies. 1 Introduction Many intelligent systems, such as recommender systems, search engines, and e-commerce platforms, present items in the form of rankings. These systems interact with users through the contextual bandit process, where a ranking policy repeatedly observes a context, produces a ranking, and observes rewards ( 7 ) . As ranking policies interact with the environment, they collect logged data that is valuable for Off-Policy Evaluation (OPE). OPE allows researchers to estimate the performance of new policies using only logged data, reducing potential risks and ethical concerns associated with repeated A/B testing ( 15 ; 1 ; 19 ; 18 ) . A typical OPE study often considers the Direct Method (DM), Inverse Propensity Score (IPS), and Doubly Robust (DR) as the baseline estimators ( 15 ) . DM predicts rewards using a learned model, but it can suffer from substantial bias when the model is misspecified. IPS reweights observed rewards by the ratio of the probabilities of taking the observed action, and DR combines both to reduce variance while being unbiased ( 1 ) . However, these typical estimators become impractical in ranking problems with extremely large action spaces, because the probability of observing the exact same ranking is often negligible ( 10 ) . Although many estimators for the ranking setup already exist to address the challenges of large ranking action spaces ( 9 ; 7 ; 10 ) , they require the logging policy to be sufficiently stochastic to produce low-bias estimates. For example, the ranking-wise IPS estimator, which reweights observed rewards by the ratio of ranking-wise probabilities, requires the logging policy to explore all the rankings that a new policy may select. The independent IPS estimator, which uses the ratio of position-wise probabilities, requires the logging policy to select all actions at each position ( 9 ) . Because of this strong dependence on the stochasticity of the logging policy, these existing estimators can only evaluate the value of actions that the logging policy explored sufficiently ( 14 ) . However, large-scale industrial systems often deploy deterministic policies, as the large action spaces make stochastic policies inefficient and risky to use. In such cases, existing methods suffer from severe bias, making them impractical. To address the substantial bias caused by deterministic logging policies, we propose a novel class of estimators, namely Click-based Inverse Propensity Score (CIPS) , which newly exploits the intrinsic stochasticity of user click behavior. In the ranking setting, a policy recommends a ranked list of multiple actions (e.g., videos, products, or job listings) to an incoming user. Therefore, even if the logging policy is fully deterministic, the user may still view and click each action in the presented ranking with some non-zero probability. Building on this property of ranking settings and user behavior, our approach uses marginalized click probabilities under the new and logging policies to define a new form of importance weights, unlike existing methods that depend solely on the stochasticity of the logging policy. This design enables unbiased or low-bias OPE even in situations where deterministic logging policies cause existing methods to produce substantial bias. In our theoretical analysis, we show that CIPS achieves unbiasedness under the stochasticity of user click behavior and a relatively mild independence condition on the reward structure. This is, to the best of our knowledge, the first method to provide low-bias value estimation of new ranking policies under completely deterministic logging. Extensive experiments show that our estimator achieves lower mean squared error than existing methods in fully deterministic logging scenarios. Table 1: Examples of clicks, potential rewards, and observable rewards. position action observable reward CR click C potential reward R k=1 a_{1} 0 0 3000 k=2 a_{2} 1000 1 1000 k=3 a_{3} 0 0 2000 k=4 a_{4} 0 0 500 Table 2: A toy example of importance weight under completely deterministic logging policy. (a) Logging and new policy A_{1} A_{2} A_{3} A_{4} A_{5} A_{6} k=1 a_{1} a_{1} a_{2} a_{2} a_{3} a_{3} k=2 a_{2} a_{3} a_{1} a_{3} a_{1} a_{2} k=3 a_{3} a_{2} a_{3} a_{1} a_{2} a_{1} \pi(A|x_{1}) 0.1 0.3 0.3 0.1 0.0 0.2 \pi_{0}(A|x_{1}) 1.0 0.0 0.0 0.0 0.0 0.0 (b) Importance weights of IPS A_{1} A_{2} A_{3} A_{4} A_{5} A_{6} w(x_{1},A) 0.1 NA NA NA NA NA (c) Importance weights of IIPS w(x_{1},A(k)) a_{1} a_{2} a_{3} k=1 0.4 NA NA k=2 NA 0.3 NA k=3 NA NA 0.4 2 Off-Policy Evaluation for Ranking This section formulates OPE for ranking policies in the contextual bandit process following existing relevant studies ( 9 ; 7 ; 8 ; 10 ) . Let x\in\mathcal{X}\subseteq\mathbb{R}^{d_{x}} be a context vector, such as user demographics, sampled from an underlying distribution p(x) . We use \mathcal{A} to denote a discrete set of unique actions, with a\in\mathcal{A} corresponding to a specific action such as a movie, news article, or product. Let A=(a_{1},a_{2},\cdots,a_{K-1},a_{K}) be a ranking vector, where K denotes the ranking length. We use A(k) to denote the action at the k -th position. The function \pi:\mathcal{X}\rightarrow\Delta(\prod_{K}(\mathcal{A})) is called a ranking policy , where \prod_{K}(\mathcal{A}) represents the set of K -permutations of \mathcal{A} , i.e., the ranking space, and \Delta(\cdot) is the probability simplex. To closely reflect realistic ranking scenarios, we introduce two types of rewards: clicks and potential rewards . The latter refers to downstream outcomes (e.g., product purchase, movie play) that may occur after a click, thus modeling the two-stage decision process of users. Let first C denote a click vector sampled from an unknown distribution conditional on the presented ranking, p(C|x,A) , where C(k) is a binary signal indicating whether a click occurs at position k . Let R then denote a potential reward vector drawn from p(R|x,A) . We refer to R as “potential” because it is only observable through CR , meaning that we observe R(k) only if C(k)=1 , i.e., after a click. This structure is based on real-world ranking systems, such as product searches, video recommendations, and many other large-scale e-commerce or content platforms, where user interactions naturally follow two or more