Doubly-Robust Estimation of Counterfactual Policy Mean Embeddings

Houssam Zenati, Bariscan Bozkurt, Arthur Gretton · 2025

Read Doubly-Robust Estimation of Counterfactual Policy Mean Embeddings by Houssam Zenati, Bariscan Bozkurt, Arthur Gretton (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02793.txt.

Introduction

Houssam Zenati, Bariscan Bozkurt, and Arthur Gretton write a 2025 statistics paper on judging what would happen under a policy that was never run. In fields from health care to advertising, teams need the full spread of outcomes under such unseen policies, not just an average. Their framework, called CPME, stores that whole spread as an object in a rich math space.

They give two ways to estimate the object from logged data. The stronger one is doubly robust, meaning it corrects for bias in two places at once and so converges faster. They also build a test from it that follows a known bell-shaped limit, which makes tests and confidence ranges easy to compute. The frame even allows drawing fresh samples from the unseen policy's outcome spread.

Catalog record

CREATORHoussam Zenati, Bariscan Bozkurt, Arthur Gretton
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE147 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02793

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Doubly-Robust Estimation of Counterfactual Policy Mean Embeddings Houssam Zenati, Bariscan Bozkurt, Arthur Gretton Abstract Estimating the distribution of outcomes under counterfactual policies is critical for decision-making in domains such as recommendation, advertising, and healthcare. We analyze a novel framework—Counterfactual Policy Mean Embedding (CPME)—that represents the entire counterfactual outcome distribution in a reproducing kernel Hilbert space (RKHS), enabling flexible and nonparametric distributional off-policy evaluation. We introduce both a plug-in estimator and a doubly robust estimator; the latter enjoys improved uniform convergence rates by correcting for bias in both the outcome embedding and propensity models. Building on this, we develop a doubly robust kernel test statistic for hypothesis testing, which achieves asymptotic normality and thus enables computationally efficient testing and straightforward construction of confidence intervals. Our framework also supports sampling from the counterfactual distribution. Numerical simulations illustrate the practical benefits of CPME over existing methods. Keywords— off-policy evaluation, causal inference, doubly robust, kernel mean embeddings, kernel two sample tests 1 Introduction Effective decision-making requires anticipating the outcomes of actions driven by given policies [ 43 ] . This is especially critical when decisions must be made from historical data, where experimentation is limited or infeasible [ 80 ] . For instance, doctors weigh drug effects before prescribing [ 37 ] , and businesses predict revenue impact from ads [ 10 ] . Off-Policy Evaluation (OPE) addresses this challenge by estimating the effect of a target policy using data sampled under a different logging policy . Each logged record includes covariates (e.g., user or patient data), an action (e.g., recommendation or treatment), and the resulting outcome (e.g., engagement or health status). The goal is to evaluate the expected outcome under the target policy , which involves inferring counterfactual outcomes—what would have happened under the alternative target policy. Although many works have focused on estimating the mean of outcome distributions, for example, with the policy expected risk (payoff) [ 45 ] or the average treatment effects [ 36 ] - and their variants thereof - seminal works have considered inference on counterfactual distributions of outcomes [ 65 ] instead. The developing field of distributional reinforcement learning (RL) [ 4 ] and distributional OPE [ 13 , 89 ] provides insights on distribution-driven decision making, which goes beyond expected policy risks. Indeed, reasoning on such distributions allows using alternative risk measure such as conditional value-at-risk (CVaR) [ 63 ] , higher moments or quantiles of the distribution [ 68 , 18 ] . However, most existing approaches leverage cumulative density functions (CDF) [ 15 , 34 ] which are not suited for inference on more complex and structured outcomes. Conversely, counterfactual mean embeddings (CME) [ 53 ] represent the outcome distributions as elements in a reproducing kernel Hilbert space (RKHS) [ 6 , 28 ] and allow inference for distributions over complex outcomes such as images, sequences, and graphs [ 25 ] . Such embeddings leverage kernel mean embeddings [ 73 ] , a framework for representing a distribution maintaining all of its information for sufficiently rich kernels [ 38 , 76 ] . This framework allows to quantify distributional treatment effects [ 53 ] , perform hypothesis testing [ 30 ] over even sample [ 88 , 14 ] from the counterfactual outcome distribution. Recent works have employed counterfactual mean embeddings for causal inference in the context of distributional treatment effects [ 57 , 51 , 21 ] , however these approaches have not been applied to OPE. Developing analogous distributional embeddings for counterfactual outcomes under target policies could enable a range of new applications, including principled evaluation, hypothesis testing, and efficient sampling from complex outcome distributions. Our estimates will employ doubly robust methods, which have become a central tool in causal inference due to the desirable property of consistency if either the outcome model or the propensity model is correctly specified [ 62 , 3 ] . DR estimators have since been studied under various functional estimation tasks, including treatment effects [ 39 ] and policy evaluation [ 20 ] . These estimators originally leverage efficient influence functions [ 7 , 83 , 23 ] and sample-splitting techniques [ 41 , 66 ] to achieve bias reduction and enable valid inference in high-dimensional and nonparametric settings [ 85 , 16 ] . Recently, doubly robust tests have been introduced for kernel treatment effects [ 51 , 21 ] . Moreover, an extension of semiparametric efficiency theory and efficient influence functions has been proposed for differentiable Hilbert-space-valued parameters [ 49 ] . Leveraging these efficient influence functions to build doubly robust estimators of counterfactual mean embeddings would therefore enable more theoretically grounded distributional OPE. In this work, we propose a novel approach to distributional OPE that embeds the counterfactual outcome distribution, which procedure we term as Counterfactual Policy Mean Embedding (CPME). Our contributions are as follows: i) First, we define and formalize the CPME in the distributional OPE problem. We proposing a plug-in estimator, and analyze its consistency with a convergence rate of up to O_{p}(n^{-1/4}) under certain regularity assumptions involving kernels and underlying distributions. ii) We then leverage the efficient influence function of the CPME to propose a doubly robust estimator, and establish its improved consistency rate of up to O_{p}(n^{-1/2}) under the same assumptions. iii) Consequently, we propose an efficient doubly robust and asymptotically normal statistic which allows a computationally efficient kernel test. iv) We demonstrate that our estimators enable sampling from the outcome distribution. v) Finally, we provide numerical simulations to support our claims in a range of scenarios. The remainder of the paper is structured as follows. Section 2 introduces the problem setting and the definition of the CPME. Section 3 describes our nonparametric plug-in estimator along with its consistency guarantees. Section 4 presents an estimator based on the efficient influence function, offering improved convergence properties. In Section 5 , we showcase two applications of our framework: hypothesis testing and sampling. Section 6 reports our numerical experiments, and Section 7 concludes. 2 Counterfactual policy mean embeddings We begin by formalizing the counterfactual policy mean embedding (CPME) framework, which provides a kernel-based foundation for distributional OPE. 2.1 Distributional off-policy evaluation setting We are given an observational dataset generated from interactions between a decision-making system and units with covariates x_{i} . For each instance i\in\{1,\dots,n\} , a context x_{i} was drawn i.i.d. from an unknown distribution P_{X} , i.e., x_{i}\sim P_{X} . Given x_{i} , an action a_{i} was sampled from a logging policy \pi_{0}\in\Pi , such that a_{i}\sim\pi_{0}(\cdot\mid x_{i}) . Following the potential outcomes framework [ 36 ] , we denote the set of potential outcomes by \{Y(a)\}_{a\in\mathcal{A}} , and observe the realized outcome y_{i}=Y(a_{i})\sim P_{Y\mid X,A=x_{i},a_{i}} . The data-generating process is therefore characterized by the joint distribution P_{0}=P_{Y\mid X,A}\times\pi_{0}\times P_{X} . The dataset consists of n i.i.d. logged observations \{(x_{i},a_{i},y_{i})\}_{i=1}^{n}\sim P_{0} . The action space \mathcal{A} may be either finite or continuous. For notational purposes, we will also abbreviate the joint distributional P_{\pi}=P_{Y\mid X,A}\times\pi\times P_{X} . Given only this logged data from P_{0} , the goal of distributional off-policy evaluation is to estimate \nu(\pi) , the distribution of outcomes induced by a target policy \pi belonging to the policy set \Pi : \nu(\pi)=\mathbb{E}_{\pi\times P_{X}}\left[P_{Y\mid X,A}(Y(a))\right]. (1) \nu(\pi) represents the marginal

Read the complete text