PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data

Aishwarya Mandyam*, Jason Meng*, Ge Gao, Jiankai Sun, Mac Schwager, Barbara E. Engelhardt, Emma Brunskill · 2025

Read PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data by Aishwarya Mandyam*, Jason Meng*, Ge Gao, Jiankai Sun, Mac Schwager, Barbara E. Engelhardt, Emma Brunskill (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20068.txt.

Introduction

Aishwarya Mandyam led a team judging new decision policies with honest error bars in a 2025 paper. Before deploying a learned policy, practitioners estimate its value from logged data, and extra synthetic data can sharpen the estimate but may be biased. Their two methods build valid confidence intervals even when such auxiliary data is used. One conditions on the starting state using a new prediction method for step-by-step processes.

The other targets average performance, borrowing double-robust ideas. Across simulators for inventory, robotics, and health care, plus a real hospital dataset, their intervals covered the true values where earlier methods failed.

Catalog record

CREATORAishwarya Mandyam*, Jason Meng*, Ge Gao, Jiankai Sun, Mac Schwager, Barbara E. Engelhardt, Emma Brunskill
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-20068

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data Aishwarya Mandyam*, Jason Meng*, Ge Gao, Jiankai Sun, Mac Schwager, Barbara E. Engelhardt, Emma Brunskill Abstract Off-policy evaluation (OPE) methods aim to estimate the value of a new reinforcement learning (RL) policy prior to deployment. Recent advances have shown that leveraging auxiliary datasets, such as those synthesized by generative models, can improve the accuracy of these value estimates. Unfortunately, such auxiliary datasets may also be biased, and existing methods for using data augmentation for OPE in RL lack principled uncertainty quantification. In high stakes settings like healthcare, reliable uncertainty estimates are important for comparing policy value estimates. In this work, we propose two approaches to construct valid confidence intervals for OPE when using data augmentation. The first provides a confidence interval over the policy performance conditioned on a particular initial state V π ​ ( s 0 ) V^{\pi}(s_{0}) – such intervals are particularly important for human-centered applications. To do so we introduce a new conformal prediction method for high dimensional state MDPs. Second, we consider the more common task of estimating the average policy performance over many initial states; to do so we draw on ideas from doubly robust estimation and prediction powered inference. Across simulators spanning robotics, healthcare and inventory management, and a real healthcare dataset from MIMIC-IV, we find that our methods can use augmented data and still consistently produce intervals that cover the ground truth values, unlike previously proposed methods. 1 Introduction Off-policy evaluation (OPE) is used to estimate the value of a new reinforcement learning (RL) policy prior to deployment using historical dataset from a different behavior policy. This strategy is especially important in high-stakes domains where directly deploying new policies can be costly or even harmful to participants. OPE has been used in a variety of domains including healthcare ( Gottesman et al. 2020 ) , education ( Mandel et al. 2014 ) , and robotics ( Fu et al. 2020 ) . Several methods have been proposed to perform OPE include importance sampling (IS) ( Precup et al. 2000 ) , direct method (DM) ( Li et al. 2010 ; Beygelzimer and Langford 2009 ; van Seijen et al. 2009 ; Harutyunyan et al. 2016 ; Le et al. 2019 ; Voloshin et al. 2021 ) , and doubly robust (DR) ( Farajtabar et al. 2018 ; Dudik et al. 2011 ; Jiang and Li 2016 ) approaches. However, standard OPE methods generally suffer from dataset coverage issues. For example, if the target policy takes actions that are underrepresented in the behavior dataset, OPE estimates can be erroneous. To address this, several recent works have proposed using augmented data to improve the coverage of the behavior dataset and subsequently the accuracy of OPE methods ( Tang and Wiens 2023 ; Gao et al. 2024 ; Mandyam et al. 2024 ) . Despite these advances, OPE estimates may still vary across different estimators ( Nie et al. 2024 ) , making it difficult for practitioners to obtain robust estimates of the policy value. One strategy to enable more meaningful comparisons between OPE methods is to provide confidence intervals for individual OPE estimates. Confidence intervals communicate a degree of uncertainty, which makes it possible to assess whether differences in OPE estimates for a given target policy are statistically significant. Prior approaches have derived asymptotic ( Thomas et al. 2015a ; Thomas et al. 2015b ) and sample-dependent confidence intervals ( Taufiq et al. 2022 ; Foffano et al. 2023 ) . However, no prior works provide valid confidence intervals for a setting in which uses both offline trajectories and a set of augmented trajectories. In this work, we introduce two strategies to generate valid confidence intervals for OPE while leveraging augmented datasets . First, in several high-stakes domains it is useful to have confidence intervals for policy values when conditioned on an initial state. For example, in a clinical domain, it is important to have confidence intervals for the value of a new treatment policy for all patients that are at similar stage of a disease. We propose our first estimator, CP-Gen, to address this need by calculating conformal prediction intervals for the value of a policy conditioned on a starting state. However, approximating the full value of the target policy via a union bound over state-condition policy values can lead to impractically wide confidence intervals. We propose a second estimator, DR-PPI, for estimating unconditional policy values. DR-PPI leverages doubly robust estimation and prediction-powered inference ( Angelopoulos et al. 2023 ) to generate valid confidence intervals for the value of the target policy by correcting for erroneous generated trajectories. Our results demonstrate that both proposed approaches are able to generate valid confidence intervals that are empirically superior to baseline approaches. Our work makes the following contributions: 1. We introduce two methods to calculate confidence intervals for OPE when using a dataset that contains augmented data ( Section 3 ). 2. We prove that both methods yield valid confidence intervals that achieve the desired coverage probability for the true policy value either asymptotically or within a margin of error for finite sample sizes ( Section 4 ). 3. We empirically evaluate the estimators in four domains including inventory control, sepsis, robotic control, and MIMIC-IV, showing that our methods leverage augmented data to often produce tighter confidence intervals with the correct coverage. In contrast, prior approaches either do not use augmented data and have valid but often wide intervals, or do use augmented data but fail to provide valid coverage ( Section 5 ). 2 Background 2.1 Problem setting We consider a decision-making setting defined by the Markov decision process (MDP) \mathcal{M}=(\mathcal{S},\mathcal{A},P,R,d_{0},\gamma,H) . \mathcal{S},\mathcal{A} denote the possibly infinite state and action spaces respectively. P:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) represents the transition dynamics, R:\mathcal{S}\times\mathcal{A}\to\Delta(\mathbb{R}) is the reward function, and d_{0}\in\Delta(\mathcal{S}) is the initial state distribution. \gamma is the discount factor and H is the fixed horizon. A trajectory is defined as \tau:\{s_{t},a_{t},r_{t}\}_{t=1}^{H} where s_{t},a_{t},r_{t} are the corresponding state, action, and instantaneous reward observed at timestep t . The return of the trajectory \tau is J(\tau)=\sum_{t=1}^{H}\gamma^{t-1}r_{t} where \pi is the policy used to generate the trajectory. The value of the policy V^{\pi}=\mathbb{E}_{\tau\sim\pi}[J(\tau)] is calculated as an expectation over the possible trajectories that could arise from the policy \pi . 2.2 Off-policy evaluation The goal of off-policy evaluation (OPE) is to estimate the value of a target policy \pi_{e} given a dataset of behavior trajectories D_{\pi_{b}} that arise from a distinct behavior policy \pi_{b} . There are several standard approaches for OPE, including IS ( Precup et al. 2000 ) , direct method (DM) ( Li et al. 2010 ; Beygelzimer and Langford 2009 ; van Seijen et al. 2009 ; Harutyunyan et al. 2016 ; Le et al. 2019 ; Voloshin et al. 2021 ) , and doubly robust approaches (DR) ( Farajtabar et al. 2018 ; Dudik et al. 2011 ; Jiang and Li 2016 ) . IS-based estimators re-weigh each trajectory in the behavior dataset using an inverse propensity score (IPS) \rho(\tau)=\prod_{t=1}^{H}\frac{\pi_{e}(s_{t},a_{t})}{\pi_{b}(s_{t},a_{t})} . DM estimators learn a reward model using the behavior trajectories and use either Monte Carlo sampling or fit a reward model to calculate the value of the target policy.

Read the complete text