Adaptive Off-Policy Inference for M-Estimators Under Model Misspecification

James Leiner, Robin Dunn, Aaditya Ramdas · 2025

Read Adaptive Off-Policy Inference for M-Estimators Under Model Misspecification by James Leiner, Robin Dunn, Aaditya Ramdas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14218.txt.

Introduction

James Leiner, Robin Dunn and Aaditya Ramdas address inference from data gathered step by step in 2025, as in bandit systems that favour options that have paid off so far. Standard methods can fail there because early choices skew later data. A second snag is a working model that does not match reality.

Leiner and colleagues define the target cleanly as the best fit under a fixed evaluation policy, even when rewards shift over time. Their method then gives valid inference for a broad class of estimators, called M-estimators, fitted on such adaptive data with a possibly wrong model. A flexible way to steady the variance sits at the core. The result extends trustworthy confidence statements to a setting where both adaptivity and model error are present.

Catalog record

CREATORJames Leiner, Robin Dunn, Aaditya Ramdas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-14218

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Off-Policy Inference for M-Estimators Under Model Misspecification James Leiner, Robin Dunn, Aaditya Ramdas Abstract When data are collected adaptively, such as in bandit algorithms, classical statistical approaches such as ordinary least squares and M M -estimation will often fail to achieve asymptotic normality. Although recent lines of work have modified the classical approaches to ensure valid inference on adaptively collected data, most of these works assume that the model is correctly specified. We propose a method that provides valid inference for M-estimators that use adaptively collected bandit data with a (possibly) misspecified working model. A key ingredient in our approach is the use of flexible machine learning approaches to stabilize the variance induced by adaptive data collection. A major novelty is that our procedure enables the construction of valid confidence sets even in settings where treatment policies are unstable and non-converging, such as when there is no unique optimal arm and standard bandit algorithms are used. Empirical results on semi-synthetic datasets constructed from the Osteoarthritis Initiative demonstrate that the method maintains type I error control, while existing methods for inference in adaptive settings do not cover in the misspecified case. 1 Department of Statistics and Data Science, Carnegie Mellon University 2 Novartis Pharmaceuticals Corporation, Advanced Methodology and Data Science 3 Machine Learning Department, Carnegie Mellon University 1 Introduction Adaptive data collection has become a common practice in modern data analysis due to the rise of reinforcement learning and sequential data collections in contexts such as personalized healthcare ( Yom-Tov et al. 2017 ) , web recommendations ( Afsar et al. 2022 ) , and clinical trials ( Zhao et al. 2009 ; Liu et al. 2020 ) . In contrast to classical settings, where i.i.d. data are observed, adaptively collected data allows an analyst to interact with the decision-making algorithm in various ways, leading to sequentially dependent data. A canonical example is the (contextual) bandit problem where an analyst is allowed to make a choice of treatment (or “arm”) at each time step and then observes a reward. Bandit algorithms attempt to maximize the total expected reward over time. These algorithms tend to result in data collection policies that explore treatment options in early rounds, and they tend to shift to a greedier strategy that chooses the treatment with the highest estimated reward in the later rounds. When inference is conducted on data collected by bandit algorithms, the greedy search process can result in unstable variances ( Deshpande et al. 2018 ; Zhang et al. 2020 ) and biased estimates of the average reward in each treatment arm ( Shin et al. 2019 ) that make naive approaches to inference invalid. There is a rich literature to adjust to the difficulties that arise for inference on adaptively collected data. Typical approaches focus on limiting the variance in each data collection using techniques such as propensity score truncation ( Cook et al. 2024 ) or post-hoc reweighting of the data when constructing an estimator ( Bibaut et al. 2021 ; Syrgkanis and Zhan 2024 ) . These approaches allow for assumption-lean inference in semiparametric settings, but they limit the study to relatively simple functionals such as the average reward for a fixed treatment or contrasts between average rewards for fixed arms. An alternative set of approaches assumes a linear model linking treatment ( X_{t} ) and rewards ( Y_{t} ), Y_{t}=\theta^{T}X_{t}+\epsilon_{t}, where \epsilon_{t} is i.i.d. random noise. It is striking that even in this simple setting, estimators such as ordinary least squares fail to achieve asymptotic normality when X_{t} depends on previous rounds \{X_{i},Y_{i}\}_{i=1}^{t-1} . A classical result ( Lai and Wei 1982 ) demonstrates that the OLS estimate for \theta is asymptotically normal when the covariance matrix (\sum_{t=1}^{T}X_{t}X_{t}^{T})^{-1} converges to a fixed deterministic sequence. More recent literature ( Zhang et al. 2020 ; Deshpande et al. 2018 ; Khamaru et al. 2021 ) makes the point that this assumption typically does not apply for common bandit algorithms when the margin (that is, the difference in mean rewards between arms) is zero. Zhang et al. 2020 proposes a solution through batching, by fixing the sampling rule for each batch and letting the number of observations within each batch go to infinity. This means that even if the sampling rule does not concentrate across time , it is fixed within each batch, allowing the empirical covariance matrix within each batch to concentrate. An alternative set of approaches ( Deshpande et al. 2018 ; Khamaru et al. 2021 ) uses an online debiasing approach, where the finite sample bias is controlled through regularization. Alternative approaches use concentration results to construct confidence regions that are finite-sample valid uniformly over time and using estimation equations ( Ying et al. 2023 ) . We note that across all of these approaches, correct specification of the working model is crucial to ensuring valid inference. Correct model specification is an unlikely assumption to hold in practical settings. In the non-adaptive setting, it is common to instead conduct inference on a projected solution, \theta^{\star}:=\argmax_{\theta}\mathbb{E}\left[\left(Y_{t}-\theta^{T}X_{t}\right)^{2}\right]. It is well known that under mild regularity conditions, the ordinary least squares estimate with sandwich estimators of the variance will cover the projected solution ( White 1982 ) , but no analogous result exists in the adaptive setting. Moving beyond the linear case, Zhang et al. 2021 considers statistical inference on M -estimators in the contextual bandit problem, allowing for more complex models to describe the reward structure, such as generalized linear models. However, this work again requires that the conditional mean of the model is correctly specified in the working model. A related paper ( Zhang et al. 2023 ) performs inference for Z -estimators and allows for a misspecified model, but only under a finite amount of adaptivity. In this framework, n individuals are tracked over T time periods, with T fixed and n growing to \infty . The advantage of this approach is that it allows for non-stationary reward and context distributions over time, but it relies on the number of individuals enrolled in the trial to tend to \infty and does not allow adaptive decisions to be made separately for each individual in the trial. 1.1 Our Contributions In this work, we propose a methodology for inferring parameters in M -estimation problems under a potentially misspecified working model and non-finite levels of adaptivity. The major result, Theorem 1 , provides a Central Limit Theorem that enables the construction of confidence intervals when the variance (which varies over time and may not converge) of the score function can be estimated consistently. The problem of estimating the variance of the score function is nontrivial because using naive empirical estimates when applying Theorem 1 is only valid when the variance is a fixed deterministic quantity. In many adaptive settings, such as bandit problems where expected rewards between arms are comparable, action selection probabilities will be non-convergent random quantities, leading to unstable variances ( Zhang et al. 2020 ) . To solve this problem, we leverage flexible machine learning methods to construct valid time-varying plug-in estimates for the variances at each time step. In a work that is concurrent to ours (the authors having released their work one week prior to the first

Read the complete text