Design Stability in Adaptive Experiments: Implications for Treatment Effect Estimation

Saikat Sengupta [email protected], Koulik Khamaru [email protected], Suvrojit Ghosh [email protected], Tirthankar Dasgupta [email protected], [0.3cm] Indian Statistical Institute, Kolkata · 2025

Read Design Stability in Adaptive Experiments: Implications for Treatment Effect Estimation by Saikat Sengupta [email protected], Koulik Khamaru [email protected], Suvrojit Ghosh [email protected], Tirthankar Dasgupta [email protected], [0.3cm] Indian Statistical Institute, Kolkata (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22351.txt.

Introduction

Saikat Sengupta, Koulik Khamaru, Suvrojit Ghosh, and Tirthankar Dasgupta shore up the math behind experiments that adapt as they go. In such designs, each new subject's chance of treatment can depend on all past assignments and outcomes. Classic trial formulas do not cover that freedom. The error bars on the average treatment effect stay unpriced.

Their key concept is design stability. As the sample grows, the assignment chances settle down. Failing that, averages of the inverse propensity scores and their complements settle instead. Under that condition, central limit theorems hold for two estimators. One is the inverse propensity weighted estimator. The other is its augmented cousin. Explicit formulas give the long-run spread of both. Adaptive trials gain a firm basis for confidence intervals.

Catalog record

CREATORSaikat Sengupta [email protected], Koulik Khamaru [email protected], Suvrojit Ghosh [email protected], Tirthankar Dasgupta [email protected], [0.3cm] Indian Statistical Institute, Kolkata
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE107 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22351

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Design Stability in Adaptive Experiments: Implications for Treatment Effect Estimation Saikat Sengupta [email protected], Koulik Khamaru [email protected], Suvrojit Ghosh [email protected], Tirthankar Dasgupta [email protected], [0.3cm] Indian Statistical Institute, Kolkata Abstract We study the problem of estimating the average treatment effect ( ATE ) under sequentially adaptive treatment assignment mechanisms. In contrast to classical completely randomized designs, we consider a setting in which the probability of assigning treatment to each experimental unit may depend on prior assignments and observed outcomes. Within the potential outcomes framework [ 1 ] , we propose and analyze two natural estimators for the ATE : the inverse propensity weighted (IPW) estimator and an augmented IPW (AIPW) estimator. The cornerstone of our analysis is the concept of design stability , which requires that as the number of units grows, either the assignment probabilities converge, or sample averages of the inverse propensity scores and of the inverse complement propensity scores converge in probability to fixed, non-random limits. Our main results establish central limit theorems for both the IPW and AIPW estimators under design stability and provide explicit expressions for their asymptotic variances. We further propose estimators for these variances, enabling the construction of asymptotically valid confidence intervals. Finally, we illustrate our theoretical results in the context of Wei’s adaptive coin design [ 2 ] and Efron’s biased coin design [ 3 ] , highlighting the applicability of the proposed methods to sequential experimentation with adaptive randomization. Keywords: Average treatment effect; Sequential treatment assignment; Design stability; Adaptive designs; IPW estimator; AIPW estimator. 1 Introduction Estimating the average treatment effect is a foundational problem in causal inference, especially when evaluating interventions in fields such as healthcare [ 4 ] , education [ 5 ] , public policy [ 6 ] , development economics [ 7 ] , and digital experimentation [ 8 ] . Traditional methods often assume simple randomized designs with independent and identically distributed (i.i.d.) units and fixed treatment assignment probabilities. However, many real-world experiments depart from this idealized setting: units often arrive sequentially, treatment assignments may adapt over time based on previous allocations, observed outcomes, or covariate information, and the study population is finite. Such sequential finite-population setups are common in adaptive clinical trials, online A/B testing, and policy evaluations. In these scenarios, adaptively assigning treatments can lead to complex dependencies among units, causing traditional ATE estimators to become biased or inefficient and undermining the applicability of conventional asymptotic results. Despite its practical importance, this setting remains methodologically less explored. This paper develops a general framework for ATE estimation and inference under sequential designs in finite populations. We study the asymptotic behavior of the average treatment effect estimators under a class of sequential Bernoulli assignment mechanisms, in which the probability of assigning treatment to unit i may depend on the observed history up to that point. Formally, \mathbb{P}(K_{i}=1\mid\mathcal{F}_{i-1})=p_{i}, p_{i}\in\mathcal{F}_{i-1}, where K_{i}=1 indicates that the i th unit is assigned to treatment, and \mathcal{F}_{i-1} denotes the sigma-field generated by the treatment assignments and outcomes of the first (i-1) units. Two classic examples of such assignments that will be discussed in this paper are Wei’s adaptive coin design [ 2 ] , and Efron’s biased coin design [ 3 ] . In the former, the assignment probabilities are expressed as a non-increasing function of the relative imbalance until the previous step, whereas in the latter design these probabilities take constant values that depend on the treatment-control imbalance up to the previous step (for example, fixed values like \eta , (1- \eta ), \tfrac{1}{2}\bigr) . Our contributions proceed in four parts. First, we begin by analyzing a standard estimator of the average treatment effect: the inverse propensity weighting (IPW) estimator, and propose an improvement by introducing a finite-population version of its augmented version (AIPW) that is commonly defined and used in model-based frameworks. Second, for both estimators, we establish central limit theorems under general sequential designs that satisfy a newly defined property called design stability . Third, under two different forms of design stability - strong and weak - we derive estimators of the asymptotic variances of the treatment effect estimators. These variance estimators, and the corresponding confidence intervals for the ATE, are conservative in that they are asymptotically positively biased, leading to asymptotic overcoverage of the confidence intervals. However, the biases vanish under certain forms of treatment effect homogeneity, yielding correct asymptotic coverage of the confidence intervals. Finally, we specialize these results to the two concrete experimental designs mentioned above, arguing that one of them (Wei’s adaptive design [ 2 ] ) satisfies the strong design stability condition, whereas the other (Efron’s design [ 3 ] ) satisfies the weak design stability condition. The remainder of the paper is organized as follows. Section 2 reviews relevant prior work. Section 3 formally defines the problem, introduces the potential outcomes framework, describes the sequential assignment structure, and presents the estimators of interest. Section 4 presents the main theoretical results, including central limit theorems for adaptive designs and conservative asymptotic variance estimators for confidence interval construction. These results are then specialized in Section 5 to two widely used adaptive treatment assignment mechanisms. Sections 5.1 and 5.2 examine the stability of these designs and present simulation studies illustrating the finite-sample performance of the proposed estimators and supporting the theoretical findings. Section 6 concludes with a discussion and directions for future research. Proofs of all main and auxiliary results are provided in the supplementary material. 2 Related work Causal inference in experimental settings is typically framed through two paradigms. The infinite-superpopulation perspective views study units as random draws from an underlying population, with randomness arising from the data-generating process. In contrast, the finite-population or design-based perspective treats the set of units as fixed, with uncertainty introduced solely through the experimental design. This latter view, combined with the potential outcomes formulation, traces back to [ 1 ] , who conceptualized each unit’s treatment and control responses as fixed quantities and attributed randomness entirely to randomization. While classical asymptotic theory [ 9 , 10 ] is often aligned with the superpopulation framework, many applications, particularly randomized trials and survey sampling, are more naturally analyzed from the finite-population perspective [ 11 , 12 , 13 , 14 ] . The study of asymptotic normality in causal inference can be traced back to results on simple random sampling. Classical central limit theorems were established by [ 15 ] , [ 16 ] , [ 17 ] , with convenient formulations presented in [ 18 , 9 ] . These sampling-based central limit theorems can also be viewed as special cases of the more general results for rank statistics [ 19 , 20 , 21 , 22 ] . Further foundational work includes the theory of U-statistics developed by [ 23 ] and the weak convergence results of [ 24 ] , which laid the groundwork for modern asymptotic theory in survey sampling and experimental design. Because treatment and control groups in randomized experiments correspond to simple random samples from the finite set of experimental units, these sampling-based central limit theorems are directly applicable to the difference-in-means estimator of the average treatment effect. This connection underlies much of the early asymptotic justification in randomization-based causal inference [ 25 , 26 ,

Read the complete text