Conformal Inference for Experimental Attrition in Social Science Research
Xiangyu Song · 2026
Read Conformal Inference for Experimental Attrition in Social Science Research by Xiangyu Song (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00504.txt.
Introduction
Song offers a conformal method for attrition in social-science trials. It means people dropping out of surveys and field trials. Standard fixes include complete-case analysis, multiple imputation, and weighting. Those fixes assume a lot that may fail in practice. Song's method joins recent statistical inference advances with old missing-data tools. It yields prediction intervals for treatment effects that are robust and precise. Tests show better coverage and narrower intervals than common rivals. Rework of two past trials shows effects for stayers versus dropouts.
Catalog record
| CREATOR | Xiangyu Song |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 147585 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.00504
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Conformal Inference for Experimental Attrition in Social Science Research Xiangyu Song Abstract Attrition in survey and field experiments presents a challenge for social science research. Common approaches to deal with this problem – such as complete case analysis, multiple imputation, and weighting methods – rely on strong assumptions that may not hold in practice. This paper introduces a new method that combines recent advances in statistical inference with established tools for handling missing data. The approach produces prediction intervals for treatment effects that are both robust and precise. Evidence from simulation studies shows that the method achieves better coverage and produces narrower intervals than common alternatives. The reanalysis of two recently published experimental studies illustrates how this framework allows researchers to compare treatment effects across participants who remain in the study, those who drop out, and the full sample. Taken together, these results highlight how the proposed approach provides a stronger foundation for causal inference in the presence of attrition. Keywords : conformal inference, missing data, experiment attrition, causal inference 1 Introduction Since the seminal contributions by Neyman and Fisher in the 1920s ( 9 ; 50 ) , the use of randomized experiments (randomized controlled trials) has become a cornerstone for estimating causal effects ( 21 ; 24 ) . While rare in the 1970s and 1980s ( 32 , e.g.,) , experimental studies have expanded rapidly in the social sciences over the past two decades ( 24 ) . Political scientists now routinely employ survey and field experiments, often alongside survey instruments, to evaluate theories and estimate causal effects ( 22 ; 26 , e.g.,) . Yet, like all empirical strategies, experiments face threats to validity, particularly attrition ( 14 ) , which complicates inference ( 12 ) . In survey experiments, non-response due to attrition prevents observation of outcomes for all participants, while in field experiments attrition reduces sample size and statistical power. When attrition is non-random, it introduces selection bias that undermines theoretical claims. Ignoring attrition and restricting analysis to observed samples generally yields biased and inconsistent estimators ( 6 ) . Existing approaches to address attrition each have their own limitations. Some rely on unverifiable assumptions about model specification or data structure, such as sample selection models ( 15 ) , instrumental variables ( 21 ) , and imputation or complete case methods ( 10 ; 29 ; 45 , e.g.,) . Some reweighting methods, like inverse propensity weighting (IPW) ( 20 ; 21 ) , adjust for missingness using observed data but do not leverage information from attrited observations and can yield biased estimates when missingness depends on treatment and covariates. Other bounding approaches, like partial identification ( 18 ; 19 ; 33 ; 6 ) , relax these parametric assumptions but are not designed to address the covariate shift problem, where the covariate distribution differs between observed and attrited groups. In this paper, I propose to leverage conformal inference, a nonparametric and distribution-free method for uncertainty quantification on individual predictions, to construct valid prediction intervals for treatment effects in the presence of attrition. Conformal inference provides a way to quantify uncertainty without relying on strong parametric assumptions on the data generating process (DGP). Intuitively, the method works by comparing the model’s predictions to the actual observed outcomes. By assessing how well the model’s predictions align with the observed values, it determines a range (or interval) around each prediction that will contain the true value with a user-specified level of confidence (e.g., 90% or 95%). The advantage of conformal inference is that, regardless of the complexity of the model or the distribution of the data, it guarantees coverage: if we ask for 95% coverage, then, on average, about 95% of the prediction intervals will indeed contain the true outcome. This property makes conformal inference particularly appealing for causal inference, where we often want reliable, distribution-free uncertainty quantification for treatment effects at the individual or subgroup level. Recent advances in the conformal inference literature have demonstrated its potential in addressing causal inference problems under attrition. 37 provide a theoretical framework for using conformal inference to construct prediction intervals for counterfactuals and individual treatment effect (ITE). However, their method is not designed specifically to handle missing data problems induced by attrition, as it relies on relatively strong missing data assumptions, and the resulting prediction intervals can be overly wide (non-informative) in practice. Building on this work, 56 and 11 introduce approaches that lead to more accurate and efficient prediction intervals. Rather than using the potential outcome framework, 56 reformulate the covariate shift problem as a missing data problem, and provide a doubly robust and computationally efficient method for constructing prediction intervals for estimands of interest. However, they do not directly tackle the issue of missing data due to attrition. This is why one of the core motivations of this paper is to formally address missing data problems induced by experimental attrition. This paper makes three contributions to the literature. First, it introduces conformal inference as a useful framework for political science research, demonstrating how it can provide reliable measures of uncertainty for causal estimands such as treatment effects and counterfactuals. By focusing on distribution-free and model-agnostic inference, this approach helps researchers quantify uncertainty even when conventional statistical assumptions may not hold. Second, building on 37 and 11 , this paper combines conformal inference with flexible estimators to construct prediction intervals for treatment effects among participants who drop out of experiments. It offers a general and practical way to handle missing data problems caused by experimental attrition under the potential outcome framework. Third, this paper presents one of the most comprehensive evaluations of how conformal inference performs across different predictive models, underscoring both its strengths and its potential limitations for empirical applications in social science research. I demonstrate the advantages of my proposed approach to deal with experimental attrition in two ways. First, through extensive Monte Carlo (MC) simulations, this paper shows that the proposed method can construct prediction intervals for the treatment effects in the attrition group with both valid coverage and narrower interval length. It also outperforms existing parametric and nonparametric methods in terms of empirical coverage and average interval length. Second, by reanalyzing two recently published experimental studies, this paper demonstrates how the proposed method can be applied in practice to compare treatment effects across participants who remain in the study, those who drop out, and the full sample. In addition, it enables researchers to aggregate individual-level treatment effects to estimate average treatment effects for different subpopulations. Overall, compared with current methods, the proposed approach produces more precise and robust prediction intervals for the causal estimands of interest. The rest of the paper is organized as follows. Section 2 goes over the current approaches to deal with the missing data problem. Section 3 introduces the problem setup. Section 4 provides a brief review of conformal inference and the covariate shift problem. Section 5 discusses the conformal inference method for the missing data problem induced by experimental attrition. Section 6 presents the proposed method for constructing prediction intervals for counterfactuals and ITE in the presence of experimental attrition. Section 7 evaluates the performance of the proposed method through extensive MC simulations. Section 8 illustrates the