A cautionary note for plasmode simulation studies in the setting of causal inference

Pamela A. Shaw, Susan Gruber, Brian D. Williamson, Rishi Desai, Susan M. Shortreed, Chloe Krakauer, Jennifer C. Nelson, Mark J. van der Laan · 2025

Read A cautionary note for plasmode simulation studies in the setting of causal inference by Pamela A. Shaw, Susan Gruber, Brian D. Williamson, Rishi Desai, Susan M. Shortreed, Chloe Krakauer, Jennifer C. Nelson, Mark J. van der Laan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11740.txt.

Introduction

Pamela Shaw and co-authors sound a warning about a popular simulation method in this 2025 statistics paper. Plasmode simulation mixes real data with made-up data to test how well estimators work, with the truth known by design. Teams use it to judge methods for drug studies drawn from health records. Two frameworks for building such simulations are in wide use.

The authors show, with math and with tests, that one framework can mislead. It can make sound estimators look biased and make their error ranges look too narrow. The flaw persists even with large samples and trial data. The paper ends with guidance on picking the safer framework, so method comparisons stay fair while the data still look real.

Catalog record

CREATORPamela A. Shaw, Susan Gruber, Brian D. Williamson, Rishi Desai, Susan M. Shortreed, Chloe Krakauer, Jennifer C. Nelson, Mark J. van der Laan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-11740

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A cautionary note for plasmode simulation studies in the setting of causal inference Pamela A. Shaw, Susan Gruber, Brian D. Williamson, Rishi Desai, Susan M. Shortreed, Chloe Krakauer, Jennifer C. Nelson, Mark J. van der Laan Abstract Plasmode simulation has become an important tool for evaluating the operating characteristics of different statistical methods in complex settings, such as pharmacoepidemiological studies of treatment effectiveness using electronic health records (EHR) data. These studies provide insight into how estimator performance is impacted by challenges including rare events, small sample size, etc., that can indicate which among a set of methods performs best in a real-world dataset. Plasmode simulation combines data resampled from a real-world dataset with synthetic data to generate a known truth for an estimand in realistic data. There are different potential plasmode strategies currently in use. We compare two popular plasmode simulation frameworks. We provide numerical evidence and a theoretical result, which shows that one of these frameworks can cause certain estimators to incorrectly appear overly biased with lower than nominal confidence interval coverage. Detailed simulation studies using both synthetic and real-world EHR data demonstrate that these pitfalls remain at large sample sizes and when analyzing data from a randomized controlled trial. We conclude with guidance for the choice of a plasmode simulation approach that maintains good theoretical properties to allow a fair evaluation of statistical methods while also maintaining the desired similarity to real data. Key words: data-generating process, outcome-generating model, parametric simulations, resampling, simulation studies, statistical plasmodes 1 Introduction Simulation studies are typically used to empirically evaluate the relative performance (e.g., bias, efficiency) of estimation methods ( Morris et al.,, 2019 ; Koehler et al.,, 2009 ) . One concern is that simplistic computer-generated data structures may lead to performance that is different than what would be expected in real-world data. This limitation motivated a simulation technique known as “plasmode simulation”, in which one or more key elements of the data, such as the treatment effect, is computer-generated, but others are generated by bootstrap sampling real-world data. Several authors have used plasmode simulation to select the best performing method to handle missingness in confounders in electronic health records (EHR) data ( Weberpals et al.,, 2024 ; Weberpals et al.,, 2025 ; Williamson et al., 2024a, ; Getz et al.,, 2023 ) . It has been used to evaluate machine learning methods to build prediction models ( Hafermann et al.,, 2022 ; Karim,, 2024 ) , to study the performance of different propensity score approaches and other causal methods to estimate treatment effects ( Conover et al.,, 2021 ; Ress and Wild,, 2024 ; Rudolph et al.,, 2023 ) , and to compare statistical methods in a variety settings ( Alfaras et al.,, 2021 ; Iyassu et al.,, 2024 ; Duchesneau et al.,, 2022 ) . Despite the frequency of use, formal mathematical justification to guide the proper application of plasmode simulation has not been fully considered. One of the first uses of the term “plasmode” to describe data where key elements are known to fit some true underlying model was by Cattell and Jaspers, (1967) . Decades later Gadbury et al., (2008) define plasmode “to describe datasets that are derived from real data but for which some truth is known,” and demonstrate the utility of plasmode simulation in high-dimensional settings. Franklin et al., (2014) described a now popular plasmode simulation procedure to evaluate estimation methodologies for a treatment exposure estimand in the context of longitudinal health care databases. Their method specifies bootstrap sampling covariates and treatment together, then generating the outcome conditional on resampled data. They suggest this Sample Treatment framework preserves covariate-treatment relationships, while still allowing the investigator to know the true (simulated) causal effect. An alternate plasmode simulation approach is to bootstrap sample only the covariates ( W ), then simulate treatment ( A ) conditional on the covariates. In keeping with the plasmode philosophy, one can generate A from a propensity score model trained on the actual data, then generate the outcome, Y , conditional on W and then generated A . Many papers have used this Generate Treatment plasmode framework in analyses of pharmacoepidemiologic data ( Wang et al.,, 2018 ; Conover et al.,, 2021 ; Franklin et al.,, 2017 ) . In a recent tutorial paper on plasmode simulation, Schreck et al., (2024) discuss that in addition to sampling covariates, one can choose to also simulate artificial covariates from a parametric model, but do not go into details as to when one would want to do so. We compare the common Sample Treatment and Generate Treatment plasmode frameworks and show only the latter approach can be reliably used to evaluate the statistical performance of many typical casual inference methods for evaluating a point treatment effect. We begin by defining a set of casual estimands and common statistical methods whose performance will be evaluated with plasmode simulation. We then describe the two plasmode frameworks and discuss why only the Generate Treatment framework will satisfy the mathematical conditions necessary to guarantee the desirable analytical properties when evaluating performance for several estimators relying on the propensity score. Numerical studies are performed to evaluate the two plasmode simulation procedures for a variety of fully synthetic data settings. We conduct a series of plasmode simulation studies using real-world data to further illustrate the superior performance of the proposed Generate Treatment framework. We conclude with a summary of lessons learned regarding plasmode simulation to evaluate statistical methods. 2 Motivating setting In observational studies, a principal concern when estimating a treatment effect is confounding bias. We show that the problematic issues that arise when doing Sample Treatment plasmode simulations are relevant quite generally for common estimators of causal estimands. A key issue is that not all estimators are affected in the same manner by this plasmode approach, which means the perceived relative performance of the estimators can be misleading. In this section, we formally define a set of treatment effect estimands and estimators to be compared using the two plasmode simulation frameworks. 2.1 Notation and estimands Consider data with n i.i.d. observations O=(W,A,Y) , W a vector of baseline covariates, A a binary treatment indicator, and outcome Y . We define potential outcome Y(a) for treatment exposure A=a . Let \mu_{a}=E[Y(a)] . We define the marginal ATE = \mu_{1}-\mu_{0} and RR =\mu_{1}/\mu_{0} . We also consider the coefficient for treatment A in a logistic regression model for Y adjusted for W , namely the log conditional odds ratio (logcOR) for A , a common estimand for binary outcomes. 2.2 Estimators We seek to evaluate the performance of algorithms that require correct specification of the propensity score (PS) for consistency, as well as those that do not. We consider the following common estimators for the ATE: - Propensity score matching (Match) . This algorithm uses the generalized full optimal matching algorithm with replacement ( Hansen,, 2004 ; Sävje et al.,, 2021 ) to generate weights for each observation suitable for evaluating the ATE. The outcome model for E(Y|A) is then estimated using a weighted, unadjusted linear regression of the outcome on treatment. - Inverse probability of treatment weighting (IPTW) . Observation weights are stabilized using the marginal treatment probability ( Robins et

Read the complete text