Double Sampling for Informatively Missing Data in Electronic Health Record‐Based Comparative Effectiveness Research Authors: Alexander W. Levis, Rajarshi Mukherjee, Rui Wang, Heidi Fischer, Sebastien Haneuse Published: 2024 ABSTRACT ABSTRACTMissing data arise in most applied settings and are ubiquitous in electronic health records (EHR). When data are missing not at random (MNAR) with respect to measured covariates, sensitivity analyses are often considered. These solutions, however, are often unsatisfying in that they are not guaranteed to yield actionable conclusions. Motivated by an EHR‐based study of long‐term outcomes following bariatric surgery, we consider the use of double sampling as a means to mitigate MNAR outcome data when the statistical goals are estimation and inference regarding causal effects. We describe assumptions that are sufficient for the identification of the joint distribution of confounders, treatment, and outcome under this design. Additionally, we derive efficient and robust estimators of the average causal treatment effect under a nonparametric model and under a model assuming outcomes were, in fact, initially missing at random (MAR). We compare these in simulations to an approach that adaptively estimates based on evidence of violation of the MAR assumption. Finally, we also show that the proposed double sampling design can be extended to handle arbitrary coarsening mechanisms, and derive nonparametric efficient estimators of any smooth full data functional. Introduction Missing data is a well‐studied problem, with researchers having a vast array of statistical methods at their disposal including inverse‐probability weighting (IPW) [1], multiple imputation [2], and doubly‐robust methods [3, 4]. For the majority of these, a missing at random (MAR) assumption [5] is, in one way or another, invoked. For settings where an MAR assumption is not viewed as plausible, methods exist based on alternative sets of identifying assumptions [6], availability of an instrumental variable [7] or a “shadow variable” [8], sensitivity analyses [9] and the estimation of bounds [10]. Interestingly, common to all of these methods is that they approach the task of dealing with missing data at the analysis stage, that is with an exclusive focus on methods for the data at‐hand. An alternative strategy is to engage in additional data collection, referred to in this paper as double‐sampling, specifically to obtain information that could either inform the plausibility of missingness assumptions or be used in an analysis to mitigate bias, or both. Such a strategy is common when addressing confounding [11] and measurement error and/or missclassification [12, 13], but seems to have been under‐explored as a strategy for addressing missing data [14, 15, 16, 17, 18]. Moreover, a general treatment of double sampling for missing data in the context of causal inference has not been developed. One important area of biomedical and public health research where missing data is almost ubiquitous is that of studies making use of electronic health records (EHR). With large sample sizes and rich covariate information over extended periods, EHR data represent a significant and cost‐effective opportunity [19]. Furthermore, these data present a key alternative when randomized clinical trials are not feasible or could not be conducted ethically. EHR systems, however, are typically designed to support clinical and/or billing activities, and not for any particular research agenda. As such, investigators who wish to use EHR data must deal with potential threats to validity including, as mentioned, missing data. Moreover, whether a particular data element is observed in an EHR is likely dependent on the complex interplay of numerous factors [20], which may, in turn, cast doubt on the plausibility of the MAR assumption. In such settings, augmentation of the EHR with additional information via double sampling may be especially helpful [21]. Indeed, Koffman et al. [22] recently reported on a telephone‐based survey used to obtain additional information for use in an investigation of the association between bariatric surgery and five‐year weight outcomes using data from an EHR [23]. Key to the latter was the fact that many subjects who had undergone bariatric surgery disenrolled from their health plan before their five‐year post‐surgery date. Towards understanding the reasons for disenrollment and to evaluate the MAR assumption, the investigators conducted the telephone‐based survey to obtain the otherwise missing weight information and other relevant factors. Although their report focuses on disenrollment in relation to missingness, the authors did stress the potential for using the augmented data to correct an otherwise invalid analysis (i.e., of the association between bariatric surgery and weight at follow‐up), but identified the need for novel statistical methods to be developed. Motivated by this backdrop, we consider double sampling as a means to deal with potentially informatively missing or MNAR data. Specifically, we present novel identification results for the causal average treatment effects in observational settings with missing outcome data. Based on these we describe a suite of five analysis strategies for the context we consider, each distinguished by the nature of the data that is taken to be available, the assumptions that analysts are required to make and the estimator that is to be employed. For the proposed strategies that, to‐date, have not been formally described, we establish asymptotic results and characterize efficiency and robustness properties. Finally, we generalize many of these results to allow for arbitrary coarsening of the desired complete data of interest, where complete data are recovered on a subsample via intensive follow‐up. Note, throughout, when not provided in the text, detailed proofs are presented in Appendices A and B in the Supporting Information. A Hypothetical EHR‐Based Study Context, Notation, and Terminology To anchor the methods we propose, consider a hypothetical EHR‐based study for which the goal is to compare two bariatric surgery procedures (e.g., Roux‐en‐Y gastric bypass vs. sleeve gastrectomy) in relation to three‐year weight change outcomes. To that end, we assume that appropriate inclusion/exclusion criteria have been specified and operationalized to identify all patients in the EHR who are ‘eligible’ for the study, resulting in a sample of size n which is taken to be a random sample from the population of interest. Formally, let A∈𝒜, with |𝒜|<∞, denote the treatment and Y∈ℝ the outcome of interest. In the hypothetical study, A represents the type of surgery, and Y the change in BMI at three years post‐surgery (slightly different from the study described above [22]) relative to baseline. Then, let Y(a) denote the potential outcome or counterfactual had we fixed treatment level A=a, for a∈𝒜. We take the target parameter of interest to be some contrast among the mean counterfactuals, 𝔼[Y(a)]. In the hypothetical bariatric surgery study, for example, a natural contrast would be the average treatment effect (ATE), 𝔼[Y(1)]−𝔼[Y(0)]. Throughout this work, towards estimating 𝔼[Y(a)] using the data from the EHR, we invoke the usual ‘causal’ identifying assumptions of consistency, no unmeasured confounding and positivity [24]. Regarding the control of confounding bias, we assume that a sufficient set of confounders L∈ℝd, available in the EHR, has been identified to render the assumption of no unmeasured confounding plausible. For the setting just described, we refer to (L,A,Y) as the complete data and conceive it as arising from some joint distribution, Pc. Given an i.i.d sample of size n from Pc one could estimate 𝔼[Y(a)] by, say, targeting the g‐formula functional, χa(Pc)=𝔼Pc[𝔼Pc[Y|L,A=a]], which identifies 𝔼[Y(a)] under the aforementioned causal assumptions [25]. For example, one could use the plugin estimator to estimate the ATE via ATE^=χ1(Pc^)−χ0(Pc^), relying on the fit of an outcome regression model for 𝔼Pc[Y|L,A=a]. To complete the context we consider, we assume that, while L and A are measured on all patients, the outcome is only partially observed; that is for some patients the value of Y is missing. In the hypothetical example this may arise because a patient disenrolled from the health plan prior to the 3‐year post‐surgery date or because they did not have an encounter within some (reasonable) window of the date. Formally, let R∈{0,1} be an indicator for the observance of Y in the EHR; at the outset, therefore, the information that is readily available consists of n i.i.d replicates of (L,A,R,RY), referred to as the incomplete data. Analysis Strategy #1 Given incomplete data, one way forward is to combine a complete data strategy, which one would use had such data been available, with some approach for ‘dealing’ with the missing data. For example, one could combine the use of the g‐formula indicated above with either inverse‐probability weighting based on a model for R or multiple imputation for the missing values of Y. In addition to the usual causal assumptions, the validity of such a procedure will hinge on a MAR assumption, such as: Assumption 1 (missing at random outcomes) R⫫Y|L,A. (missing at random outcomes) R⫫Y|L,A. Crucially, with this assumption in hand, one can proceed as would be done otherwise on the basis of those individuals with R=1, since the distribution of Y|L,A is the same as Y|L,A,R=1—that no information on the distribution of Y|L,A,R=0 is available can be safely ignored. For instance, one can employ the g‐formula on the basis of the complete‐case outcome model 𝔼[Y|L,A=a,R=1]. The Potential for MNAR Suppose, however, that a discussion among the collaborators at the design stage of the study (i.e., at the time the study is being planned and/or a grant/proposal is being written) raises the possibility that the outcome data are MNAR; that is, that Assumption 1 may not hold with respect to the baseline covariates L that will be available. In the hypothetical bariatric surgery study, for example, it may be that patients with worse outcomes (in a manner beyond what can be predicted with L and A) interact more often with the health care system, and thus have less missing data and/or are less likely to disenroll from their health plan. It is also possible that subjects with worse outcomes are more likely to drop out, perhaps to receive care outside of their original health plan. The key challenge that a violation of Assumption 1 poses is that the distribution of Y|L,A,R=0 can no longer be safely ignored, and yet there is no information to learn about it. Since MNAR is not testable, the literature on methods for data that are MNAR has generally focused on frameworks for sensitivity analyses and analyses that directly provide bounds on the effect of interest. An alternative to these typically post‐hoc approaches, especially if the potential for MNAR is established early in the research process, is to engage in additional data collection efforts that are specifically and preemptively tailored to being able (at least partially) to move ‘beyond’ MNAR. Double Sampling When MNAR is Suspected Central to the proposed work is that follow‐up is performed for a subsample of the patients for whom R=0, and that the corresponding (otherwise missing) value of Y is ascertained. While such additional data collection is employed in a wide range of settings, in this paper we follow Frangakis and Rubin [15] and use the label double sampling. Notation and Terminology Let S∈{0,1} be an indicator for whether a given patient is selected into the follow‐up subsample and outcome data are successfully obtained. Note, by design and as can be seen from Figure 1, S≡S(1−R); S can only be 1 if R=0 and is equal to 0 deterministically if R=1. With this notation, we refer to O=(L,A,R,S,(R+S)Y) as the final observed data and the corresponding joint distribution by Po. Throughout this section, we assume that we observe a random sample O1,…,On∼iidPo. Schematic illustrating the double sampling study design. Gray boxes indicate subgroups for which the outcome is observed: either subjects are observed in the EHR directly (R=1) or are selected in the follow‐up sample (R=0 and S=1). To complete terminology, we refer to (L,A,Y,R,S) as the full data and denote the corresponding joint distribution by Pf. Note, both Pc and Po are induced by Pf via appropriate marginalization. As will become clear, the use of distinct labels is to help clarify where and how key identifying conditions are employed. Identification As with analysis strategy #1, we proceed by specifying assumptions that permit the identification of the complete data distribution, Pc, despite only having access to the final observed data. Specifically, consider the following assumptions: Assumption 2 (no informative second‐stage selection) S⊥⊥Y|L,A,R=0. (no informative second‐stage selection) S⊥⊥Y|L,A,R=0. Assumption 3 (positivity of second‐stage selection probabilities) For some ϵ>0, it holds that Po[S=1|L,A,R=0]≥ϵ,Po‐almost surely. (positivity of second‐stage selection probabilities) For some ϵ>0, it holds that Po[S=1|L,A,R=0]≥ϵ,Po‐almost surely. Note, Assumption 2 can be viewed as an MAR‐type assumption, specifically in relation to the mechanism underpinning who is selected to be double‐sampled and successfully followed‐up, and, thus, represents an alternative to or replacement for the usual MAR assumption regarding R. Based on these, consider the following identification result: Proposition 1 Under Assumptions 2 and 3 , the full data distribution Pf is identified from the final observed data distribution Po. Under Assumptions 2 and 3 , the full data distribution Pf is identified from the final observed data distribution Po. Let po=∂Po/∂μ and pf=∂Pf/∂μ denote the densities for Po and Pf, respectively, both with respect to some dominating measure μ. We can then factor the full data distribution as: pf(L,A,R,S,Y)=pf(L,A,R,S,RY,(1−R)Y)=po(L,A,R,S,RY)pf(Y|L,A,R=0,S)1−R=po(L,A,R,S,RY)po(Y|L,A,R=0,S=1)1−R with the last of these steps enabled by Assumption 2. Since the last expression depends solely on po, it follows that Pf is identified by Po. Note, we may safely introduce S=1 in the conditioning event by Assumption 3. Let po=∂Po/∂μ and pf=∂Pf/∂μ denote the densities for Po and Pf, respectively, both with respect to some dominating measure μ. We can then factor the full data distribution as: pf(L,A,R,S,Y)=pf(L,A,R,S,RY,(1−R)Y)=po(L,A,R,S,RY)pf(Y|L,A,R=0,S)1−R=po(L,A,R,S,RY)po(Y|L,A,R=0,S=1)1−R with the last of these steps enabled by Assumption 2. Since the last expression depends solely on po, it follows that Pf is identified by Po. Note, we may safely introduce S=1 in the conditioning event by Assumption 3. An immediate consequence of Proposition 1 is that the complete data distribution, Pc, is identified and, hence, any functional depending on Pc is identified. Therefore, given Assumptions 2 and 3, one can use the final observed data to estimate quantities of interest, such as mean counterfactuals. This observation, in turn, is the basis for a series of additional analysis strategies proposed in the remainder of this section. Analysis Strategy #2 Given an i.i.d sample of O=(L,A,R,S,(R+S)Y), Proposition 1 implies that, if Assumptions 2 and 3 hold, we can nonparametrically identify 𝔼[Y(a)] via the g‐formula representation: (1) τa(Po)=𝔼Po[μa,R(L)γa(L)+μa,S(L)(1−γa(L))] with γa(L)=Po(R=1|L,A=a), μa,R(L)=𝔼Po[Y|L,A=a,R=1], and μa,S(L)=𝔼Po[Y|L,A=a,S=1]. With this representation, one can construct an estimator of the ATE by targeting τa(Po). One simple approach would be to estimate each component nuisance function, that is γa(L), μa,R(L) and μa,S(L), via parametric modeling and combine via an empirical version of expression (1). In the following, we derive and characterize a nonparametric efficient and multiply robust estimator τ^a of τ(Po). As we will see, efficient estimation additionally requires a model for the treatment probability πa(L)=Po(A=1|L), as well as for the double sampling probabilities ηa,0(L)=Po(S=1|L,A=a,R=0). Note, as we will prove, this approach has the advantage that under relatively mild conditions, n‐rate convergence can still be attained while using flexible machine learning‐based models for each nuisance function. The following result is a straightforward application of Proposition A.2 proved in Appendix A of the Supporting Information. Theorem 1 Let μa(L)=μa,R(L)γa(L)+μa,S(L)(1−γa(L)). The nonparametric influence function with respect to the maximal tangent space of τa(Po) is τ˙a(O;Po)=μa(L)−τa(Po)+1(A=a)πa(L)R+Sηa,0(L)Y−μa(L)+(1−R)1−Sηa,0(L)μa,S(L) Let μa(L)=μa,R(L)γa(L)+μa,S(L)(1−γa(L)). The nonparametric influence function with respect to the maximal tangent space of τa(Po) is τ˙a(O;Po)=μa(L)−τa(Po)+1(A=a)πa(L)R+Sηa,0(L)Y−μa(L)+(1−R)1−Sηa,0(L)μa,S(L) With this efficient influence function in hand, one can proceed by using the standard one‐step estimator [26, 27], specifically: τ^a=τa(P^o)+1n∑i=1nτ˙a(O;P^o) For simplicity we assume in the following results that the nuisance functions in P^o are trained on a separate independent sample. In practice, one can use cross‐fitting which involves splitting the data into training and test folds, fitting P^o on the training fold, and computing the one‐step estimator in the test fold [27, 28]. Full efficiency can be recovered by swapping the roles of the folds and averaging the resulting estimators [29]. The following result is obtained directly from Theorem A.1 and Proposition A.4 (see Appendix A of the Supporting Information): Theorem 2 Suppose τ˙a(·;P^o)−τ˙a(·;Po)=oP(1). Then τ^a−τa(Po)=OP1n+Biasτa(P^o;Po), where Biasτa(P˜o;Po)=𝔼Po1−πa(L)π˜a(L)μ˜a(L)−μa(L)+𝔼Po(1−γa(L))πa(L)π˜a(L)1−ηa,0(L)η˜a,0(L)μ˜a,S(L)−μa,S(L). for any (fixed) P˜o. Moreover, if Biasτa(P^o;Po)=oP(n−1/2), then n(τ^a−τa(Po))→d𝒩(0,V), where V=VarPo(τ˙a(O;Po)) is the nonparametric efficiency bound. Suppose τ˙a(·;P^o)−τ˙a(·;Po)=oP(1). Then τ^a−τa(Po)=OP1n+Biasτa(P^o;Po), where Biasτa(P˜o;Po)=𝔼Po1−πa(L)π˜a(L)μ˜a(L)−μa(L)+𝔼Po(1−γa(L))πa(L)π˜a(L)1−ηa,0(L)η˜a,0(L)μ˜a,S(L)−μa,S(L). for any (fixed) P˜o. Moreover, if Biasτa(P^o;Po)=oP(n−1/2), then n(τ^a−τa(Po))→d𝒩(0,V), where V=VarPo(τ˙a(O;Po)) is the nonparametric efficiency bound. In addition to consistency, asymptotic normality, and efficiency, Theorem 2 reveals a set of robustness properties of τ^a. Specifically, observe that Biasτa(P˜o;Po)=0 if: (i) (μ˜a,R,μ˜a,S,γ˜a)=(μa,R,μa,S,γa); (ii) (μ˜a,S,π˜a)=(μa,S,πa); or (iii) (π˜a,η˜a,0)=(πa,ηa,0). In particular, if the double sampling probabilities ηa,0 are known by design, then Biasτa(P˜o;Po)=0 if (μ˜a,R,μ˜a,S,γ˜a)=(μa,R,μa,S,γa) or π˜a=πa. This robustness extends to the rate of convergence of the estimator (as in Rotnitzky, Smucler, and Robins [30]), in that the asymptotic bias term is bounded under mild conditions by a sum of product of L2(Po)‐errors in nuisance function estimation: ‖π^a−πa‖·‖μ^a−μa‖ + ‖η^a,0−ηa,0‖·‖μ^a,S−μa,S‖, where ‖f‖2=𝔼Po(f(O)2) for any function f. In particular, if π^a, μ^a, η^a,0, and μ^a,S are each L2(Po)‐consistent at rate at least n−1/4, then (under mild conditions) τ^a is n‐consistent and asymptotically efficient. Another appealing consequence of the asymptotic normality result in Theorem 2 is that simple Wald‐type asymptotically valid confidence intervals are immediately available. For example, we can estimate Var(τ^a) with Var^(τ^a)=1nV^=1n2∑i=1n(τ˙a(Oi;P^o))2, with corresponding Wald‐type confidence interval given by τ^a±z1−α/2Var^(τ^a), where zα denotes the α‐quantile of the standard normal distribution. Analysis Strategy #3 A key feature of τ^a is that there was no need to invoke the usual MAR assumption for R; indeed, no assumptions regarding R are invoked. Suppose, however, that following data collection via double‐sampling, the investigative team decides that MAR Assumption 1 may indeed be plausible. It may be, for example, that new information regarding the mechanisms that underpin R becomes available; see Section 4. Alternatively, suppose the MAR assumption was always viewed as being potentially plausible and that the additional data was collected for the purpose of gaining efficiency in estimating the parameter of interest. In either of these settings, combining Assumption 1 with S≡S(1−R) and Assumption 2, gives that (R,S)⊥⊥Y|L,A. With this, one can nonparametrically identify 𝔼[Y(a)] via the g‐formula representation: τa∗(Po)=𝔼Po[μa,MAR(L)], where μa,MAR(L)=𝔼Po[Y|L,A,R+S=1]. As in Section 3.4, while we could construct an estimator of τa∗(Po) based on a parametric model for the nuisance function μa,MAR(L), we derive a semiparametric efficient estimator of τa∗(Po) that is of a robust augmented IPW form. The following result is proved in Appendix B of the Supporting Information. Theorem 3 Under Assumptions 1 and 2 , and with S≡S(1−R), the semiparametric efficient influence function of τa∗(Po) is τ˙a∗(O;Po)=μa,MAR(L)−τa∗(Po)+1(A=a)(R+S)πa(L){γa(L)+(1−γa(L))ηa,0(L)}×(Y−μa,MAR(L)) Under Assumptions 1 and 2 , and with S≡S(1−R), the semiparametric efficient influence function of τa∗(Po) is τ˙a∗(O;Po)=μa,MAR(L)−τa∗(Po)+1(A=a)(R+S)πa(L){γa(L)+(1−γa(L))ηa,0(L)}×(Y−μa,MAR(L)) We remark that the influence function in Theorem 3 generalizes that discussed in Gilbert, Yu, and Rotnitzky [31], where they pursue two‐phase sampling for outcomes that are expensive to collect—in particular, they assume Po[S=0]=1 and πa is fixed and known. With the result of Theorem 3, one can again proceed using the standard one‐step estimator, specifically: τ^a∗=τa∗(P^o)+1n∑i=1nτ˙a∗(O;P^o) the asymptotic properties of which are established in the following theorem, itself a straightforward consequence of Theorem A.1 and Proposition A.4 in Appendix A of the Supporting Information. Theorem 4 Suppose τ˙a∗(·;P^o)−τ˙a∗(·;Po)=oP(1). Then τ^a∗−τa∗(Po)=OP1n+Biasτa∗(P^o;Po), where Biasτa∗(P˜o;Po)=𝔼Po1−πa(L){γa(L)+(1−γa(L))ηa,0(L)}π˜a(L){γ˜a(L)+(1−γ˜a(L))η˜a,0(L)}×μ˜a,MAR(L)−μa,MAR(L) for any P˜o. Furthermore, if Biasτa∗(P^o;Po)=oP(n−1/2), then n(τ^a∗−τa∗(Po))→d𝒩(0,V∗), where V∗=VarPo(τ˙a∗(O;Po)) is the semiparametric efficiency bound under the assumptions of Theorem 3 . Suppose τ˙a∗(·;P^o)−τ˙a∗(·;Po)=oP(1). Then τ^a∗−τa∗(Po)=OP1n+Biasτa∗(P^o;Po), where Biasτa∗(P˜o;Po)=𝔼Po1−πa(L){γa(L)+(1−γa(L))ηa,0(L)}π˜a(L){γ˜a(L)+(1−γ˜a(L))η˜a,0(L)}×μ˜a,MAR(L)−μa,MAR(L) for any P˜o. Furthermore, if Biasτa∗(P^o;Po)=oP(n−1/2), then n(τ^a∗−τa∗(Po))→d𝒩(0,V∗), where V∗=VarPo(τ˙a∗(O;Po)) is the semiparametric efficiency bound under the assumptions of Theorem 3 . Note that analogous comments to those following Theorem 2 for τ^a can be made here for τ^a∗ as well. First, robustness of τ^a∗ follows from the product bias elucidated in Theorem 4: Biasτa∗(P˜o;Po)=0 if (i) μ˜a,MAR=μa,MAR; or (ii) (π˜a,γ˜a,η˜a,0)=(πa,γa,ηa,0). Moreover, τ^a∗ is n‐consistent and asymptotically efficient if each nuisance function estimate is L2(Po)‐consistent at rate at least n−1/4. Second, we can estimate Var(τ^a∗) with Var^(τ^a∗)=1nV^∗=1n2∑i=1n(τ˙a∗(Oi;P^o))2, and construct the corresponding Wald‐type confidence interval. Analysis Strategy #4 The key distinction between analysis strategies #2 and #3 is in relation to whether the MAR Assumption 1 holds. In practice there may not be a consensus as to whether it plausibly holds. For example, one collaborator may believe firmly that the outcomes were initially MAR given (L,A) while another may believe that there remains residual dependence of missingness status on the outcome (either directly or through some other, as‐yet unmeasured, factor). In this setting, one option may be to conduct a hypothesis test using the observed data to assess Assumption 1; while this assumption is untestable if all one has access to is the initially observed data, under Assumptions 2 and 3 the complete data distribution is identified and, in principle, MAR can be tested. Depending on the results of this test, one could report an analysis based on τ^a which does not rely on Assumption 1, or τ^a∗ which does. While appealing in its simplicity, naïve use of the corresponding standard error estimator for the chosen estimator would not account for the uncertainty in the estimator selection. As such, inference will, in general, not be valid. Because of this, we do not derive any theory for this approach but do consider it as a comparator in the simulation study of Section 5.1. Analysis Strategy #5 Finally, building on analysis strategy #4, one could proceed with a more explicitly data‐adaptive approach that selects between the two candidate estimators and provides valid post‐selection confidence intervals. To this end, we employ the recently developed methods of Rothenhäusler [32]. The approach from this work assumes that one has access to a collection of estimators, and that one ‘base’ estimator in this collection is consistent for the true target parameter of interest. At a high level, the methodology then anchors the analysis on this ‘base’ estimator, and chooses among the larger collection of estimators (some of which being potentially biased or highly variable) by minimizing a proxy for mean‐squared error. Briefly, as an overview of the method in the one‐parameter case, consider estimation of a generic parameter θ0(P) when there are k+1 asymptotically linear estimators θ^0,θ^1,…,θ^k, such that: (i) the ‘base’ estimator θ^0 is n‐consistent for θ0(P); and (ii) θ^j is n‐consistent for θj(P), for j=1,…,k, where θj(P) may or may not equal θ0(P). With this collection of k+1 candidate estimators in hand, Rothenhäusler [32] proposed an estimator that selects among them by minimizing an estimate of the mean squared error. Formally, let σ^j2 be an estimator of the asymptotic variance of n(θ^j−θj(P)), and ρ^j2 an estimator of the asymptotic variance of n(θ^j−θj(P)−θ^0+θ0(P)). Then the procedure selects the j‾th candidate estimator, where j‾=argminjR‾(j) with R‾(j)=max{0,(θ^j−θ^0)2−ρ^j2/n}+σ^j2/n. Importantly, Rothenhäusler [32] derive asymptotically valid confidence intervals that take into account the uncertainty due to the selection procedure (see their Theorem 4). Toward applying the Rothenhäusler [32] approach to the context of this paper, we take τ^a of analysis strategy #2 to be the ‘base’ estimator (since it does not rely on Assumption 1), with τ^a∗ as an alternative estimator that is, in principle, more efficient than τ^a if Assumption 1 does hold. Then, let V^ = nVar^[τ^a] and V^∗ = nVar^[τ^a∗], and define Q^=1n∑i=1n{τ˙a(Oi;P^o)−τ˙a∗(Oi;P^o)}2 as an estimator of VarPo[τ˙a(O;Po)−τ˙a∗(O;Po)]. The final analysis strategy considers the following ‘data‐adaptive’ estimator of the mean counterfactual: τ^a†:=τ^aifV^z1−α/2, and otherwise choose τ^a∗ if we fail to reject. We computed this estimator across all simulation settings. To evaluate confidence intervals, we again focused on the three parameter values βRA∈{0,0.016,0.032}. In the 5000 simulated datasets for each value, we constructed confidence intervals for the four estimators described above. For τ^1−τ^0 and τ^1∗−τ^0∗, we used influence function‐based Wald‐type confidence intervals. For τ^1†−τ^0†, we constructed confidence intervals based on Theorem 4 of Rothenhäusler [32]. For the ad hoc adaptive estimator, we used the Wald‐type interval corresponding to the baseline estimator chosen according to the hypothesis test—a naive approach which we expect will lead to undercoverage. The results on the grid of βRA values are shown in Figure 3. When βRA=0 (i.e., MAR holds), all estimators are unbiased, τ^1∗−τ^0∗ is most efficient, and the two adaptive estimators have variance somewhere between that of τ^1∗−τ^0∗ and τ^1−τ^0. As βRA increases, the bias of τ^1∗−τ^0∗, which wrongly assumes MAR, increases roughly linearly. The two adaptive estimators also inherit some bias due to being pulled away by τ^1∗−τ^0∗. Interestingly, when βRA becomes really large, indicating quite a substantial violation of MAR, the bias of the two adaptive estimators returns back towards zero, as it becomes increasingly rare for either of these to select the estimator which assumes MAR. Simulation results for experiments of Section 5.2, comparing Strategy #2 of the nonparametric influence function‐based estimator (τ^1−τ^0), Strategy #3 of the semiparametric efficient estimator under MAR (τ^1∗−τ^0∗), Strategy #4 of the ad hoc approach described in Section 5.1, and Strategy #5 of the adaptive estimator of Rothenhäusler [32] (τ^1†−τ^0†). Subplot (a) shows the empirical mean squared error (MSE) of each approach, divided by the MSE of the nonparametric efficient estimator. Subplot (b) shows the empirical variance of each approach, divided by the variance of the nonparametric efficient estimator. Subplot (c) shows the empirical bias of each approach. The results on the focused set of values βRA∈{0,0.016,0.032} are arranged in Table 2. The confidence interval for τ^1−τ^0 has the appropriate coverage in all scenarios, as does the interval for τ^1∗−τ^0∗ when MAR holds. In the two MNAR settings, however, τ^1∗−τ^0∗ is biased and its confidence interval is off target. The confidence interval for the adaptive estimator τ^1†−τ^0† also appears to be valid, with perhaps a bit of undercoverage in finite samples for moderately large values of βRA. Finally, the naive confidence intervals of the ad hoc adaptive estimator tend to be overly narrow. TABLE 2Coverage and other statistics of influence function‐based and adaptive estimators.% BiasRel. varianceRel. MSE% CoverageAverage length βRA=0 τ^1−τ^0 ‐0.021.001.0094.50.036 τ^1∗−τ^0∗ 0.060.500.5095.40.026 τ^1†−τ^0† 0.040.800.8095.80.035Ad hoc adaptive0.080.640.6493.80.026 βRA=0.016 τ^1−τ^0 ‐0.011.001.0094.50.036 τ^1∗−τ^0∗ 10.260.501.0681.80.026 τ^1†−τ^0† 2.871.081.1294.10.036Ad hoc adaptive5.671.011.1882.60.028 βRA=0.032 τ^1−τ^0 ‐0.031.001.0094.50.036 τ^1∗−τ^0∗ 19.570.512.7245.30.026 τ^1†−τ^0† 1.691.261.2891.30.037Ad hoc adaptive4.561.561.6873.60.031 Notes: MSE, mean squared error; % Bias, 100×mean−(τ1(Po)−τ0(Po))τ1(Po)−τ0(Po); Rel. variance, empirical variance of estimator divided by that of τ^1−τ^0; Rel. MSE, empirical MSE of estimator divided by that of τ^1−τ^0; % Coverage, percentage of confidence intervals across 5000 simulation replications that cover the true ATE; Average length, average confidence interval length across 5000 simulation replications. Coverage and other statistics of influence function‐based and adaptive estimators. Notes: MSE, mean squared error; % Bias, 100×mean−(τ1(Po)−τ0(Po))τ1(Po)−τ0(Po); Rel. variance, empirical variance of estimator divided by that of τ^1−τ^0; Rel. MSE, empirical MSE of estimator divided by that of τ^1−τ^0; % Coverage, percentage of confidence intervals across 5000 simulation replications that cover the true ATE; Average length, average confidence interval length across 5000 simulation replications. Data Application In this section, we present an analysis of the proposed methods to data from an EHR‐based study comparing the effect of RYGB (A=1) versus VSG (A=0) bariatric surgery procedures on percent weight change at three years post‐surgery (Y). Data were obtained from three health care sites within Kaiser Permanente: Northern California, Southern California, and Washington. Namely, in line with Arterburn et al. [23], we use data on n=13,514 adult patients who underwent RYGB or VSG between January 2005 and September 2015, with complete weight data at baseline (closest measurement pre‐surgery, up to 6 months) and follow‐up (closest measurement within ± 90 days). See Table S1 in the Supporting Information for a summary of baseline characteristics. These data comprised the “complete‐cases” from a larger collection of 30,991 patients for which follow‐up outcomes were only partially observed. We artificially imposed missingness in the outcome Y on the complete‐case data according to an MAR mechanism, as well as an MNAR mechanism. To construct a realistic MAR mechanism, we modeled the probability of missingness from the original larger collection of 30,991 patients using the following baseline covariates L: baseline weight, health care site, year of surgery, age, gender, race/ethnicity, number of days of health care use in 7‐12 month period pre‐surgery, number of days hospitalized in pre‐surgery year, smoking status, Charlson/Elixhauser comorbidity score, insurance type, clinical statuses for hypertension, coronary artery disease, diabetes, dyslipedemia, retinopathy, neuropathy, and mental health disorders, and use of medicines including, insulin, ACE inhibitors, ARB, statins, other lipid lowering medications, and other antihypertensives. We regressed the indicator for observing Y on A and L via a SuperLearner ensemble (with library {SL.glm,SL.ranger,SL.rpart}) using the corresponding R package [37], thus obtaining γ^ satisfying MAR, that is, the probabilities of initially observing the outcome are γ^≡Po[R=1|L,A]. Next, to construct a MNAR mechanism, we augmented the fitted values γ^ to include dependence on the outcome Y: γ^↦expit{(logit(γ^)+Y˜[0.7+0.8(1−A)−1.21(diabetes) +0.61(non‐commercial insurance)])}, where Y˜ is standardized BMI change at 3 years. Finally, these models were used to impose missing outcomes on the sample of n=13,514 patients by sampling R according to a Bernoulli with probability given by the fitted values from the models. The resulting marginal probabilities of missingness were 26% and 28% in the MAR and MNAR settings, respectively. In each of the missingness settings described above, we considered collecting a random subsample (i.e., those with S=1) of initially missing outcomes of size 500, 1000, and 1500. For each of the six resulting datasets, we computed and compared point estimates and 95% confidence intervals for analysis strategies #1, #2, #3, and #5. For analysis strategy #1, we targeted ξ1(Po)−ξ0(Po), where ξa(Po)=𝔼Po(μa,R(L)), with an augmented‐IPW estimator as in Davidian, Tsiatis, and Leon [35] and Williamson, Forbes, and Wolfe [36]. Analysis strategies #2, #3, and #5, correspond to estimators τ^1−τ^0, τ^1∗−τ^0∗, and τ^1†−τ^0†, respectively. As a benchmark, we compare to a standard full‐data augmented‐IPW estimator that uses outcome data from all n=13,514 patients. For all estimators, we used flexible SuperLearner ensembles for each component nuisance function, with a library of SL.glm, SL.ranger, and SL.rpart. Results are summarized in Figure 4. When MAR holds, all estimators perform well with respect to the benchmark analysis, with τ^1∗−τ^0∗ having smallest variance, as anticipated. On the other hand, the estimators that assume MAR appear to be biased under MNAR, while the nonparametric efficient estimator τ^1−τ^0 and adaptive estimator τ^1†−τ^0† (which do not assume MAR) are robust to this violation of MAR. As expected, as the second stage subsample size increases, precision improves for analysis strategies #2, #3, and #5, which incorporate this data. Results for data application in Section 6, showing the estimated effect on percent total weight change at 3 years. Subplots (a), (b), and (c) correspond to follow‐up subsamples of size 500, 1000, and 1500 individuals. Blue and red lines refer to estimated 95% confidence intervals when outcomes are initially MAR, and MNAR, respectively. Point estimates are marked with black dots. Strategy #1 = augmented‐IPW estimator with incomplete data only, assuming MAR; Strategy #2 = influence‐function based estimator using double sampling, τ^1−τ^0; Strategy #3 = semiparametric efficient estimator under MAR, τ^1∗−τ^0∗; Strategy #5 = adaptive estimator of Rothenhäusler [32], τ^1†−τ^0†. The black dashed line represents the point estimate for the benchmark full data analysis. Discussion In brief, this article proposes a set of substantive assumptions and statistical methods for the use of double‐sampling as a means to address potentially MNAR missing data, when scientific interest lies in estimating causal ATEs. Key to our contributions are a suite of novel analysis strategies that exploit data arising from the double sampling scheme, coupled with identifying assumptions that guarantee the corresponding estimators to be asymptotically normal, efficient, and robust. To summarize and highlight our main empirical and theoretical findings, consider now being faced with partially missing outcomes as in the hypothetical EHR‐based study of Section 2. If one is determined not to allocate budget for additional (i.e., second stage) data collection, then one may proceed with an MAR‐based analysis using the initially observed incomplete data (Strategy #1), though there is potential for substantial bias when the MAR assumption is violated (see e.g., Figure 2b,c). Alternatively, one may be content with computing nonparametric bounds or conducting a sensitivity analysis to be more robust to potential violations of MAR. These approaches have downsides: (i) under no assumptions about the missing data structure, nonparametric bounds are necessarily very wide, and (ii) while sensitivity analyses can provide confidence sets that are robust to certain degrees of violation of MAR [9], there may be no data to support the actual value of the magnitude of this violation, making it difficult to draw actionable conclusions. If instead one is able to obtain second‐stage follow‐up data, several more options become available: nonparametric estimation based on Assumption 2 (Strategy #2), semiparametric estimation assuming the initially observed outcomes are MAR (Strategy #3), or an adaptive approach that selects between the latter two based on evidence of MAR at the first stage (Strategies #4 and #5). If one is certain that the initially observed outcomes are MAR, then Strategy #3 is valid and semiparametric efficient (see Theorem 4). In fact, in our simulations, the estimator under Strategy #3 had minimal mean squared error even under some small violations of MAR (see e.g., Figure 3a). Otherwise, if one is unsure whether MAR held, we recommend that one opt for Strategy #2 or #5 (we would rule out Strategy #4 due to the lack of associated inference tools). The nonparametric efficient estimator (Strategy #2) represents a good default choice, as it has favorable theoretical properties (see Theorem 2), and in all simulation scenarios is unbiased and its confidence intervals attained nominal coverage (see e.g., Table 2). For small or absent MAR violations, the adaptive estimator (Strategy #5) can be more efficient, but is less efficient than Strategy #1 for moderate violations (see e.g., Figure 3a). The methods we have discussed focus on the specific causal problem outlined in Section 2. That said, the nonparametric identification and estimation results are entirely generic, and do not depend on either the data structure of the given problem nor the specific mean counterfactual estimand of interest. In Appendix A of the Supporting Information, we lay out the notation for arbitrary coarsening [38] of a given full data structure and show that under a generalization of Assumptions 2 and 3, double sampling identifies the complete data distribution; derive a transformation of the full data nonparametric influence function of an arbitrary smooth functional that yields the observed data nonparametric influence function; construct influence function‐based estimators using sample splitting; and characterize the asymptotic behavior of these estimators, including multiple robustness properties. The general coarsening formulation is powerful, and justifies methodology—analogous to the approaches described in the main paper—that are applicable for many estimands in settings with more complicated missing data patterns, right‐censoring and/or truncation (e.g., as occurs often in survival analysis settings), and beyond. Table 1 emphasizes that each of the proposed estimators require Assumption 2 to hold. In any given applied setting, this assumption will need to be carefully evaluated. As indicated in Section 3.2, depending on the context, one practical issue is that selection by the double sampling scheme may not necessarily yield complete data. In such settings, investigators will need to work through the same thought experiments that one usually would for the standard MAR assumption (such as Assumption 2) to try to understand why some individuals engage and others do not. If it is felt that engagement remains dependent on outcome status (beyond what is explained by what is known about the design and covariates (A, L)), then sensitivity analyses or alternative identification schemes may be necessary; this is an on‐going area of our work. A second practical issue is that, even if all those selected actually engage, the data that arises from the double‐sampling scheme may be subject to error or recall bias [21]. In some settings, the potential for recall bias may be mitigated through the design. In Koffman et al. [22], for example, the outcome of interest was weight change at five years post‐surgery, so the investigators timed the invitation to participants to coincide with the five‐year anniversary. Additionally, following the same broad philosophy of this paper, one could directly learn about potential recall bias by including some participants for whom R = 1 in the double‐sampling scheme. This, in turn, would enable a comparison between information provided by the patient and what is available in the EHR. How best to do this and use the resulting information, though, are open questions. We also mention that, while we believe the simulations and case study illustrated in finite samples the theoretical properties we have developed for the various approaches, they were necessarily fairly limited in scope. A larger simulation study, comparing a wider variety of scenarios and component modeling strategies, will be a goal for future research. Notwithstanding these practical issues and the fact that logistical or financial considerations may altogether preclude the use of double sampling in some settings, we believe the proposed methodology presents new options for researchers as they contend with potentially informative missing or coarsened data. Beyond those mentioned above, there are many opportunities for future work in this vein, including how best to use the available information in the EHR when allocating resources for double‐sampling, as well as developing estimators for a broader set of analysis goals, such as mediation, and outcome types, such as time‐to‐event outcomes. Conflicts of Interest The authors declare no conflicts of interest. Supporting information Data S1. Supporting Information. Data S2. Supporting Information.