Efficient Estimation under Multiple Missing Patterns via Balancing Weights

Jianing Dong, Raymond K. W. Wong, Kwun Chuen Gary Chan · 2025

Read Efficient Estimation under Multiple Missing Patterns via Balancing Weights by Jianing Dong, Raymond K. W. Wong, Kwun Chuen Gary Chan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13467.txt.

Introduction

Jianing Dong, Raymond Wong, and Kwun Chuen Gary Chan tackle in 2025 estimation when data go missing in many tangled patterns. Real surveys rarely lose values in a tidy order, and the loss is often not random, which breaks standard fixes. The team builds on pattern graphs, diagrams that link the observed patterns in a sample.

Their method sets balancing weights step by step along the graph, avoiding the instability of a cruder local scheme. The resulting estimator reaches the theoretical efficiency bound for moment-based models, regressions included. Simulations favor the method, and real-data checks show the answers hold up under different assumptions about the missingness.

Catalog record

CREATORJianing Dong, Raymond K. W. Wong, Kwun Chuen Gary Chan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE150 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13467

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Estimation under Multiple Missing Patterns via Balancing Weights Jianing Dong, Raymond K. W. Wong, Kwun Chuen Gary Chan Abstract As one of the most commonly seen data challenges, missing data, in particular, multiple, non-monotone missing patterns, complicates estimation and inference due to the fact that missingness mechanisms are often not missing at random, and conventional methods cannot be applied. Pattern graphs have recently been proposed as a tool to systematically relate various observed patterns in the sample. We extend its scope to the estimation of parameters defined by moment equations, including common regression models, via solving weighted estimating equations with weights constructed using a sequential balancing approach. These novel weights are carefully crafted to address the instability issue of the straightforward approach based on local balancing. We derive the efficiency bound for the model parameters and show that our proposed method, albeit relatively simple, is asymptotically efficient. Simulation results demonstrate the superior performance of the proposed method, and real-data applications illustrate how the results are robust to the choice of identification assumptions. Keywords: Non-monotone missing; Missing not at random (MNAR); Covariate balancing; Missing Pattern; Pattern Mixture Model 1 Introduction Incomplete data is a prevalent issue in data analysis, arising in a variety of fields, including clinical trials, social sciences, and machine learning. Proper handling of missing data is crucial, as inappropriate assumptions or methods can lead to biased inferences and invalid conclusions. The theoretical framework for handling missing data was first formalized by Rubin (1976) , who categorized missingness mechanisms into three broad classes: Missing Completely at Random (MCAR), Missing at Random (MAR), and Missing Not at Random (MNAR). While MCAR assumes that missingness is entirely unrelated to the data, and MAR posits that missingness depends only on observed data, MNAR describes scenarios where the probability of missingness depends on unobserved variables. These definitions provide a foundation for understanding the relationship between observed and missing data, but practical applications often require more nuanced structures to model real-world missingness mechanisms. Although complete-case analysis excludes missing data from the dataset, offering a convenient approach, this straightforward technique works under the stringent MCAR assumption, which rarely holds in real-world data. The MAR assumption is often restrictive and may not align with real-world missing data scenarios, such as non-monotone missingness, where the missing data does not follow a structured sequence ( Robins, 1997 ; Troxel et al., 1998 ) . In general, MNAR is a more appropriate assumption than MAR. One real-world example is that high-income individuals are less likely to report their income due to privacy concerns, stigma, or social desirability bias. The missingness is directly related to the unobserved income values themselves. However, the MNAR assumption introduces significant challenges since missingness cannot be ignored when making inferences from incomplete data. Identifying assumption is required to specify which parameters in the full data model can be estimated despite the missing data. The pattern mixture model is a widely used method ( Little, 1993 ; Tchetgen et al., 2018 ) that classifies data based on missing patterns and imposes assumptions on the conditional densities of missing variables for specific patterns. The pattern graph introduced in Chen (2022) provides a visualization of more complex identifying assumptions, which hierarchically link the conditional densities of missing variables across multiple patterns. It is worth noting that pattern graphs are different from graphical models ( Mohan and Pearl, 2021 ; Nabi et al., 2020 ) and the casual graph ( Bhattacharya et al., 2020 ; Shpitser, 2016 ) . Imputation-based methods are commonly used with the pattern mixture model to estimate the parameters of interest, but they can be computationally intensive due to repeated iterations or samplings. Alternatively, identification conditions can often be stated in a selection model, leading to inverse probability weighting (IPW) estimation. However, they may lack stability due to extreme estimated weights. Balancing weights ( Zubizarreta, 2015 ; Wong and Chan, 2018 ; Dong et al., 2024 ) are attractive since they are designed to achieve a more balanced distribution of variables between groups, which can lead to a more stable and efficient estimation. In this paper, we extend the balancing approach to missing mechanisms that can be visualized by a pattern graph or its generalization. A local estimation encourages the covariate balance between patterns directly connected by edges when hierarchical structures exist in the graph, while it may lead to extreme weights due to model extrapolation or fail to account for the errors accumulated through the multiplications used to construct the inverse propensity weights. A sequential estimation procedure is proposed to address instability in the estimation. We expand the scope of estimation under pattern graphs to model equations that include common regression models. We study the semiparametric efficiency bound and show the consistency and asymptotic efficiency of the proposed estimator. 2 Missing data assumptions 2.1 Preliminaries In this section, we formally describe the setup of the problem. Let L=(L_{(1)},\ldots,L_{(d)})\in\prod_{j=1}^{d}\mathcal{L}_{(j)} , where \mathcal{L}_{(j)}\subseteq\mathbb{R} , be a vector of potentially observable random variables. To indicate the observation of these variables, let R=(R_{(1)},\ldots,R_{(d)})\in\{0,1\}^{d} be a binary random vector such that R_{(j)}=1 when L_{(j)} is observed. Let \mathcal{R}=\{r\in\{0,1\}^{d}:P(R=r)>0\} be the set of all possible missing patterns and M=|\mathcal{R}| be the number of missing patterns in the study. We define a partial ordering of missing pattern vectors: for two patterns s,r\in\mathcal{R} such that s\neq r , we say s>r if s_{(j)}\geq r_{(j)} for all 1\leq j\leq d . Denote the complete-case pattern by 1_{d}=(1,\ldots,1) . For each missing pattern r , we denote the observed variables by L^{[r]} and the missing variables by L^{[\overline{r}]} . For example, L^{[101]}=(L_{(1)},L_{(3)}) and L^{[\overline{101}]}=L_{(2)} . So, the observations are \{(L^{[R_{i}]}_{i},R_{i})\}_{i=1}^{N} . Denote \mathrm{dom}_{r}:=\prod_{\{j:R_{(j)}=1\}}\mathcal{L}_{(j)}\subseteq\mathbb{R}^{d_{r}} where d_{r} is the number of observed variables in pattern r , then L^{[r]}\in\mathrm{dom}_{r} and L^{[\overline{r}]}\in\prod_{\{j:R_{(j)}=0\}}\mathcal{L}_{(j)}\subseteq\mathbb{R}^{d-d_{r}} . Let \theta_{0}\in\mathbb{R}^{q} be the parameter of interest which is the unique solution to \mathbb{E}\{\psi_{\theta}(L)\}=0 , with a known vector-valued estimating function \psi_{\theta}(L)=\psi(L,\theta) that takes values in \mathbb{R}^{q} . For instance, we could use the quasi-likelihood estimating functions for the generalized linear models. If full data were observed, a solution to the estimating equations N^{-1}\sum_{i=1}^{N}\psi_{\theta}(L_{i})=0 is a common Z-estimator. However, \psi_{\theta}(L_{i}) can only be evaluated at samples with complete observations of L_{i} . When missing data is present, practitioners often solve the complete-case estimating equation N^{-1}\sum_{i=1}^{N}\mathsf{1}_{R_{i}=1_{d}}\psi_{\theta}(L_{i})=0 , but it is typically biased unless R and L are independent, i.e., missing completely at random. There are two directions to reconstruct the full data density and address the bias issues. Conditional density of missing variables. The joint density of L can be expressed as \displaystyle p(l)=\sum_{r\in\mathcal{R}}P(R=r)p(l^{[r]}\mid R=r)p(l^{[\overline{r}]}\mid l^{[r]},R=r)\ . Note that p(l^{[\overline{r}]}\mid l^{[r]},R=r) cannot be identified without assumptions since l^{[\overline{r}]} is never observed when R=r . Given assumptions such that estimators \hat{p}(l^{[\overline{r}]}\mid l^{[r]},R=r) for each pattern r are available, imputation can be performed repeatedly to generate multiple complete datasets, or the conditional density can be directly integrated into the analysis. Selection probability. The joint density of L can be expressed as p(l)=p(l,r)/P(R=r\mid l) . Using the selection probability, the population-level expectation can be reconstructed by weighting the complete cases: \displaystyle\mathbb{E}\{\psi_{\theta}(L)\}=\mathbb{E}\left\{\frac{\mathsf{1}_{R=1_{d}}}{P(R=1_{d}\mid L)}\psi_{\theta}(L)\right\}\ . (1) Given an estimator \hat{\pi}(l) for \pi(l)=P(R=1_{d}\mid l) , an estimator of \theta can be obtained by solving the weighted estimating equation: N^{-1}\sum_{i=1}^{N}\mathsf{1}_{R_{i}=1_{d}}\psi_{\theta}(L_{i})/\hat{\pi}(L_{i})=0 .

Read the complete text