Missing data and cluster graphs: cluster-level missingness vs variable-level missingness

Willow Scott, Eugenio Valdano, Charles K. Assaad · 2026

Read Missing data and cluster graphs: cluster-level missingness vs variable-level missingness by Willow Scott, Eugenio Valdano, Charles K. Assaad (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20943.txt.

Introduction

Willow Scott, Eugenio Valdano, and Charles Assaad study missing data and cluster graphs, contrasting cluster-level and variable-level missingness. Missing data is pervasive in public health, environmental science, and social sciences. Recoverability is usually studied with fully specified variable-level models, despite applications often offering only coarse structural information, for instance when variables group into clusters for limited knowledge or interpretability reasons. The authors introduce two cluster-based missingness graph classes, m-C-DMG and cm-C-DMG, formalizing compatibility between abstract graphs and underlying variable-level models.

Catalog record

CREATORWillow Scott, Eugenio Valdano, Charles K. Assaad
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75533
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20943

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Missing data and cluster graphs: cluster-level missingness vs variable-level missingness Willow Scott, Eugenio Valdano, Charles K. Assaad Abstract Missing data is pervasive in many scientific domains such as public health, environmental science, and the social sciences. Recoverability from missing data is typically studied using fully specified variable-level missingness models despite that, in many applications, only coarse structural information is available, for instance when variables are grouped into clusters due to limited knowledge or interpretability reasons. In this paper, we investigate recoverability from such abstract representations. We introduce two classes of cluster-based missingness graphs: the m-C-DMG, which retains variable-specific missingness indicators, and the cm-C-DMG, which aggregates missingness mechanisms at the cluster level. We formalize the notion of compatibility between these abstract graphs and underlying variable-level missingness models, and study how this abstraction affects the recoverability of probabilistic and causal queries. In particular, we give graphical conditions of recovering the joint distribution as well as graphical conditions of recovering a macro causal effect. Overall, our results clarify when cluster-level missingness information is sufficient for valid inference, and when finer-grained modeling is necessary. 1 Introduction Missing data are ubiquitous in many scientific domains, including public health, environmental science, and the social sciences. Understanding when meaningful quantities can be recovered from incomplete data is therefore a fundamental problem. Classical missing data theory addresses this question at the level of fully specified variable-level models, where the missingness mechanism of each variable is explicitly represented 12 ; 7 ; 8 ; 9 . In such settings, graphical models ( 10 ) provide a powerful framework to characterize recoverability of joint, conditional, and causal quantities. In many modern applications, however, the underlying data-generating mechanisms are only partially known or are intentionally abstracted ( 11 ; 6 ; 4 ; 1 ) . Variables are thus often grouped into clusters, either for interpretability or because only aggregated structural information is available. For example, clusters are necessary to deal with uncertainty around the relationships between sexual practices and use of prevention in studies on predictors of risk of acquiring Human Immunodeficiency Virus (HIV). In a conceptual graph, sexual behavior is treated as a cluster of measures such as condom use and number of sexual partners. Table 1 exhibits missingness in a synthetic dataset of clusters representing Sexual Behaviors, Mental Health and HIV Status. As is the case in most real-world data, missingness may occur in any of the variables and for a variety of reasons. One must thus reason about recoverability from abstract representations of missingness mechanisms rather than from fully specified variable-level graphs. In this paper, we introduce and study two such abstractions: the m-C-DMG and the cm-C-DMG . The m-C-DMG retains variable-specific missingness indicators while representing causal relations between variables at the cluster level. The cm-C-DMG is a coarser abstraction that merges missingness indicators within each cluster into a single vertex. While both representations encode missingness mechanisms at a higher level of abstraction, they differ in the amount of information they preserve, raising the question of how this impacts recoverability. Our goal is to understand how recoverability of queries – such as joint distributions and causal effects – depends on the level of abstraction. We formalize the notion of compatibility between abstract graphs and variable-level missingness models and study when recoverability can be inferred from these abstractions. We provide results for the recovery of joint distributions and as well as results for the recovery of specific type of causal effects, known as macro causal effects. Overall, this work contributes to the theoretical understanding of missing data problems under structural abstraction. The remainder of the paper is organized as follows. Section 2 presents the background and preliminaries. Section 3 introduces the two proposed abstractions, which differ in their level of granularity, and compares them. Section 4 provides conditions for recovering joint distributions from these abstractions. Section 5 gives conditions for recovering macro causal effects. Finally, Section 6 concludes the paper with a discussion. Table 1 : Example of synthetic individual-level public-health data with missing values. Only the cluster-level graph between HIV status, mental health, and sexual behavior is assumed to be known, since the relationships within each cluster are unspecified. ID Anxiety scale Depression scale Condom use # of partners HIV status 1 Moderate None NA 3 NA 2 None Minimal Never 1 NA 3 None Severe NA 5 Negative 4 Minimal None Almost always 21 Acute 5 Severe Minimal Sometimes NA Negative 6 None None Always 1 Chronic 2 Background and preliminaries Consider a structural causal model ( 10 ) that induces a causal graph, also known as an acyclic directed mixed graph (ADMG) and a distribution of interest \Pr(\mathbb{V}) over a set of variables \mathbb{V} . Let \mathbb{V} be partitioned into \mathbb{V}_{o} and \mathbb{V}_{m} , where \mathbb{V}_{o}\subseteq\mathbb{V} denotes the set of variables observed for all records in the population, and \mathbb{V}_{m}\subseteq\mathbb{V} denotes the set of variables missing in at least one record. We refer to \mathbb{V}_{m}\cup\mathbb{V}_{o} as the set of substantive variables. A variable V_{i} is said to be fully observed if V_{i}\in\mathbb{V}_{o} , and partially observed if V_{i}\in\mathbb{V}_{m} . For each partially observed variable V_{i}\in\mathbb{V}_{m} , we associate two additional variables: a missingness indicator R_{V_{i}} and a proxy variable V_{i}^{\ast} . The intended semantics is that R_{V_{i}} indicates the missingness status of V_{i} in the dataset, and is what we actually observe of V_{i} in the available dataset, i.e. , V_{i}^{\ast}=\begin{cases}V_{i}&\text{if }R_{V_{i}}=0,\\ NA&\text{if }R_{V_{i}}=1.\end{cases} (1) In what follows we call \Pr(\mathbb{V}^{o},\mathbb{V}^{*},R) the manifest distribution. It represent the distribution of what we observe in addition to the missingness indicator variables. Missingness is typically categorized depending on the process that causes it ( 12 ) : • Missing Completely at Random (MCAR): the probability of missingness in V_{i} is completely independent of both the value of V_{i} and all other variables in the dataset: R_{V_{i}}{\rotatebox[origin]{90.0}{$\models$}}(V_{o}\cup V_{m}) . For example, a folder from a file cabinet organized in alphabetical order is lost. • Missing at Random (MAR): the probability of missingness may depend on observed variables, but not on the missing value itself: R_{V_{i}}{\rotatebox[origin]{90.0}{$\models$}}V_{m}\mid V_{o} . For example, a person is less likely to disclose their condom use if they have a sexual partner living with HIV. • Missing Not at Random (MNAR): the probability of missingness depends on unobserved data. For example, not using condoms decreases the likelihood of being willing to share one’s own test results for sexually transmitted infections, and condom usage is not observed. Given a manifest distribution \Pr(\mathbb{V}^{*},\mathbb{V}^{o},\mathbb{R}) , a query Q is said to be recoverable if we can compute a consistent estimate of Q as if no data were missing. i.e. , if it is decomposable into terms that contain the missingness mechanism R_{V_{i}}=1 of every partially observed variable V_{i} that appears in Q ( 7 ) . For MCAR data, all V_{i}\in\mathbb{V}_{m} , R_{V_{i}}{\rotatebox[origin]{90.0}{$\models$}}(\mathbb{V}_{o}\cup\mathbb{V}_{m}) . Therefore, the joint probability is recoverable trivially: \Pr(\mathbb{V})=\Pr(\mathbb{V}\mid R_{\mathbb{V}_{m}}=0)=\Pr(\mathbb{V}_{o},{\mathbb{V}_{m}^{\ast}}\mid{R_{\mathbb{V}_{m}}}=0) . Since conditional on {R_{V_{i}}}=0 , V_{i}^{\ast} is observed, considering only complete cases (listwise deletion) allows recovery of the joint probability \Pr(V_{i}) . For MAR data, all V_{i}\in\mathbb{V}_{m} , R_{V_{i}}{\rotatebox[origin]{90.0}{$\models$}}\mathbb{V}_{m}\mid\mathbb{V}_{o} . Thus, \Pr(\mathbb{V})=\Pr(\mathbb{V}_{m}\mid\mathbb{V}_{o})\Pr(\mathbb{V}_{o})=\Pr(\mathbb{V}_{m}\mid\mathbb{V}_{o},R_{\mathbb{V}_{m}}=0)\Pr(\mathbb{V}_{o})=\Pr(\mathbb{V}_{m}^{*}\mid\mathbb{V}_{o},R_{\mathbb{V}_{m}}=0)\Pr(\mathbb{V}_{o}) ( 7 ) , making it possible to estimate the distribution of \Pr(\mathbb{V}) . Recoverability is more complicated under MNAR, leading many to

Read the complete text