Data Augmentation via Causal-Residual Bootstrapping
Mateusz Gajewski, Sophia Xiao, Bijan Mazaheri · 2026
Read Data Augmentation via Causal-Residual Bootstrapping by Mateusz Gajewski, Sophia Xiao, Bijan Mazaheri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15335.txt.
Introduction
Gajewski and co-authors augment data by bootstrapping causal residuals. Standard augmentation edits data points with domain know-how, like tinting an image. Causal knowledge, such as A causes B causes C, can guide richer edits. Their method fits models on marginal distributions, then permutes the residuals between points. This respects the rule that separate mechanisms act on their own. Models trained on the augmented data predict more accurately. Theory backs the gain in linear Gaussian settings, and the approach uses facts beyond a Markov equivalence class.
Catalog record
| CREATOR | Mateusz Gajewski, Sophia Xiao, Bijan Mazaheri |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 112862 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15335
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Data Augmentation via Causal-Residual Bootstrapping Mateusz Gajewski, Sophia Xiao, Bijan Mazaheri Abstract Data augmentation integrates domain knowledge into a dataset by making domain-informed modifications to existing data points. For example, image data can be augmented by duplicating images in different tints or orientations, thereby incorporating the knowledge that images may vary in these dimensions. Recent work by Teshima and Sugiyama has explored the integration of causal knowledge (e.g, A causes B causes C) up to conditional independence equivalence. We suggest a related approach for settings with additive noise that can incorporate information beyond a Markov equivalence class. The approach, built on the principle of independent mechanisms, permutes the residuals of models built on marginal probability distributions. Predictive models built on our augmented data demonstrate improved accuracy, for which we provide theoretical backing in linear Gaussian settings. 1 Introduction Large, comprehensive datasets have driven a meteoric rise in the scale and capabilities of machine learning systems. Nevertheless, data-sparse regimes remain, particularly in human health and science, where financial and ethical barriers prohibit extensive data collection. By adding noise or anticipated transformations, data augmentation creates new training data points, thereby enhancing the power of small datasets [ 46 ] . Such augmentation strategies can be thought of as incorporating prior knowledge as new data. For example, an augmented image dataset may account for known variations by varying lighting conditions and performing rotations [ 45 ] . Causal information is often represented in the form of a structural causal model (SCM), which is known to imply conditional independencies according to d-separation rules [ 29 ] . For example, consider a causal chain with A causes B and B causes C , and no other causal dependencies. Such a system is represented using a directed acyclic graph A\rightarrow B\rightarrow C , which is constrained by a Markov property of A\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}C\>|\>B . Causal structures or conditional independencies are often known from prior experiments or by design. For example, randomized controlled trials (which are often small) are known to assign a randomized treatment ( T ) that contains no arrows to or from other attributes of the participants ( \mathbf{X} ). Such knowledge tells us that \mathbf{X}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}T without the need for statistical testing. In other settings, causal structures can be learned using a process known as causal discovery [ 40 ] . Recently, foundation models based on the prior–data fitted network (PFN) paradigm have gained traction in the ML community. This includes TabPFN [ 27 , 15 ] , which was pre-trained on synthetic datasets generated using random structural causal models and Bayesian networks. Surprisingly, models pre-trained on synthetic data perform well on real-world datasets, seemingly benefiting from information that a causal structure exists , even though the exact causal structure of the eventual real-world data was not necessarily provided. This suggests that structural priors capture complex independence relations in real-world data [ 17 ] . Seminal work by 42 has explored incorporating causal information into data augmentation. In particular, they noted that conditional independencies allow entries of a dataset to be permuted. For example, if A\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}B , then any observed value of A may be paired with any observed value of B . These permutations augment the data, e.g., a two-point set (0,0),(1,1) may be augmented to (0,0),(1,1),(0,1),(1,0) . Their approach shows moderate improvement in prediction accuracy. The approach by 42 is limited in two ways. First, their approach requires reweighting data according to estimates of probability density, which thereby requires tuning the radius of a RBF kernel [ 32 ] . Reweighting based on density was criticized by 24 for its dependence on tuning and general instability. Second, their approach is limited to augmenting conditional independence properties, which cannot account for causal direction. As such, the entire directed acyclic graph (DAG) cannot be utilized. Conditional independence information alone cannot fully specify a causal structure. For example, A\rightarrow B\rightarrow C , A\leftarrow B\leftarrow C , and A\leftarrow B\rightarrow C display the same set of conditional independence properties, known as a Markov equivalence class (MEC), which is represented using a “completed partial DAG” (CP-DAG) when there are no latent variables, and an acyclic directed mixed graph (ADMG) in the presence of latent confounding [ 39 ] . Literature on causal discovery has shown that systems with nonlinear dependencies and Gaussian additive noise contain additional information beyond the conditional independence properties that specify an MEC [ 16 , 31 ] . In particular, nonlinear relationships shift Gaussian distributions to non-Gaussian distributions, yielding Gaussian residuals in the causal direction and non-Gaussian residuals in the anti-causal direction. The goal of this paper is to address the limitation of 42 by incorporating information from the full causal structure , including causal directions. We do this using the principle of independent mechanisms, which states that if X causes Y , then the mechanism that turns X into Y (i.e., \Pr(Y|X) ) does not depend on how X is distributed ( \Pr(X) ) [ 30 ] . As such, the distribution of residuals of the effect Y is independent from the value of the cause X . 1.1 Summary of Contributions We begin by motivating the incorporation of causal structure via data augmentation, rigorously demonstrating its ability to improve downstream regression tasks. Section 3 provides a theoretical justification for this claim, showing that the improvement in regression error scales proportionally to the inverse of the starting dataset size (smaller datasets are improved more). Motivated by these theoretical results, we build on the work of 42 to develop a more broadly-applicable causal data augmentation approach, called “Causal-Residual Bootstrapping” (CRB). CRB is given in Section 4 , utilizing the principle of independent mechanisms. The key observation is that predictions of variables from their direct causes should exhibit residuals that are independent from their predecessors’ values. As such, the residuals may be permuted, allowing us to “bootstrap” the noise term and generate additional data. CRB satisfies two goals: it may preserve existing structural information by augmenting data based on the output of a causal discovery algorithm, or it may help inject new structural information into a dataset by leveraging a known set of causal dependencies. To emphasize the need for new methods like CRB, we demonstrate that genAI-based augmentation not only fails to capture causal structure but also contributes to its decline. Section 5 shows that VAEs, GANs, and diffusion models all weaken the performance of causal discovery algorithms run on their augmented datasets. Meanwhile, CRB and the approach by 42 preserve information about causal structure, as evidenced by unchanging performance in causal discovery. We further test both of these causally-informed augmentation methods on linear models with non-Gaussian noise, which exhibit additional information beyond their MECs. In these experiments, the direct LinGAM algorithm [ 36 ] does not perform well on data augmented by 42 ’s approach, but retains performance for CRB. This demonstrates the added value of augmenting according to the full causal structure. Since many datasets do not satisfy the linear restrictions of our theoretical results, Section 6 provides an empirical study of CRB’s improvements to downstream regression tasks.