Improving TabPFN’s Synthetic Data Generation by Integrating Causal Structure

Davide Tugnoli, Andrea De Lorenzo, Marco Virgolin, Giovanni Cinà · 2026

Read Improving TabPFN’s Synthetic Data Generation by Integrating Causal Structure by Davide Tugnoli, Andrea De Lorenzo, Marco Virgolin, Giovanni Cinà (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10254.txt.

Introduction

Davide Tugnoli and coauthors improve synthetic table generation by TabPFN by feeding in causal structure. TabPFN builds features one by one in input order, so a bad order that fights the true causal links creates spurious correlations. Their DAG aware conditioning samples each variable given its causal parents, with a partially directed variant for partial knowledge. Across controlled tests and CSuite data sets, the approach improves fidelity and preserves treatment effects better.

Catalog record

CREATORDavide Tugnoli, Andrea De Lorenzo, Marco Virgolin, Giovanni Cinà
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61445
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10254

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving TabPFN’s Synthetic Data Generation by Integrating Causal Structure Davide Tugnoli, Andrea De Lorenzo, Marco Virgolin, Giovanni Cinà Abstract Synthetic tabular data generation addresses data scarcity and privacy constraints in a variety of domains. Tabular Prior-Data Fitted Network (TabPFN) , a recent foundation model for tabular data, has been shown capable of generating high-quality synthetic tabular data. However, TabPFN is autoregressive: features are generated sequentially by conditioning on the previous ones, depending on the order in which they appear in the input data. We demonstrate that when the feature order conflicts with causal structure, the model produces spurious correlations that impair its ability to generate synthetic data and preserve causal effects. We address this limitation by integrating causal structure into TabPFN ’s generation process through two complementary approaches: Directed Acyclic Graph (DAG) -aware conditioning, which samples each variable given its causal parents, and a Completed Partially Directed Acyclic Graph (CPDAG) -based strategy for scenarios with partial causal knowledge. We evaluate these approaches on controlled benchmarks and six CSuite datasets, assessing structural fidelity, distributional alignment, privacy preservation, and Average Treatment Effect (ATE) preservation. Across most settings, DAG -aware conditioning improves the quality and stability of synthetic data relative to vanilla TabPFN . The CPDAG -based strategy shows moderate improvements, with effectiveness depending on the number of oriented edges. These results indicate that injecting causal structure into autoregressive generation enhances the reliability of synthetic tabular data. 1 Introduction Synthetic tabular data generation is becoming an important approach to handle data scarcity and privacy concerns in many domains [ 23 ] . This is especially relevant in fields such as healthcare, finance, and policy research, where privacy regulations and ethical constraints often limit access to real data. In pharmaceutical research, for example, synthetic data can be used to simulate drug effects for safety and efficacy, while protecting patient confidentiality. For tabular data, generation methods must avoid producing close copies of training samples, especially when data is limited. However, generating reliable synthetic tabular data is challenging because real datasets usually contain complex causal relations among variables. Generation methods that ignore such dependencies may create spurious correlations that differ from the true data-generating process. When synthetic data are used to augment limited real data or simulate clinical scenarios, misleading dependencies can propagate to treatment effect estimates. In drug development, for instance, inaccurate estimation of treatment effects from flawed synthetic data could lead to costly trials on ineffective drugs or the exclusion of promising ones. Recent foundation models for tabular data, such as Tabular Prior-Data Fitted Network (TabPFN) [ 10 ] , have shown promising results by pre-training on millions of synthetic datasets derived from Structural Causal Models (SCMs) [ 17 ] . Pre-training on such diverse datasets is particularly valuable when real data are limited, a common scenario in domains like healthcare. This advantage becomes even more important for synthetic data generation, which requires predicting all features rather than a single target variable as in traditional classification or regression tasks. Extensions of TabPFN 1 1 1 https://github.com/PriorLabs/tabpfn-extensions enable unsupervised, autoregressive generation of synthetic tabular data. However, TabPFN ’s autoregressive approach generates variables sequentially without explicitly accounting for causal structure. Although the model averages predictions over random permutations of the conditioning set, this mitigates but does not eliminate order sensitivity: during generation, each variable can only condition on features that appear earlier in the sequence. When the generation order conflicts with causal dependencies, particularly in the presence of colliders, the model may introduce spurious correlations that compromise the reliability of synthetic data. This work studies the autoregressive nature of TabPFN -based synthetic data generation, identifies a fundamental limitation, and proposes methods to address it. Our contributions are as follows: 1. We demonstrate that TabPFN synthetic data quality depends heavily on the order of the features in the input data due to the absence of causal reasoning, with this sensitivity persisting even at large training sizes. 2. We propose causal conditioning strategies that leverage known or discovered causal structure, showing improvements for both full Directed Acyclic Graph (DAG) knowledge and partial Completed Partially Directed Acyclic Graph (CPDAG) information, when sufficiently oriented, across distributional quality metrics. 3. We quantify how synthetic data errors propagate to treatment effect preservation, demonstrating substantial misestimation that could lead to incorrect decisions in applications such as drug development. All code, datasets, and detailed numerical results are publicly available. 2 2 2 https://github.com/DavideTugnoli/tabpfn-causal-synthetic 2 Related Work Generative approaches to synthetic tabular data range from classical statistical methods to deep models based on Generative Adversarial Networks (GANs) , Variational Autoencoders (VAEs) , diffusion models, and autoregressive architectures [ 23 ] . Early deep generative models adapted architectures from image generation. TGAN [ 28 ] applied GANs to mixed continuous–categorical tables using Long Short-Term Memory-based generation, while CTGAN [ 27 ] addressed class imbalance and multimodal distributions. CTAB-GAN [ 31 ] further improved semantic fidelity. More recently, diffusion models have achieved strong performance [ 11 , 30 , 22 ] . While effective for statistical fidelity, these methods generate all features simultaneously without explicit causal structure. Autoregressive approaches factorize the joint distribution into sequential conditionals p(x_{j}\mid x_{C(j)}) , where C(j) denotes the conditioning set. Early work used Classification And Regression Trees (CART) to generate features sequentially [ 18 ] . GReaT treats tabular rows as text sequences for language model generation [ 2 ] . TabularARGN [ 24 ] randomizes feature orderings during training to learn conditionals across different subsets. Among foundation models, TabPFN [ 9 , 10 ] achieves state-of-the-art performance on small datasets via in-context learning. Pre-trained on millions of synthetic datasets sampled from SCMs , it acquired strong inductive biases without requiring retraining on target data. TabPFGen extends TabPFN to generative tasks via energy-based sampling rather than autoregressive conditioning [ 14 ] , but neither approach explicitly conditions on causal graphs during generation. Causal approaches integrate structural knowledge directly into generation. DECAF embeds SCMs into the architecture for interventional debiasing [ 25 ] . Causal-TGAN structures its generator according to a causal graph [ 26 ] . DATGAN incorporates expert DAGs to enforce structural dependencies [ 13 ] . CA-GAN combines causal discovery with reinforcement learning [ 16 ] . Unlike TabPFN, these models are trained from scratch per dataset and explicitly use DAG structure to guide generation. Recent work has applied transformer-based meta-learning to causal effect estimation. Do-PFN [ 21 ] adapts the PFN framework to predict conditional interventional distributions from observational data, taking covariates and treatment as input. CausalPFN [ 1 ] trains a transformer on a causal prior to estimate expected potential outcomes, also conditioning on covariates and treatment. Both models predict outcomes under intervention but do not model the full joint distribution over all variables. MACE-TNP [ 5 ] addresses a related task, estimating Bayesian model-averaged interventional distributions from observational data via neural processes; it similarly targets interventional queries rather than joint density modeling. To our knowledge, this is the first work to combine a foundation model with explicit causal structure for synthetic tabular data generation, addressing autoregressive ordering violations and collider bias through DAG-aware and CPDAG-based conditioning strategies. 3 Methodology 3.1 TabPFN Causal Extensions TabPFN is a transformer-based model pre-trained on approximately 130 million synthetic datasets, each sampled

Read the complete text