Learning Causal Graphs at Scale: A Foundation Model Approach
Naiyu Yin, Tian Gao, Yue Yu · 2025
Read Learning Causal Graphs at Scale: A Foundation Model Approach by Naiyu Yin, Tian Gao, Yue Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18285.txt.
Introduction
Naiyu Yin, Tian Gao, and Yue Yu pre-train a foundation model that learns causal graphs. A causal graph maps which factors drive which others, with no loops allowed. Learning such graphs is costly and shaky when data is scarce. Their model, Attention-DAG, learns many linear cause models at once through an attention-based design.
Pre-training across tasks lets the model store shared structure as a prior. That prior steadies graph learning when a new task has few samples. On standard fake data sets, the model beat past methods on graph accuracy. It also answered new tasks at once, with no fresh training run. The authors call it the first practical pre-trained model built for graph learning of this kind.
Catalog record
| CREATOR | Naiyu Yin, Tian Gao, Yue Yu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-18285
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Causal Graphs at Scale: A Foundation Model Approach Naiyu Yin, Tian Gao, Yue Yu Abstract Due to its human-interpretability and invariance properties, Directed Acyclic Graph (DAG) has been a foundational tool across various areas of AI research, leading to significant advancements. However, DAG learning remains highly challenging, due to its super-exponential growth in computational cost and identifiability issues, particularly in small-sample regimes. To address these two challenges, in this work we leverage the recent success of linear transformers and develop a foundation model approach for discovering multiple order-consistent DAGs across tasks. In particular, we propose Attention-DAG (ADAG), a novel attention-mechanism-based architecture for learning multiple linear Structural Equation Models (SEMs). ADAG learns the mapping from observed data to both graph structure and parameters via a nonlinear attention-based kernel, enabling efficient multi-task estimation of the underlying linear SEMs. By formulating the learning process across multiple tasks as a continuous optimization problem, the pre-trained ADAG model captures the common structural properties as a shared low-dimensional prior, thereby reducing the ill-posedness of downstream DAG learning tasks in small-sample regimes. We evaluate our proposed approach on benchmark synthetic datasets and find that ADAG achieves substantial improvements in both DAG learning accuracy and zero-shot inference efficiency. To the best of our knowledge, this is the first practical approach for pre-training a foundation model specifically designed for DAG learning, representing a step toward more efficient and generalizable down-stream applications in causal discovery. 1 Introduction Causality plays a fundamental role in explaining the underlying mechanisms of systems in many scientific and decision-making domains, including computer science, medicine, biology, and economics ( Pearl et al. 2000 ; Sachs et al. 2005 ; Lu et al. 2021 ; Subbaswamy and Saria 2020 ) . This has led to significant interest within the machine learning community in developing advanced methods for causal discovery. A common approach to modeling causal relationships is to identify causal models among a set of random variables in the form of Directed Acyclic Graphs (DAGs), which offer a compact, interpretable, and theoretically grounded representation of the underlying data-generating process. Figure 1: The schematic of the proposed ADAG training and inference procedures. Given order-consistent data from multiple domains, we begin by training a nonlinear kernel map from data to the corresponding hidden DAG. The trained model is then capable of zero-shot DAG inference, allowing it to predict weighted adjacency matrices when given data from new test domains. Numerous well-established methods have been developed for DAG learning, relying on either combinatorial search, score-based methods, or continuous relaxations. However, learning DAGs from observational data remains highly challenging due to the super-exponential space of possible graph structures, inherent identifiability issues, and data scarcity in real-world applications. Moreover, most existing DAG learning approaches operate on a per-task basis, lacking the ability to generalize across tasks or domains. As a result, there is growing interest in developing foundation models that can transfer knowledge across causal tasks, yet little progress has been made in this direction due to the unique structural constraints and identifiability challenges inherent to DAGs. Inspired by the recent success of foundation models and their capacity to encode vast amounts of transferable information, we propose to address the high computational cost and poor performance in low-sample regimes by pre-training a DAG-learning foundation model that generalizes across tasks to infer order-consistent DAGs accurately and efficiently. Specifically, we introduce a novel attention-mechanism-based formulation for learning Structural Equation Models (SEMs). Although the proposed framework can be extended to more general models, for the purpose of demonstration, we focus primarily on the linear SEM setting in this work. As illustrated in Figure 1 , the key of our approach is to define a nonlinear kernel map using the attention blocks, which takes observational data as the input and the corresponding weighted adjacency matrices as the output. During the pre-training phase, the model is trained across multiple tasks, which each task treated as a DAG discovery problem (recovering a hidden DAG from a set of data observations). As such, the weighted adjacency matrix is inferred in an unsupervised way, and the model is capable to capture both the structural causal relationships and the associated causal mechanisms. The multi-task training and attention mechanism-based blocks were known to possess advantages in inferring shared low-dimensional structure across tasks and applying such a structure in the downstream test tasks ( Lu and Yu 2025 ) . Therefore, our proposed ADAG architecture is anticipated to reduce the ill-posedness of individual DAG estimation problems and improve recovery of both graph structure and parameters, especially in data-scarce settings. Furthermore, once trained, our model enables zero-shot inference on previously unseen tasks, offering significant improvements in both accuracy and efficiency for downstream causal discovery. Notably, our framework does not require ground truth graphs, as the DAG kernel map is learned automatically using a data reconstruction loss, which differs from many supervised DAG learning approaches ( Li et al. 2020 ) . To the best of our knowledge, this is the first work to propose a practical approach for pre-training a foundation model specifically for DAG learning. Existing efforts to integrate causality with foundation models have very different focuses, such as causal inference ( Zhang et al. 2024a ) or semantic information extraction from existing large language models ( Ban et al. 2023 ; Wan et al. 2024 ; Wu et al. 2024 ) , as they do not directly train foundation models to estimate DAG structures and underlying causal model parameters from observed data alone. While several prior works Chen et al. 2021 ; Lu and Gao 2023 ; Zhang et al. 2017 ; Zhou et al. 2022 have developed advanced algorithms for multi-task DAG learning, aiming to simultaneously estimate DAGs from either heterogeneous or order-consistent data. Those approaches primarily improve efficiency or accuracy through joint optimization across tasks. However, they do not generalize to unseen tasks without further optimization or training steps. In contrast, our approach supports zero-shot generalization to new tasks within heterogeneous and order-consistent data settings 1 1 1 In heterogeneous data settings, the data for new tasks are generated from the same DAG as the training data but with different causal mechanisms. In order-consistent data settings, the data for new tasks are generated from different DAGs and mechanisms, while sharing the same causal ordering as the training data. , without requiring access to ground truth DAGs during training. This sets the stage for a new direction in causal discovery: building scalable, pre-trained foundation models capable of generalizing structural knowledge across domains. Major Contributions. 1) We propose a novel attention-mechanism-based formulation for DAG learning, that learns a nonlinear kernel map from observational data to the underlying causal graph structure and associated parameters. 2) Using a pre-training procedure on multiple order-consistent DAG tasks, we obtain a foundation model for DAG learning. This model captures shared low-dimensional structures and enables zero-shot inference on unseen tasks within both heterogeneous and order-consistent data settings. 3) We empirically demonstrate that our method significantly outperforms both single-task and multi-task DAG learning baselines in terms of inference efficiency and accuracy, especially in small-sample regimes. 2 Related Works Attention Mechanism for Inverse Problems. In recent years, the transformer based on the attention mechanism has been increasingly adopted to tackle