A Knowledge-Informed Pretrained Model for Causal Discovery
Wenbo Xu, Yue He, Yunhai Wang, Xingxuan Zhang, Kun Kuang, Yueguo Chen, Peng Cui · 2026
Read A Knowledge-Informed Pretrained Model for Causal Discovery by Wenbo Xu, Yue He, Yunhai Wang, Xingxuan Zhang, Kun Kuang, Yueguo Chen, Peng Cui (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20842.txt.
Introduction
Xu and co-authors build a pretrained model for causal discovery that uses weak domain knowledge. Pure data driven tools lack guidance, while strong priors cost too much. Their model takes coarse prior knowledge as a middle path. A dual source encoder decoder reads observational data with the knowledge in mind. Pretraining spans varied graphs, densities, and variable counts. A curriculum adapts the model to priors of unlike strength. Tests on in distribution, shifted, and real data beat baselines. Robustness and practical value stood out in their results.
Catalog record
| CREATOR | Wenbo Xu, Yue He, Yunhai Wang, Xingxuan Zhang, Kun Kuang, Yueguo Chen, Peng Cui |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33648 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20842
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Knowledge-Informed Pretrained Model for Causal Discovery Wenbo Xu, Yue He, Yunhai Wang, Xingxuan Zhang, Kun Kuang, Yueguo Chen, Peng Cui Abstract Causal discovery has been widely studied, yet many existing methods rely on strong assumptions or fall into two extremes: either depending on costly interventional signals or partial ground truth as strong priors, or adopting purely data driven paradigms with limited guidance, which hinders practical deployment. Motivated by real-world scenarios where only coarse domain knowledge is available, we propose a knowledge-informed pretrained model for causal discovery that integrates weak prior knowledge as a principled middle ground. Our model adopts a dual source encoder-decoder architecture to process observational data in a knowledge-informed way. We design a diverse pretraining dataset and a curriculum learning strategy that smoothly adapts the model to varying prior strengths across mechanisms, graph densities, and variable scales. Extensive experiments on in-distribution, out-of distribution, and real-world datasets demonstrate consistent improvements over existing baselines, with strong robustness and practical applicability. † † footnotetext: Contact: Wenbo Xu ( [email protected] ) and Yue He ( [email protected] ). 1 Introduction Causal discovery aims to recover underlying dependencies from observational data and is fundamental to building reliable explanations, especially when randomized controlled trials (RCTs) are costly and ethically constrained. It has broad impact in domains such as industrial process control [ 1 ] , biological networks [ 2 ] , environmental systems [ 3 ] , and policy evaluation [ 4 ] , where understanding causal mechanisms is often more valuable than prediction alone. Structural Causal Models (SCMs) [ 5 ] provide a formal representation of causal relationships by encoding system mechanisms in a directed acyclic graph (DAG), where nodes denote variables and directed edges specify causal influence. This formalism naturally motivates causal discovery algorithms that estimate a DAG (and associated mechanisms) from observational data, seeking structures that are consistent with the observed distribution. Despite steady progress, causal discovery in real-world systems remains difficult due to heterogeneous mechanisms and noise. Existing methods rely on idealized assumptions and strong inductive biases that fail under mechanism mismatch [ 6 , 7 , 8 ] . Some try to incorporate prior knowledge through rigid forms that require explicit edge specification [ 9 ] . Though achieving significant improvement, this rigidity limits flexibility and applicability. Recent pretrained approaches improve inference efficiency but still leave room for improvement: they either rely on costly intervention signals [ 10 , 11 ] or operate purely data-driven [ 12 ] , leaving a gap between precise priors and pure observations. Figure 1: Simple causality in chemistry. (A) Expert knowledge is structured as graph. (B) The corresponding ground-truth causal graph. Real-world knowledge is often available in a weaker form, while remaining informative. As illustrated in Fig. 1 (A), an expert may distill experience into knowledge that both encourages (dashed lines) and rejects candidate relations. Importantly, encouraged relations may be direct causal effects, e.g., Feeding Ratio \rightarrow Output Density , or indirect one, e.g., Feeding Ratio \rightarrow Reaction Time in Fig. 1 (B). Both could provide heuristic guidance in causal discovery. We will provide a more formalized definition in Section 3.3 . To fully leverage such knowledge, we propose Kode , a knowledge-informed pretrained model for causal discovery. To the best of our knowledge, we propose the first pretrained model for causal discovery that jointly leverages observational data and prior knowledge. Kode introduces a dual-source encoder-decoder structure that preserves the distinct semantics of data and knowledge while enabling cooperative inference. To train such a model, we build a diverse synthetic training dataset spanning heterogeneous mechanisms, graph densities, and variable scales, together with a curriculum strategy that smoothly schedules prior strength. Extensive evaluations including in-/out-of-distribution, and real-world dataset demonstrate consistent improvements over strong baselines, even without any knowledge input. In summary, we have four contributions: • We introduce the first knowledge-informed pretrained model for causal discovery, establishing a new practical paradigm for real applications. • We propose a dual-source encoder-decoder architecture that enables synergistic inference through semantic interaction between data and knowledge. • We construct a diverse training corpus and implement a curriculum learning strategy to ensure seamless adaptation across varying data distributions and prior strengths. • We provide extensive evaluations demonstrating that our method consistently outperforms baselines across diverse scales, mechanisms, and sparsity levels. 2 Related Work Under the framework of Structural Causal Models (SCMs) [ 5 ] , a large body of work has been developed for causal discovery. Existing approaches can be broadly categorized into three types [ 13 ] . Constraint-based approaches [ 14 ] infer causal relations via conditional independence tests. Function-based methods achieve identifiability by imposing explicit assumptions on generating mechanisms [ 7 , 15 ] . Score-based ones [ 16 ] apply a specific score function to guide structure searching. More recently, NOTEARS [ 8 , 17 ] introduces a differentiable score for DAG and has inspired a series of extensions [ 18 , 19 , 20 , 21 ] . Despite their empirical success, these methods introduce distinct forms of inductive bias. Traditional methods build on specific assumptions, while NOTEARS imposes additive equal-variance noise implicitly [ 22 ] . To improve practical usability, prior knowledge is incorporated in implementation. It’s either used to reduce the search space [ 9 ] , optimization space [ 23 , 24 ] or as reward function [ 21 , 25 ] . And it’s difficult to encode heuristic or uncertain domain knowledge, which is common in practice. In recent years, Transformers have shown promising potential. TabPFN [ 26 ] demonstrates its sensitivity to causality. Meanwhile, attention-based approaches for causal discovery have emerged. They are pretrained on a synthetic dataset and infer via a single forward pass. Representative methods include AVICI [ 11 ] , CSIvA [ 10 ] , and BCNP [ 12 ] , which share similar encoding paradigms but adopt distinct decoder designs with different modeling emphases. However, these methods tend to fall into two extremes: either relies on interventional signals which are often costly in practice or follows a purely data-driven paradigm without any prior. To bridge this gap, we propose a knowledge-informed pretrained model for causal discovery. 3 Preliminary Study To develop a solution for the aforementioned difficulties, we first provide formal definitions for causal discovery and knowledge encoding with corresponding motivation. 3.1 Problem of Causal Discovery Causal discovery is commonly framed as learning a directed acyclic graph (DAG) \mathbf{G}=(\mathbf{V},\mathbf{E}) that represents causal relations among variables \mathbf{X}=\{X_{1},\dots,X_{d}\} , where each node corresponds to a variable and directed edges encode causal influences. Under this formulation, causal discovery acts as a posterior estimator that infers the distribution over graph structures given observational data, i.e., p(\mathbf{{G}}\mid\mathbf{X}) . 3.2 Motivation As illustrated in Fig. 1 , real-world knowledge is often available in a coarse-grained yet easily attainable form, while remaining informative. Experts can readily provide high-level structural cues, such as procedural order or feasibility constraints, without specifying exact causal mechanisms. We abstract such information as an optional prior graph G_{P} . When available, it provides auxiliary heuristic guidance for posterior inference in causal discovery p(\mathbf{{G}}\mid\mathbf{X},G_{P}) . This view motivates the development of knowledge-informed models that can flexibly incorporate G_{P} without over-reliance. Unlike prior approaches that utilize partial ground-truth DAGs as