Transforming Causality: Transformer-Based Temporal Causal Discovery with Prior Knowledge Integration

Jihua Huang, Yi Yao, Ajay Divakaran · 2025

Read Transforming Causality: Transformer-Based Temporal Causal Discovery with Prior Knowledge Integration by Jihua Huang, Yi Yao, Ajay Divakaran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15928.txt.

Introduction

Jihua Huang, Yi Yao, and Ajay Divakaran introduce, in a 2025 paper, a transformer based method for discovering causal links in time series data. Finding which variables drive which others, and with what delay, is hard when relationships bend and fake correlations abound. Time order helps, but classic tools struggle with long range, nonlinear patterns.

Their framework first trains a multi layer transformer to forecast the series. Gradients from that forecaster then reveal the causal structure and the lag on each link, which assemble into a causal graph. Spurious links are curbed by prior knowledge: links a user rules out are masked in the attention at every layer, so the model can never use them. In tests, the method beat leading rivals, gaining about 13 percent on the discovery score. Its lag estimates were right nearly 99 percent of the time.

Catalog record

CREATORJihua Huang, Yi Yao, Ajay Divakaran
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15928

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Transforming Causality: Transformer-Based Temporal Causal Discovery with Prior Knowledge Integration Jihua Huang, Yi Yao, Ajay Divakaran Abstract We introduce a novel framework for temporal causal discovery and inference that addresses two key challenges: complex nonlinear dependencies and spurious correlations. Our approach employs a multi-layer Transformer-based time-series forecaster to capture long-range, nonlinear temporal relationships among variables. After training, we extract the underlying causal structure and associated time lags from the forecaster using gradient-based analysis, enabling the construction of a causal graph. To mitigate the impact of spurious causal relationships, we introduce a prior knowledge integration mechanism based on attention masking, which consistently enforces user-excluded causal links across multiple Transformer layers. Extensive experiments show that our method significantly outperforms other state-of-the-art approaches, achieving a 12.8% improvement in F1-score for causal discovery and 98.9% accuracy in estimating causal lags. Keywords Causal Discovery \cdot Causal Inference \cdot Time-series 1 Introduction Understanding the causal mechanisms governing dynamic systems is essential for forecasting, decision-making, and scientific discovery ( Prosperi et al. 2020 ; Runge et al. 2019a ; Seth et al. 2015 ) . In many real-world domains, such as healthcare, finance, climate science, and neuroscience, data are usually represented as multivariate time series, where variables influence each other over time. In this setting, temporal causal discovery aims to uncover not only which variables influence each other, but also when those influences occur. Going further, causal inference enables reasoning under interventions – asking what would happen if certain variables were manipulated. These tasks are foundational for interpretability and decision support in dynamic systems. Despite their importance, temporal causal discovery and inference remain challenging due to nonlinear relationships, hidden confounders, lagged effects, and spurious correlations. Classical approaches such as constraint-based methods ( Spirtes et al. 2000 ; Runge et al. 2019b ) , score-based optimization ( Zheng et al. 2018 ; Pamfil et al. 2020 ) , and Granger causality ( Granger 1969 ) often rely on restrictive assumptions and do not scale well in high-dimensional or nonlinear settings. To address these limitations, neural methods such as cMLP and cLSTM ( Tank et al. 2021 ) , TCDF ( Nauta et al. 2019 ) , DVGNN ( Liang et al. 2024 ) , and CUTS ( Cheng et al. 2023 ) leverage deep architectures to improve modeling expressiveness. However, these models are typically built on recurrent or convolutional structures, which are limited in capturing long-range dependencies and global variable interactions. In contrast, Transformer architectures, with their self-attention mechanisms, offer superior modeling of complex temporal structures and global dependencies. Transformer-based forecasting models like Informer ( Zhou et al. 2021 ) and Autoformer ( Wu et al. 2021 ) demonstrate strong performance in long-term time-series prediction, but they do not explicitly reason about causality and therefore cannot support interpretable or counterfactual analysis. The most directly related method, CausalFormer ( Kong et al. 2024 ) , introduces a Transformer-based approach for causal structure learning. However, it is restricted to single-layer architecture, limiting its ability to capture complex nonlinear causal dependencies or accommodate very long temporal sequences. In this work, we propose a novel framework for temporal causal discovery and inference, composed of a causality-aware time-series forecaster and an architecture-agnostic causal graph extractor. The forecaster employs a multi-layer Transformer-based architecture to capture complex nonlinear and long-range temporal dependencies among time-series variables during training. The multi-layer design further enables the model to capture multi-resolution temporal features, enhancing its ability to represent both short- and long-term causal influences. Once these causal relationships are implicitly encoded within the forecaster, the causal graph extractor identifies causal links and their corresponding time lags by analyzing the model’s gradients, ultimately constructing a directed causal graph. To integrate human domain knowledge with the learned causal structure, we introduce an attention masking based prior knowledge integration mechanism, which enforces user-specified exclusions of causal links consistently across all Transformer layers. This enables a human-in-the-loop refinement process, allowing domain experts to iteratively prune spurious causal relationships and improve the reliability of the resulting model for causal inference. In summary, our main contributions are as follows: • We propose a novel framework for temporal causal discovery and inference that employs a multi-layer Transformer architecture to model complex nonlinear relationships and long-range temporal dependencies in time-series data. • We design an architecture-agnostic, gradient-based module for causal graph extraction that interprets a trained time-series forecaster to recover causal graphs and associated time lags. Its model-agnostic nature ensures broad applicability across diverse model architectures. • We introduce a prior knowledge integration mechanism based on attention masking, which consistently enforces user-specified causal constraints across all Transformer layers. This design enables our framework to exploit a multi-layer architecture necessary for modeling complex nonlinear dependencies. It also allows for human-in-the-loop refinement to prevent causal inference from being misled by spurious correlations in data. • We conduct extensive experiments demonstrating that our method substantially outperforms other state-of-the-art approaches, achieving a 12.8% improvement in F1-score for causal discovery and 98.9% accuracy in estimating causal lags. 2 Related Work 2.1 Classical Methods for Temporal Causal Discovery Causal discovery in time-series data is a long-standing problem that aims to recover both the causal graph and the corresponding time lags among variables. Classical approaches can be broadly grouped into constraint-based, score-based, and Granger causality frameworks. Constraint-based methods such as the PC algorithm and FCI ( Spirtes et al. 2000 ) identify causal relations by testing conditional independencies. PCMCI and PCMCI+ ( Runge et al. 2019b ; Gerhardus and Runge 2020 ) extend these methods to time-series by handling temporal autocorrelation and offering improved scalability with nonlinear independence tests. Score-based approaches optimize a global objective, often involving penalized likelihoods or structural constraints. Examples include NOTEARS ( Zheng et al. 2018 ) and its dynamic extension DYNOTEARS ( Pamfil et al. 2020 ) , which enforce acyclicity using continuous optimization. Granger causality ( Granger 1969 ) remains widely used in econometrics and neuroscience. It tests whether past values of one variable help predict another, typically using linear vector autoregressive (VAR) models. While popular, standard Granger causality fails in nonlinear or confounded settings. Nonlinear extensions have been proposed ( Marinazzo et al. 2008 ) , but they generally lack identifiability guarantees. These classical approaches laid the foundation for causal inference in time-series, but often struggle to capture complex nonlinear dependencies and offer limited support for high-dimensional data. 2.2 Deep Learning-Based Causal Discovery Recent years have seen a surge of interest in deep learning methods for causal discovery, which offer improved flexibility and scalability. cMLP and cLSTM ( Tank et al. 2021 ) use sparse feedforward and recurrent networks, respectively, to predict each variable from its potential causes. By applying sparsity-inducing penalties, these methods infer causal graphs from learned connectivity patterns. TCDF ( Nauta et al. 2019 ) extends this idea with dilated CNNs and gradient-based attention to recover both structure and causal lags. More recently, CausalFormer ( Kong et al. 2024 ) introduces Transformers into causal discovery, using attention weights and structural masks to uncover time-lagged causal relationships. However, it is limited to single-layer architectures and lacks support for enforcing prior constraints or human input. Graph-based models such as DVGNN ( Liang et al. 2024 ) and CUTS ( Cheng et al. 2023 ) leverage dynamic

Read the complete text