Transformer Is Inherently a Causal Learner

Xinyue Wang, Stephen Wang, Biwei Huang · 2026

Read Transformer Is Inherently a Causal Learner by Xinyue Wang, Stephen Wang, Biwei Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05647.txt.

Introduction

Xinyue Wang and 2 co-authors argue, in a 2026 paper, that transformers trained to predict the future naturally learn causal structure from time series. A causal graph shows which past variables actually drive which future ones. Their claim is that no explicit causal training objective is needed for that structure to appear.

When a transformer predicts future values, the sensitivity of its outputs to past inputs recovers the underlying causal graph. The authors prove this link under standard identifiability conditions and build a practical extraction method from aggregated gradient scores. On nonlinear, long-dependency, and non-stationary tests, the approach reportedly beat leading discovery algorithms, improving further as data volume and diversity grew.

Catalog record

CREATORXinyue Wang, Stephen Wang, Biwei Huang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE109 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-05647

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Transformer Is Inherently a Causal Learner Xinyue Wang, Stephen Wang, Biwei Huang Abstract We reveal that transformers trained in an autoregressive manner naturally encode time-delayed causal structures in their learned representations. When predicting future values in multivariate time series, the gradient sensitivities of transformer outputs with respect to past inputs directly recover the underlying causal graph, without any explicit causal objectives or structural constraints. We prove this connection theoretically under standard identifiability conditions and develop a practical extraction method using aggregated gradient attributions. On challenging cases such as nonlinear dynamics, long-term dependencies, and non-stationary systems, this approach greatly surpasses the performance of state-of-the-art discovery algorithms, especially as data heterogeneity increases, exhibiting scaling potential where causal accuracy improves with data volume and heterogeneity, a property traditional methods lack. This unifying view lays the groundwork for a future paradigm where causal discovery operates through the lens of foundation models, and foundation models gain interpretability and enhancement through the lens of causality. 1 1 1 Project page: https://www.charonwangg.com/project/transformers-scale-discovery 1 Introduction The ability to discover causality is foundational to intelligence, enabling understanding, prediction, and decision-making about the world. As an evolving field, causal discovery aims to formalize theoretical frameworks for identification criteria and to propose search algorithms to find the true causal structure from observational data ( 61 ; 75 ) . In this area, causal discovery from time series focuses on identifying temporal causal dynamics by exploiting the temporal ordering that naturally constrains the direction of causation. Granger causality ( 25 ; 79 ; 52 ) formalizes this intuition: a variable X Granger-causes Y if past values of X contain information that helps predict Y beyond what is available from past values of Y alone. Additional methods extend this foundation, including constraint-based approaches like PCMCI and its variants that iteratively test conditional independence to examine the existence of causal edges ( 69 ) , score-based methods like DYNOTEARS ( 58 ) that optimize graph likelihood with structural prior regularizations, and functional approaches like TiMINo and VAR-LiNGAM that leverage structural equation models and non-Gaussianity for identifiability ( 65 ; 30 ) . Real-world systems exhibit complex interactions among many variables. For example, financial markets are highly non-stationary and involve very large variable sets ( 19 ) ; neural recordings exhibit strongly nonlinear population dynamics ( 10 ) ; climate sensor networks display long and short-term teleconnections ( 82 ; 53 ) ; and unstructured modalities such as video require modeling long-range spatiotemporal dependencies ( 9 ; 6 ) . Despite rigorous theoretical foundations, prevailing algorithms are often constrained in practice by complex heuristics. Specifically, constraint-based and score-based approaches scale poorly: the number of statistical tests grows rapidly with dimension and lag, and non-parametric tests are computationally expensive ( 69 ; 13 ) . Optimization approaches require careful tuning to achieve the right balance between likelihood and structural regularization ( 91 ; 54 ; 58 ; 92 ) . More fundamentally, these estimators are not scalable representation learners: their learning is not transferable and thus offers little generalizability for zero-shot or few-shot adaptation; their effective capacity and expressiveness are not well-suited for pretraining on diverse systems. Motivated by the striking performance and scaling behavior of autoregressive foundation models ( 11 ; 33 ; 27 ) , we ask whether the properties that make transformers strong forecasters can help causal discovery. Building this connection is valuable in two directions: for discovery, it promises data efficiency by leveraging pretrained representations and a scalable learning paradigm suited to complex dependencies; for foundation models, causal principles offer ways to diagnose limitations in memory and hallucinations, and guide architecture and objective choices. In this paper, we take a first step toward these goals: we revisit common identifiability assumptions in lagged data generation processes and show how decoder-only transformers trained for forecasting, together with input–output gradient attributions via Layer-wise Relevance Propagation (LRP) ( 2 ; 7 ) , reveal lagged causal structure. This view turns modern sequence models into practical, scalable estimators for temporal graphs while opening a path to analyze and strengthen foundation models through causal perspectives. 2 Background In this section, we review causal discovery methods for time-series data and interpretability work on transformers, focusing mainly on language modeling. We then introduce our motivations for connecting these two fields. 2.1 Related Work Time-series Causal Discovery. Recovering causal dynamics from temporal observations requires structural assumptions that determine when and how the true causal graph is identifiable. Constraint-based methods, assuming causal sufficiency and faithfulness, prune spurious edges via conditional independence tests ( 20 ; 69 ; 47 ) . PCMCI and its variants extend the PC algorithm to handle nonlinear, contemporaneous, non-stationary, and high-dimensional settings ( 70 ; 48 ; 71 ) . Score-based methods search for structures that optimize scores encoding functional form and complexity preferences ( 23 ; 57 ) , with recent continuous relaxations reducing computational cost ( 76 ; 58 ) . Functional methods exploit noise asymmetry to resolve edge orientation: VAR-LiNGAM assumes linear dynamics with non-Gaussian noise ( 30 ) , while TiMINo generalizes to nonlinear additive-noise processes ( 63 ) . Granger causality tests whether past values of one series improve forecasts of another, reflecting predictive rather than structural causation ( 25 ) ; neural extensions handle nonlinear dependencies ( 79 ; 52 ; 46 ) . Transformer Interpretability. The success of large-scale pretrained transformers has motivated diverse interpretability efforts ( 26 ; 16 ; 59 ; 15 ; 83 ; 77 ) . Most relevant to our work are methods for input–output token attribution in autoregressive models. Early work equated attention with explanation ( 86 ; 14 ; 87 ; 22 ) , but attention proves manipulable and misaligned with perturbation effects ( 31 ; 72 ; 8 ) . Post-processing techniques like attention rollout aggregate across layers and heads, yet remain forward-weight heuristics rather than faithful causal measures ( 1 ) . Gradient-based methods such as saliency, Integrated Gradients, and Layer-wise Relevance Propagation estimate output-conditioned sensitivities through attention, residual, and MLP paths, yielding attributions that better align with perturbation tests ( 77 ; 7 ; 2 ) . Perturbation-based methods directly quantify importance via prediction changes under input erasure or counterfactuals ( 40 ; 37 ) . We ground token-level attributions in identifiability theory for dynamic systems, providing a causally principled framework for interpreting transformers as structure learners. 2.2 Motivations Modern scientific and industrial systems are high-dimensional, nonlinear, non-stationary, and data-rich—precisely the regime where classical causal discovery faces brittle assumptions, exploding conditioning sets, and poor scaling with many variables, long lags, and complex dependencies ( 68 ; 51 ) . Decoder-only transformers sit at the opposite point in the design space: a single autoregressive objective, contextualized dependency modeling, and predictable gains from scale, data, and compute ( 33 ) . Empirically, the same architecture transfers across modalities and domains and often outperforms specialized models in text, vision, and long-context time series ( 44 ; 41 ) . Our premise is pragmatic: if many real-world regularities arise from a small set of sparse, independent mechanisms, then an autoregressive learner that already scales and generalizes is a natural interface to structure learning. This brings three immediate benefits: (i) amortization—forecasting supplies a ubiquitous training signal, turning

Read the complete text