Does TabPFN Understand Causal Structures?
Omar Swelam, Lennart Purucker, Jake Robertson, Hanne Raum, Joschka Boedecker, Frank Hutter · 2025
Read Does TabPFN Understand Causal Structures? by Omar Swelam, Lennart Purucker, Jake Robertson, Hanne Raum, Joschka Boedecker, Frank Hutter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07236.txt.
Description
Omar Swelam and five co-authors ask whether TabPFN, a transformer foundation model for tabular data, secretly understands causality. TabPFN was pre-trained on synthetic data sets generated from structural causal models, so its internal representations might encode causal information as a byproduct. The authors build an adapter with a learnable decoder and special causal tokens that read signals from the model's frozen embeddings and decode them into adjacency matrices describing causal graphs.
The extracted information supports causal discovery that outperforms several standard algorithms, and the causal signal concentrates in the model's middle layers rather than its surface. That localization suggests the representation, not the output head, carries the causal content. The findings open a direction for interpretable foundation models: pre-trained tabular models might be repurposed as causal discovery engines with only a light adapter.
Catalog record
| CREATOR | Omar Swelam, Lennart Purucker, Jake Robertson, Hanne Raum, Joschka Boedecker, Frank Hutter |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07236