CausaLab : A Scalable Environment for Interactive Causal Discovery Toward AI Scientists
Junlin Yang *, Dylan Zhang *, Xiangchen Song, Qirun Dai, Xiao Liu, Yuen Chen, Aniket Vashishtha, Jing Shi, Chenhao Tan, Hao Peng · 2026
Read CausaLab : A Scalable Environment for Interactive Causal Discovery Toward AI Scientists by Junlin Yang *, Dylan Zhang *, Xiangchen Song, Qirun Dai, Xiao Liu, Yuen Chen, Aniket Vashishtha, Jing Shi, Chenhao Tan, Hao Peng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26029.txt.
Introduction
Yang and nine colleagues build CausaLab, a testbed where language-model agents must discover causes by experiment. Each episode sets an agent in a mock lab with prior readings, a crystal it can prod, and a second crystal whose pitch it must predict. The hidden rule is a random causal model, so recall alone cannot pass. Scores split prediction from true grasp: GPT-5.2-high hits 92 percent on tasks yet only 0.471 on full graph recovery. Mixed watching and prodding beats pure prodding, and a consistency check curbs early stopping.
Catalog record
| CREATOR | Junlin Yang *, Dylan Zhang *, Xiangchen Song, Qirun Dai, Xiao Liu, Yuen Chen, Aniket Vashishtha, Jing Shi, Chenhao Tan, Hao Peng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54553 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26029
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CausaLab : A Scalable Environment for Interactive Causal Discovery Toward AI Scientists Junlin Yang *, Dylan Zhang *, Xiangchen Song, Qirun Dai, Xiao Liu, Yuen Chen, Aniket Vashishtha, Jing Shi, Chenhao Tan, Hao Peng Abstract We introduce CausaLab , a scalable environment for evaluating interactive causal discovery by LLM agents. Unlike prior evaluations, CausaLab evaluates both whether an agent can solve a problem using causal evidence and whether its answer is grounded in a faithful recovered causal mechanism. Each episode places an agent in a synthetic laboratory: it receives prior measurement records, intervenes on a manipulator crystal, and predicts the resonance frequency of a held-out reactor crystal governed by the same mechanism. The hidden data-generating process is a randomly sampled structural causal model (SCM), so success requires recovering both a causal graph and structural equations rather than recalling prior knowledge. Experiments show a persistent gap between prediction and mechanism recovery: in the purely observational 6-node setting, GPT-5.2-high reaches 92% task accuracy but only 0.471 all-edge F 1 F_{1} . Mixed observation–intervention strategies improve structural fidelity, while pure intervention remains difficult even for strong agents. We identify premature stopping as a major weakness and show that consistency verification mitigates it. CausaLab therefore separates predictive success from causal understanding and exposes current LLM agents’ limits as experimental causal reasoners. Code: https://github.com/DylanZSZ/CausaLab * Junlin Yang and Dylan Zhang contributed equally and both serve as project leads. Junlin Yang’s work was done at the University of Illinois Urbana-Champaign. 1 Introduction Causal reasoning is important because scientific, medical, and policy decisions depend on how systems would respond to interventions, not only on observed associations ( Pearl 2009 ; Pearl and Mackenzie 2018 ; Imbens and Rubin 2015 ) . Yet measuring and making progress in causal reasoning remains challenging, particularly for today’s large language models (LLMs). Existing benchmarks generally translate causal graphs, datasets, or narratives into question-answering and classification tasks ( Qin et al. 2019 ; Romanou et al. 2023 ; Stolfo et al. 2023 ; Jiang et al. 2024 ; Vashishtha et al. 2025 ; Jin et al. 2023a ; Wang 2024 ; Chen et al. 2024b ; Jin et al. 2023b ) . While useful, they leave open the “causal parrot” concern ( Zečević et al. 2023 ) : models can succeed with memorized causal facts or linguistic cues rather than causal reasoning behaviors needed to discover causal mechanisms ( Zheng et al. 2023 ; Liu et al. 2023 ) . Figure 1 : Overview of a CausaLab episode. (1) A hidden SCM generates prior records, a manipulator crystal, and a held-out reactor crystal. (2) The agent observes records and performs budgeted interventions on the manipulator crystal. (3) At each step it emits a DSL thought parsed against the ground-truth SCM. (4) It predicts the reactor frequency ; we score both the prediction and recovered-mechanism trajectory. To illustrate, let’s consider the following thought experiment. Suppose we are interested in studying the causal relationship between temperature and the resonance frequency of a crystal. An LLM agent might appear useful in at least two different ways. (1) It may retrieve from existing sources, such as Wikipedia or its training data, that temperature causes resonance frequency. (2) It may observe paired measurements of temperature and frequency, formulate hypotheses, design experiments, perform interventions, observe the resulting changes, and infer causation from evidence ( Pearl 2009 ; Hauser and Bühlmann 2012 ; Lampinen et al. 2023 ) . While both are valuable in practice, (1) offers little help when the relevant causal knowledge lies beyond the current frontier of human knowledge. We therefore argue that (2) is especially important, particularly for important applications such as scientific discovery, because it enables LLM agents to help advance the frontiers of knowledge in a manner closer to what human scientists would do ( Langley 2019 ; Dunbar and Fugelsang 2005 ; Jansen et al. 2024 ) . We introduce CausaLab (Figure 1 ), a scalable environment for evaluating LLM agents as interactive causal discoverers, joining a recent line of interactive scientific-agent and causal-discovery benchmarks ( Jansen et al. 2024 ; Havrilla et al. 2025 ; Chen et al. 2026 ; Chen et al. 2025 ; Geng et al. 2025 ) . Each episode asks the agent to use evidence from prior records and interventions on one crystal to predict the held-out frequency of another crystal. The shared data-generating mechanism is a hidden structural causal model (SCM) ( Pearl 2009 ) , with a causal graph and structural equations that determine the crystal properties and frequency. The agent receives prior measurement records, can run budgeted interventions on a manipulator crystal through a property manipulator, and must predict the frequency of a separate reactor crystal governed by the same SCM (Figure 1 ; § 3 ). Two design choices distinguish CausaLab from prior causal-reasoning evaluations. First, the hidden SCM is sampled per episode rather than drawn from public causal corpora, which sidesteps the “causal parrot” concern that scores reflect memorized causal lexicon. Second, a lightweight domain-specific language (DSL; § 4 ) records the agent’s accumulated evidence, current graph and equation hypothesis, planned experiment, and action at each step, so we can score not only the final prediction but also the trajectory-level faithfulness of the recovered mechanism to the ground-truth SCM (§ 5 ). Our experiments span closed and open-weight models, multiple model sizes, and thinking versus non-thinking variants, surfacing four findings that prior static benchmarks cannot reach. (1) Correct predictions often do not reflect correct mechanism discovery. Across matched functional-form controls, hidden-perturbation controls, and target-edge controls, endpoint accuracy and mechanism fidelity move separately: agents can find plausible parents while missing quantitative equations, preserve task success while degrading all-edge recovery, or lose accuracy mainly when the target equation itself is perturbed. (2) Observation-conditioned online intervention best balances prediction and graph recovery. Pure observation can boost endpoint accuracy without recovering structure, and pure intervention is weak before observations narrow the hypothesis space. For GPT-5.2-high on 6-node graphs, pure observation reaches 92% accuracy but only 0.47 all-edge F_{1} , while mixed online observation–intervention reaches 80%/0.80. Offline intervention traces do not replace online experimental choice: injecting “Golden” chains raises GPT-5-mini accuracy to 90% on 4 nodes while lowering all-edge F_{1} . (3) Model family and scale pay off unevenly across the two axes. GPT-5.2-high has the best endpoint accuracy and lowest directed all-edge structural Hamming distance (SHD) at every graph size, but gains are not uniform across graph sizes or metrics. Open-weight Qwen3.5 can approach GPT-5-mini on some task scores, yet its SHD rises faster as graphs grow; thinking generally lowers Qwen SHD. Even GPT-5.2-high drops to 64% accuracy and directed SHD 4.761 at 7 nodes. (4) Many failures come from premature commitment, not exhausted budget. Both successful and failed runs leave roughly half the intervention budget unspent, failed runs end with hypotheses inconsistent with their own data, and a single explicit verification step lifts 4-node accuracy from 48% to 60%. CausaLab therefore separates predictive success from causal understanding, revealing how current LLM agents still struggle to explore unfamiliar environments interactively, test candidate mechanisms, and revise toward the causal regularities that govern them. 2 Background and Related Work Causal reasoning goes beyond associational prediction by asking how