ReplaySCM: A Benchmark for Executable Causal Mechanism Induction from Interventions
Serafim Batzoglou · 2026
Read ReplaySCM: A Benchmark for Executable Causal Mechanism Induction from Interventions by Serafim Batzoglou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08197.txt.
Description
Serafim Batzoglou introduces ReplaySCM, a benchmark that grades causal reasoning by running the answer. Most causal tests for language models score isolated answers or graph sketches. Here each of the thirteen hundred items hides a Boolean structural causal model, and the system must output a working mechanism in a restricted language.
Submissions are parsed, checked for legality and cycles, then replayed on training and held-out intervention worlds, so a mechanism earns credit for behaving correctly even if its formulas look different. The benchmark varies how much structure is revealed, from full ordering to hidden roots, and includes tasks that ask for a genuinely different model that still fits the evidence.
Catalog record
| CREATOR | Serafim Batzoglou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 156 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08197