ReplaySCM: A Benchmark for Executable Causal Mechanism Induction from Interventions

Serafim Batzoglou · 2026

Read ReplaySCM: A Benchmark for Executable Causal Mechanism Induction from Interventions by Serafim Batzoglou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08197.txt.

Description

Serafim Batzoglou introduces ReplaySCM, a benchmark that grades causal reasoning by running the answer. Most causal tests for language models score isolated answers or graph sketches. Here each of the thirteen hundred items hides a Boolean structural causal model, and the system must output a working mechanism in a restricted language.

Submissions are parsed, checked for legality and cycles, then replayed on training and held-out intervention worlds, so a mechanism earns credit for behaving correctly even if its formulas look different. The benchmark varies how much structure is revealed, from full ordering to hidden roots, and includes tasks that ask for a genuinely different model that still fits the evidence.

Catalog record

CREATORSerafim Batzoglou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE156 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08197

Browse the full library · About the Tower of Babel