ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?
Christine Ye, Sihan Yuan, Suchetha Cooray, Steven Dillmann, Ian L. V. Roque, Dalya Baron, Philipp Frank, Sergio Martin-Alvarez, Nolan Koblischke, Frank J. Qu, Diyi Yang, Risa Wechsler, Ioana Ciucă, [2mm] Stanford University · 2025
Read ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers? by Christine Ye, Sihan Yuan, Suchetha Cooray, Steven Dillmann, Ian L. V. Roque, Dalya Baron, Philipp Frank, Sergio Martin-Alvarez, Nolan Koblischke, Frank J. Qu, Diyi Yang, Risa Wechsler, Ioana Ciucă, [2mm] Stanford University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24591.txt.
Introduction
Christine Ye and twelve co-authors introduce ReplicationBench, a benchmark testing whether AI agents can replicate entire astrophysics research papers. Frontier agents show promise as research assistants, but novel research use requires first proving their faithfulness and correctness on known work. Astrophysics suits this test because it relies on archival data and computation, with little physical experimentation.
Each paper is split into tasks that an agent must complete to reproduce the paper's results. Scoring then measures how faithfully the agent recreated the analysis and conclusions. The framework gives a concrete standard for trusting agents in extended, open ended scientific workflows.
Catalog record
| CREATOR | Christine Ye, Sihan Yuan, Suchetha Cooray, Steven Dillmann, Ian L. V. Roque, Dalya Baron, Philipp Frank, Sergio Martin-Alvarez, Nolan Koblischke, Frank J. Qu, Diyi Yang, Risa Wechsler, Ioana Ciucă, [2mm] Stanford University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24591
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers? Christine Ye, Sihan Yuan, Suchetha Cooray, Steven Dillmann, Ian L. V. Roque, Dalya Baron, Philipp Frank, Sergio Martin-Alvarez, Nolan Koblischke, Frank J. Qu, Diyi Yang, Risa Wechsler, Ioana Ciucă, [2mm] Stanford University Abstract Frontier AI agents show increasing promise as scientific research assistants, and may eventually be useful for extended, open-ended research workflows. However, in order to use agents for novel research, we must first assess the underlying faithfulness and correctness of their work. To evaluate agents as research assistants, we introduce ReplicationBench, an evaluation framework that tests whether agents can replicate entire research papers drawn from the astrophysics literature. Astrophysics, where research relies heavily on archival data and computational study while requiring little real-world experimentation, is a particularly useful testbed for AI agents in scientific research. We split each paper into tasks which require agents to replicate the paper’s core contributions, including the experimental setup, derivations, data analysis, and codebase. Each task is co-developed with the original paper authors and targets a key scientific result, enabling objective evaluation of both faithfulness (adherence to original methods) and correctness (technical accuracy of results). ReplicationBench is extremely challenging for current frontier language models: even the best-performing language models score under 20%. We analyze ReplicationBench trajectories in collaboration with domain experts and find a rich, diverse set of failure modes for agents in scientific research. ReplicationBench establishes the first benchmark of paper-scale, expert-validated astrophysics research tasks, reveals insights about agent performance generalizable to other domains of data-driven science, and provides a scalable framework for measuring AI agents’ reliability in scientific research. 1 Introduction Current AI agents already demonstrate expert-level scientific reasoning, and agents built on large language models are increasingly being applied across the research lifecycle, from literature review and hypothesis generation ( Su et al. 2025 ; Saeedi et al. 2025a ; Iyer et al. 2024 ) to experimental design, data analysis, and coding ( Yamada et al. 2025 ; Gottweis et al. 2025 ; Baek et al. 2025 ; Schmidgall & Moor 2025 ; Boiko et al. 2023 ; Saeedi et al. 2025b ) . However, existing experiments using agents for scientific research demonstrate failures in robustness and domain understanding ( Beel et al. 2025 ) . Astrophysics is a particularly promising scientific domain for developing and studying AI-assisted scientific research . First, while in other domains LLM-based agents may be limited by real-world interaction and experimentation, the day-to-day workflows of most astrophysics researchers are entirely computational, so the necessary tools can be be easily sandboxed for agent use. Moreover, the astrophysics research community has strong standards for reproducibility and data sharing ( Allen et al. 2012 ; Rebull et al. 2017 ) ; it is common practice for papers to be published with the full datasets required to replicate core results. Thus, astrophysics is a promising testbed for studying the capabilities and failure modes of scientific research agents. To produce high-quality, original scientific research, agents need both expert-grade domain knowledge and high-level autonomous capabilities. Currently, few methods exist to measure the usefulness of AI agents in open-ended scientific research tasks, especially astrophysics. Existing benchmarks testing domain understanding, such as MMLU, GPQA, and HLE, only evaluate static knowledge ( Hendrycks et al. 2021 ; Rein et al. 2023 ; Phan et al. 2025 ) . Alternatively, existing agent benchmarks assess planning ( Valmeekam et al. 2023 ; Zheng et al. 2024 ) , tool use ( Yao et al. 2024 ) , and coding ( Jimenez et al. 2024 ; Hu et al. 2024 ; Kwa et al. 2025 ) , but do not require domain expertise. Current benchmarks fall short of actually evaluating research ability, and evaluating long-horizon agency and expert-level conceptual understanding through realistic scientific contexts. In this work, we introduce ReplicationBench , a benchmark to evaluate AI agents on end-to-end replications of astrophysics research papers. We choose to measure replication on the basis that the ability of agents to execute correct, faithful replications of existing expert-level research is likely predictive of agent quality and reliability during open-ended workflows in similar domains. Replicating established results also provides extremely challenging but well-scoped testbeds for study. We operationalize this by constructing a benchmark of high-quality, reproducible astrophysics research papers to replicate. Given a paper’s manuscript and metadata, an AI agent must completely reproduce the implementation, methodology, and core findings, and is scored on the accuracy of its final results. By enforcing that agents exactly reproduce the original paper, ReplicationBench enables precise measurement of agents’ instruction-following faithfulness and execution correctness within astrophysics research contexts. ReplicationBench is designed for sandboxing , realism , and scalability . To sandbox evaluations, replications do not require real-world experimentation, intermediate results from other works, or expensive computing setups. To construct realistic tasks, we directly use the core results of real research papers, generating tasks which require expert-level reasoning, data analysis, implementation, and scientific interpretation. Regarding scalability , rather than requiring extensive human-designed rubrics (as in Starace et al. 2025 ), ReplicationBench grades final results directly as the outcome of the integrated process, reducing data generation and evaluation costs. The core contributions of this work are: • ReplicationBench Dataset: We publish an original dataset of 19 reproducible, self-contained astrophysics papers decomposed into 107 research-grade tasks. We also publish the ReplicationBench-Plus extension, a set of 11 additional papers with 58 LLM-generated, expert-curated tasks. • Evaluation Framework for Scientific Agents: We establish the broader ReplicationBench evaluation framework, measuring agent reliability in astrophysics research by measuring replication success in existing papers, and provide scaffolding for future dataset expansions. The ReplicationBench framework can be extended to evaluate agents in any data-driven scientific research process. • Frontier Language Model Evaluations: We benchmark five frontier language models in agentic environments. We also present detailed evaluations of agent traces by the authors of the original papers. ReplicationBench introduces a general approach for evaluating AI agents on scientific research replication, and establishes a challenging standard for agents as reliable scientific collaborators in astrophysics. We release the ReplicationBench dataset for use by the community. 2 Related Work Many existing benchmarks evaluate agents on tasks related to code understanding and software engineering, which are core to agentic scientific research. SWE-Bench ( Jimenez et al. 2024 ) evaluates agents on patching real-world GitHub issues, offering a realistic evaluation setting but a shorter agent horizon. MLEBench ( Chan et al. 2025 ) , MLAgentBench ( Huang et al. 2024 ) , RE-Bench ( Wijk et al. 2024 ) , and InfiAgent-DABench ( Hu et al. 2024 ) assess agents on machine learning or data analysis tasks in narrower settings. Domain-specific benchmarks like BixBench ( Mitchener et al. 2025 ) , TPBench ( Chung et al. 2025 ) , LAB-Bench ( Laurent et al. 2024 ) , and GravityBench ( Koblischke et al. 2025 ) evaluate expert-level knowledge and reasoning in scientific domains, but are limited in open-endedness. Some existing benchmarks also attempt to evaluate language model agents on long-horizon scientific tasks. CURIE ( Cui et al. 2025 ) evaluates both reasoning and in-domain expertise using tasks based on scientific papers, but aims for well-scoped subtasks rather than full replication. CORE-Bench ( Siegel et al. 2024 ) tests agents on replicating scientific paper