SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, Yulan He · 2025
Read SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers by Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, Yulan He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00255.txt.
Description
Yanzheng Xiang and co-authors test whether language models can rebuild algorithms from research papers. Their 2025 paper sets a hard task: an agent has to read a paper, grasp the method, find the right code tools, and write a working program. That takes both deep reading and coding skill.
Their benchmark holds one hundred tasks from recent natural language processing papers, with detailed notes and test cases. Their own two-agent system splits the work between a paper reader and a code writer. A new score compares the reasoning graph of made code with the reference. Even the best model setup reached only a modest success rate. The gap shows how far agents are from truly reproducing published research.
Catalog record
| CREATOR | Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, Yulan He |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-00255