SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers

Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, Yulan He · 2025

Read SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers by Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, Yulan He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00255.txt.

Description

Yanzheng Xiang and co-authors test whether language models can rebuild algorithms from research papers. Their 2025 paper sets a hard task: an agent has to read a paper, grasp the method, find the right code tools, and write a working program. That takes both deep reading and coding skill.

Their benchmark holds one hundred tasks from recent natural language processing papers, with detailed notes and test cases. Their own two-agent system splits the work between a paper reader and a code writer. A new score compares the reasoning graph of made code with the reference. Even the best model setup reached only a modest success rate. The gap shows how far agents are from truly reproducing published research.

Catalog record

CREATORYanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, Yulan He
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-00255

Browse the full library · About the Tower of Babel