RePro: Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verification

Mingyang Zhou, Quanming Yao, Lun Du, Lanning Wei, Da Zheng · 2025

Read RePro: Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verification by Mingyang Zhou, Quanming Yao, Lun Du, Lanning Wei, Da Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16671.txt.

Introduction

Mingyang Zhou, Quanming Yao, Lun Du, Lanning Wei, and Da Zheng reproduce machine learning papers from text to code with the help of a checklist the system writes itself. Reproduction is where published claims meet reality, and automated agents often garble formulas or algorithmic logic. Reflection with feedback helps agents, but paper reproduction lacked a way to generate good feedback. The team's framework, RePro, first extracts a paper's fingerprint.

The fingerprint is a set of atomic, checkable criteria drawn from the paper. Code is generated, then verified criterion by criterion in an iterative loop. Mismatches trigger targeted revisions rather than wholesale rewrites. On a public code reproduction benchmark the method beat baselines by a clear double-digit margin. Complex logical and mathematical criteria, the usual graveyard, were revised correctly in reflection. Human debuggers use checklists; RePro gives agents one.

Catalog record

CREATORMingyang Zhou, Quanming Yao, Lun Du, Lanning Wei, Da Zheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-16671

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RePro: Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verification Mingyang Zhou, Quanming Yao, Lun Du, Lanning Wei, Da Zheng Abstract Reproducing machine learning papers is essential for scientific progress but remains challenging for both humans and automated agents. Existing agent-based methods often struggle to fully and accurately reproduce implementation details such as mathematical formulas and algorithmic logic. Previous studies show that reflection with explicit feedback improves agent performance. However, current paper reproduction methods fail to effectively adopt this strategy. This gap mainly arises from the diverse paper patterns, complex method modules, and varied configurations encountered in research papers. Motivated by how humans use systematic checklists to efficiently debug complex code, we propose RePro , a Re flective Paper-to-Code Repro duction framework that automatically extracts a paper’s fingerprint, referring to a comprehensive set of accurate and atomic criteria serving as high-quality supervisory signals. The framework first generates code based on the extracted information, and then leverages the fingerprint within iterative verification and refinement loop. This approach systematically detects discrepancies and produces targeted revisions to align generated code with the paper’s implementation details. Extensive experiments on the PaperBench Code-Dev benchmark have been conducted, RePro achieves 13.0% performance gap over baselines, and it correctly revises complex logical and mathematical criteria in reflecting, on which the effectiveness is obvious. 1 Introduction Machine learning paper reproduction is the task of generating code that faithfully and accurately replicates a paper’s methods, which could include but not limited to data-processing procedures, methodological designs, experimental evaluations ( Starace et al. 2025 ; Semmelrock et al. 2025 ; Semmelrock et al. 2023 ; Albertoni et al. 2023 ) . With the rapid developments in AI, this task has become increasingly critical, as the ability to reliably reproduce new findings is essential for accelerating scientific progress, as highlighted in top-tier conference submission checklist like AAAI and NeurIPS. However, achieving the required implementation fidelity is a highly demanding endeavor that requires time and expertise ( Starace et al. 2025 ) . Figure 1: A comparison between human and agent system approaches to paper-to-code reproduction. The left panel shows that manually debugging code against a paper is inefficient and labor-intensive, while a professional checklist can always offer a systematic way to identify and fix errors. Our agent system (right) mimics the process by designing fine-grained supervisory signals, acting as a tailored checklist. This guides the agent through a reflective loop of verification and refinement, ensuring the generated code aligns with the paper’s methods. Recent work has begun to explore how to automate paper reproduction by designing agents based on powerful large language models (LLMs) ( OpenAI 2024 ; DeepSeek-AI 2025a ; DeepSeek-AI 2025b ; Anthropic 2025 ; Yang et al. 2025 ) . Pioneering the autonomous approach, PaperBench ( Starace et al. 2025 ) introduces BasicAgent and IterativeAgent, which leverage a ReAct ( Yao et al. 2022 ) framework to attempt end-to-end reproduction. In contrast, other methods employ more structured workflows. PaperCoder ( Seo et al. 2025 ) decomposes the generation process into distinct steps of planning, analysis, and coding. Similarly, AutoReproduce ( Zhao et al. 2025 ) first performs a paper summary and enriches its context retrieving relevant prior work and their code. This is then passed to a code agent for implementation, followed by a high-level evaluation against the initial summary. While decomposing the generation process has proven beneficial, existing methods fail to fully and accurately align with the exhaustive evaluation details specified in the source paper, as empirically evaluated in ( Starace et al. 2025 ) . This challenge primarily stems from the complexity of procedures and the diversity of configurations in papers, which also result in ambiguous or insufficient supervisory signals for effective reflection ( Seo et al. 2025 ; Wang et al. 2024a ) . The high-quality supervisory signals could be unambiguous scalar, such as unit test pass-or-fail evaluation results or compiler results ( Shinn et al. 2023 ; Jiang et al. 2025 ; Hong et al. 2023 ) . Therefore, motivated by these and the human practice of using systematic checklists to efficiently debug complex code (see Figure 1 ) ( Zhong, Wang, and Shang 2024 ; Amershi et al. 2019 ) , a promising approach to bridge this gap is to adopt similar supervisory signals which can also effectively guide the subsequent reflection. ( Shinn et al. 2023 ; Hong et al. 2023 ; Gu et al. 2024 ) . However, how to obtain high-quality supervisory signals remains challenging due to the complexity of the papers. In this paper, we tackle this challenge by introducing a novel Re flective Paper-to-Code Repro duction framework for paper reproduction, called RePro . As shown in Figure 2 , our approach begins by addressing the supervisory signal issue through the automatic extraction of a paper’s unique fingerprint, defined as a comprehensive set of verifiable binary criteria that encapsulate the core implementation details. This fingerprint is generated by extracting multi-level guides, standardizing into atomic criteria, and filtering criteria steps, and they are designed to ensure the comprehensive, accurate, and atomic principles of fingerprints. In reproducing paper, we first generate high-level framework and then fill it with detailed implementations following the guides. In refining code, a verifier agent reviews code and provides pass-or-fail feedback, and a planner agent generates a comprehensive revision plan, which is then executed by an editor agent through targeted, minimal modifications to the code. By doing this, the paper could be reproduced in a reflective manner enabled by the paper fingerprints. Extensive experiments on the PaperBench Code-Dev benchmark are conducted, and RePro achieves state-of-the-art performance, outperforming strong baselines by a significant margin of 13.0%. Its improvements come from correcting complex logical and mathematical errors when refining code. Our main contributions can be summarized as follows: • Motivated by the checklist used by humans during code debugging, we design high-quality supervisory signals that can guide subsequent reflection. • We propose RePro, a complete and reflective multi-agent framework that effectively integrates supervisory signals for paper’s code reproduction, evaluation, and revision. • We show through extensive experiments that our paper-to-code reproduction framework achieves state-of-the-art performance, and validate the effectiveness of our designed paper fingerprint. 2 Related Work 2.1 Automated Paper Reproduction by LLM To accelerate machine learning progress, existing methods attempted to automate paper reproduction with LLMs ( Starace et al. 2025 ; Seo et al. 2025 ; Zhao et al. 2025 ; Gandhi et al. 2025 ; Hua et al. 2025 ; Qian et al. 2024 ) . To fully leverage the autonomous capabilities of LLM-based agent, PaperBench ( Starace et al. 2025 ) proposes the BasicAgent built upon a ReAct framework, equipped with tools like a bash shell, Python executor to independently generate, run and submit code. IterativeAgent extends this by enabling iterative, step-by-step reproduction within a fixed time. Other recent methods design careful workflows involving paper analysis, planning and code development. PaperCoder ( Seo et al. 2025 ) follows a single-pass workflow with stages for planning implementation-level abstractions, analyzing modules, and generating code files while managing context for consistency. AutoReproduce ( Zhao et al. 2025 ) employs collaborative research and code agents that gather knowledge, generate preprocessing scripts and iteratively refine implementations mainly for executability. Despite progress, existing

Read the complete text