NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

Horizon Research, Frontis.AI Tsinghua University Correspondence: [email protected] Leaderboard GitHub HuggingFace · 2026

Read NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers? by Horizon Research, Frontis.AI Tsinghua University Correspondence: [email protected] Leaderboard GitHub HuggingFace (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24530.txt.

Introduction

Horizon Research, the team behind this report from Frontis.AI and Tsinghua University, asks whether AI coding agents can match the best published results of papers in the Nature family of journals. Agents are moving from reproducing code toward autonomous research. Existing benchmarks carry limitations that blur genuine advances with pattern matching, though. NatureBench rebuilds the test around real published state of the art results as the bar.

Tasks span six scientific domains drawn from Nature family papers. Source figures anchor each problem. Ten models are ranked on a public leaderboard by a Surpass-SOTA score. It measures whether an agent's solution actually clears the published result, rather than merely approaching it. The benchmark's premise is strict. Fitting familiar patterns to new data earns nothing, unless the number beats the paper. Horizon Research positions the suite as the rigorous evaluation autonomous science needs.

Catalog record

CREATORHorizon Research, Frontis.AI Tsinghua University Correspondence: [email protected] Leaderboard GitHub HuggingFace
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24530

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers? Horizon Research, Frontis.AI Tsinghua University Correspondence: [email protected] Leaderboard GitHub HuggingFace Abstract \uselogo \fvlogodividerfalse \fvtitlecentertrue \setheadertext NatureBench Figure 1 : NatureBench overview. (a) Six task domains with representative source figures [ Miao et al. 2025 , Bai et al. 2025 , Pineda et al. 2025 , Oppliger et al. 2024 , Igashov et al. 2024 , Hasani et al. 2022 ] . (b) NatureBench leaderboard by Surpass-SOTA ( g>0.1 ) across ten models. 1 Introduction AI coding agents are rapidly moving toward autonomous scientific research [ Karpathy 2026 , Lu et al. 2026 , Gottweis et al. 2026a ] , from reproducing published implementations to conducting end-to-end research workflows. As these systems begin to target real scientific problems, rigorous evaluation becomes critical: without reliable benchmarks, it is impossible to tell whether an agent is genuinely advancing the state of the art or merely fitting familiar patterns to new data. However, existing benchmarks for evaluating agent capabilities on scientific research have several limitations. Paper-based benchmarks [ Starace et al. 2025 , Siegel et al. 2024 , Wang et al. 2026 ] measure whether an agent can re-implement a published method, but stop short of the more consequential question: can an agent discover a competitive method on its own? Engineering-optimization benchmarks [ Chan et al. 2025 , Rank et al. 2026 , Nathani et al. 2025 , Qiang et al. 2026 ] target Kaggle competitions or post-training tasks, which do not require the domain reasoning, specialized tooling, or cross-discipline knowledge that characterize research in the natural sciences, and suffer from environment fragmentation that makes independent re-running fragile. Credibly evaluating whether autonomous research agents can advance the frontier of AI-for-Science requires a benchmark that is both challenging and bidirectional. It must test discovery , whether an agent can devise methods that surpass the published state of the art, on genuine scientific problems drawn from the natural sciences rather than on engineering proxies. We present NatureBench , a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery. NatureBench simultaneously extends both axes: the PaperBench axis from Understanding \to Coding to Discovery, and the PostTrainBench axis from Engineering Optimization to Science. It is built on NatureGym , an automated pipeline that converts a published paper into a containerized task package comprising a task brief, the paper’s dataset, a held-out test set with hidden ground truth, and an automated evaluator, addressing the environment-fragmentation problem in prior benchmarks. We collect approximately 5{,}500 papers from ten Nature-family journals published between 2022 and 2025 and apply a three-stage build-then-verify pipeline to yield the final 90 task packages (Figure 2 ). An information firewall removes the source method from each package, so agents must discover solutions rather than reproduce them. The benchmark spans six scientific task domains (cellular omics, protein biology, biomedical modeling, physical modeling, molecular design, and relational reasoning) and uses a SOTA-normalized relative gap g as the primary metric, supplemented by a post-hoc validity judge that detects shortcut behaviors such as output fabrication and feedback gaming. We evaluate ten agents spanning three coding-agent harnesses (Claude Code, Codex CLI, Gemini CLI) and ten frontier models under a strict web-search-disabled protocol as shown in Figure 1 . The strongest agent, Claude Opus 4.7, surpasses the published SOTA ( g>0.1 ) on only 17.8\% of tasks and matches it on 47.8\% . Analysis of the 900 task–agent runs reveals that success is driven primarily by methodological translation, where agents convert scientific tasks into familiar supervised-prediction problems, accounting for 45.5\% of validated successes, rather than by scientific invention. Failures are dominated by wrong method choice ( 45.1\% ) and insufficient compute budget ( 24.4\% ), not by task misunderstanding. Our contributions are as follows: • NatureGym , an automated pipeline that constructs reproducible, containerized per-task environments from Nature-family papers, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. • NatureBench , a benchmark of 90 Nature-sourced tasks across six scientific task domains with a Discovery-oriented evaluation protocol (Surpass-SOTA, Match-SOTA, validity judge) that separates genuine algorithmic progress from engineering optimization and shortcut-taking. 2 NatureGym We introduce NatureGym , a pipeline that turns a published Nature-family paper into a ready-to-run agentic task. Each task is a containerized package comprising a task brief, the dataset, a held-out test set, an automated evaluator, and a SOTA anchor score. NatureGym standardizes papers with heterogeneous formats, toolchains, and data modalities into one reproducible task format, while imposing an information firewall that withholds the original method so that agents must discover solutions rather than reproduce them. 2.1 Pipeline Overview Figure 2 : The NatureGym pipeline. Three review-gated stages turn one Nature-family paper into a containerized task package, refining a shared per-paper record T=(A,D,M,S,B) along the way. An information firewall removes the source method, so the agent receives only dataset inputs, task brief, and a held-out test set, and tries to discover rather than reproduce. As shown in Fig. 2 , NatureGym builds each task through three stages: Paper Filtering (§ 2.2 ), Dataset Acquisition and Verification (§ 2.3 ), and Task Package Construction (§ 2.4 ). Each stage ends with an independent review that catches and corrects errors through a verify–repair loop before the next stage begins. Every stage serves two purposes. First, it makes a binary pass-or-reject decision, terminating all downstream processing for rejected papers. Second, it extracts and refines structured task information into a per-paper record that accumulates across stages, so that task package construction can consume this record directly without re-reading the paper. We represent each task as a tuple T=(A,D,M,S,B) , namely a core algorithm A , a dataset D , a metric M , a SOTA score S , and an optional baseline B . The pipeline starts to fill in this tuple at the filtering stage and refines it in every later stage. Every stage is run by an LLM agent, and a human confirms the critical corrections that each review surfaces. 2.2 Paper Filtering Paper filtering identifies candidate papers suitable for task construction through three steps: preprocessing, a three-level cascade filter, and an adversarial review. Preprocessing. Each paper is converted into three structured components that the subsequent filtering stages consume. After retaining only research articles and dropping non-research content (e.g., news, editorials, corrections, reviews), we produce from each article: (i) markdown text preserving document structure and formulas with citation markers removed; (ii) full-page screenshots of every figure and table; and (iii) a section-tagged list of hyperlinks categorized as data, code, supplementary material, or other, with surrounding context. Three-level filtering. We then apply three filtering levels, each targeting a distinct feasibility dimension: task extractability, evaluation automatability, and data completeness. • Level 1: Task. The paper’s core contribution must yield an extractable ML task: an algorithmic innovation, an ML formulation of a scientific problem, or a domain adaptation of an established method. We exclude papers in which ML serves only as an auxiliary tool, non-computational studies (wet-lab experiments, pure theory, hardware), and tasks that require physical interaction. • Level 2: Evaluation. The paper must claim state-of-the-art performance on a quality-related metric, rather than on speed, cost,

Read the complete text