ScientistOne : Towards Human-Level Autonomous Research via Chain-of-Evidence

Rui Meng, Bhavana Dalvi Mishra*, Jiefeng Chen*, Chun-Liang Li, Palash Goyal, Mihir Parmar, Yiwen Song, Yale Song, Rajarishi Sinha, Parthasarathy Ranganathan, Burak Gokturk, Jinsung Yoon, Tomas Pfister · 2026

Read ScientistOne : Towards Human-Level Autonomous Research via Chain-of-Evidence by Rui Meng, Bhavana Dalvi Mishra*, Jiefeng Chen*, Chun-Liang Li, Palash Goyal, Mihir Parmar, Yiwen Song, Yale Song, Rajarishi Sinha, Parthasarathy Ranganathan, Burak Gokturk, Jinsung Yoon, Tomas Pfister (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26340.txt.

Introduction

Rui Meng and twelve colleagues' 2026 paper tackles verifiability failures in autonomous research agents, such as fabricated citations, unreproducible scores, and method accounts that diverge from code. Their Chain-of-Evidence scheme demands every claim trace to its evidence source, and ScientistOne maintains such chains from review to writing. A post-hoc audit checks scores, specs, sources, and method-code alignment. Across seventy-five papers from five systems, every rival fails systematically, while it reaches zero hallucinated references, perfect score checks, and best method-code scores.

Catalog record

CREATORRui Meng, Bhavana Dalvi Mishra*, Jiefeng Chen*, Chun-Liang Li, Palash Goyal, Mihir Parmar, Yiwen Song, Yale Song, Rajarishi Sinha, Parthasarathy Ranganathan, Burak Gokturk, Jinsung Yoon, Tomas Pfister
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108393
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26340

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ScientistOne : Towards Human-Level Autonomous Research via Chain-of-Evidence Rui Meng, Bhavana Dalvi Mishra*, Jiefeng Chen*, Chun-Liang Li, Palash Goyal, Mihir Parmar, Yiwen Song, Yale Song, Rajarishi Sinha, Parthasarathy Ranganathan, Burak Gokturk, Jinsung Yoon, Tomas Pfister Abstract Autonomous research agents produce competitive solutions and professional-looking manuscripts, yet their outputs can contain verifiability failures undetectable by evaluations that only assess surface presentation rather than evidence grounding: fabricated citations, unreproducible scores, and method descriptions that diverge from the implementation. These failures share a common root: no existing evaluation protocol audits whether claims are supported, and no existing autonomous research system is designed to trace claims back to evidence. We address this gap through three contributions. First, Chain-of-Evidence (CoE), a verifiability framework requiring every claim to be traceable to its evidence source. Second, ScientistOne , an end-to-end autonomous research system that maintains evidence chains by construction throughout literature review, solution discovery, and paper writing. Third, CoE Integrity Audit , a post-hoc audit whose four integrity checks—score verification, specification violation, reference verification, and method–code alignment—apply uniformly to all systems. Across 75 papers spanning five systems and five frontier research tasks, we find that every baseline exhibits at least one systematic failure mode: hallucinated reference rates reach 21%, score verification passes in as few as 42% of papers, and method–code alignment ranges from 20% to 80%. ScientistOne is the only system to achieve zero hallucinated references (0/337 bibliography entries), perfect score verification (12/12), and the highest method–code alignment (14/15), while matching or exceeding human expert performance on all five tasks. We further demonstrate that ScientistOne generalizes to six additional tasks spanning medical imaging, fine-grained recognition, 3D perception, and parameter-constrained language modeling, achieving state-of-the-art on Parameter Golf and gold medals on MLE-Bench tasks where baselines fail entirely. Project website: https://scientist-one.github.io/ * * footnotetext: These authors contributed equally to this work. 1 Introduction Large language models are increasingly deployed not as isolated assistants but as autonomous agents that conduct entire research workflows—from literature review and hypothesis generation through experimental design and execution to manuscript writing ( Lu et al. 2024 ; Yamada et al. 2025 ; Weng et al. 2025 ; Tang et al. 2025 ; Schmidgall et al. 2025 ; Jansen et al. 2025 ) . On systems-optimization tasks, such agents now produce solutions competitive with human experts ( Cheng et al. 2025b ; Novikov et al. 2025 ) , and end-to-end pipelines have generated papers accepted at peer-reviewed workshops ( Yamada et al. 2025 ) . The resulting artifacts—code, experimental results, and professional-looking manuscripts—are increasingly difficult to distinguish from human-authored research on surface quality alone. This rapid capability growth exposes a structural tension between generation and verification . Autonomous research systems operate as multi-stage pipelines in which each stage consumes the output of the previous one: a literature summary shapes the hypothesis, the hypothesis determines the experiment, and experimental results feed into the manuscript. In such architectures, errors introduced at any stage are not merely preserved but amplified—a flawed summary can bias experimental design, and a misinterpreted result can carry through into a paper that appears internally coherent, precisely because the same error is reflected consistently across sections. The risk grows with trajectory length: agents struggle to track an ever-expanding context ( Liu et al. 2024 ; Liu et al. 2023b ) , hallucinate, and drift from the original objective. The problem is exacerbated by fundamental limitations in how language models handle evidence: generated text is difficult to verify against sources ( Liu et al. 2023a ) , factual claims drift from their grounding ( Min et al. 2023 ) , and scientific citations are frequently inaccurate or fabricated ( Press et al. 2024 ) . In autonomous pipelines, these failure modes interact and compound—a model can overstate method descriptions beyond what the code implements, report scores that do not reproduce under the benchmark’s own evaluator, and populate bibliographies from parametric memory rather than retrieval, all while producing text that reads as technically sound. Existing evaluation protocols, whether automated review scores or benchmark leaderboards, assess surface presentation (i.e., how the paper reads) and procedural completion but do not check whether individual claims trace to supporting evidence. This verifiability gap is not hypothetical. In a systematic audit of 75 papers from five autonomous research systems across five benchmark tasks, we find that every baseline system exhibits evidence chain failures : hallucinated references that do not correspond to any real publication (up to 21% of all bibliography entries), method sections that describe algorithms not present in the submitted code, unreproducible scores, and solution code that exploits the evaluator rather than solving the task. These failures share a common root cause: no existing evaluation protocol audits whether claims are supported, and no existing autonomous research system is designed to trace claims back to evidence. We address this with Chain-of-Evidence (CoE), a verifiability framework for AI-driven research. Just as ACID 1 1 1 Atomicity, consistency, isolation, durability. ( Härder and Reuter 1983 ) defines what “reliable” means for a database transaction, CoE defines what “verifiable” means for a research claim: every claim must trace, through a recorded evidence chain, to a grounding source. We instantiate CoE in three ways: 1. The CoE Standard (§ 3 ): a claim taxonomy (citation, numerical, methodological, conclusion) and the evidence chain structure required for each type. 2. ScientistOne (§ 4 ): an end-to-end autonomous research system whose pipeline—Problem Investigator, Discovery Engine, and Paper Writer with Claim Verifier—is designed to satisfy CoE natively. The Problem Investigator reads up to 100 full-text PDFs per topic, producing grounded experiment briefs. And the Claim Verifier checks every claim in the draft against its declared evidence source before the final paper is produced. 3. CoE Integrity Audit (§ 5 ): a post-hoc audit for verifying an AI-driven research paper through four integrity checks—Score Verification, Specification Violation, Reference Verification, and Method-Code Alignment—targeting the most damaging evidence chain failures. We apply CoE Integrity Audit to 15 papers from each of five systems across five frontier systems-research tasks from ADRS ( Cheng et al. 2025b ; Liu et al. 2026c ) (§ 6 ). Every baseline exhibits at least one integrity check failure. ScientistOne achieves zero hallucinated references (0/337 bibliography entries), perfect score verification (12/12), and the highest method–code alignment (14/15), while matching or exceeding human expert solver performance on all five tasks. We further demonstrate that ScientistOne generalizes to six additional tasks spanning medical imaging, fine-grained recognition, 3D perception, and parameter-constrained language modeling, achieving state-of-the-art on Parameter Golf and gold medals on MLE-Bench tasks where baselines fail entirely. 2 Related Work Autonomous research agents. End-to-end autonomous research systems have rapidly expanded from constrained ML templates to multi-stage pipelines that coordinate literature grounding, hypothesis generation, experimentation, and paper writing. The AI Scientist ( Lu et al. 2024 ) pioneered end-to-end automation but operates on fixed ML templates with frequent hallucinations in writing and limited paper quality. AI Scientist-v2 ( Yamada et al. 2025 ) advances this with best-first tree search (BFTS) over experimental branches and review-aware reporting, achieving workshop-level paper quality. Concurrent systems extend the pipeline in different directions. On the ideation side, PiFlow ( Pu et al. 2025 ) steers hypothesis exploration via information-theoretic principle selection and CodeScientist ( Jansen et

Read the complete text