BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers?
Fengqing Jiang Yichen Feng Yuetai Li, Luyao Niu Basel Alomair Radha Poovendran · 2025
Read BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers? by Fengqing Jiang Yichen Feng Yuetai Li, Luyao Niu Basel Alomair Radha Poovendran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18003.txt.
Introduction
Fengqing Jiang, with five co-authors, builds a research agent whose job is to write fake but convincing science papers. The point is defensive: as AI writing and AI reviewing grow, fake work might pass review with no human in the loop. Their BadScientist generator uses presentation tricks and needs no real experiments at all.
A strict scoring frame with error bounds tests how often multi-model review systems accept the fakes. Reviewers often flagged integrity worries yet still gave passing scores, a conflict the authors name and measure. Fixes they tried helped only slightly. The paper is a warning for journals drifting toward automated review.
Catalog record
| CREATOR | Fengqing Jiang Yichen Feng Yuetai Li, Luyao Niu Basel Alomair Radha Poovendran |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18003
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers? Fengqing Jiang Yichen Feng Yuetai Li, Luyao Niu Basel Alomair Radha Poovendran Abstract The convergence of LLM-powered research assistants and AI-based peer review systems creates a critical vulnerability: fully automated publication loops where AI-generated research is evaluated by AI reviewers without human oversight. We investigate this through BadScientist , a framework that evaluates whether fabrication-oriented paper generation agents can deceive multi-model LLM review systems. Our generator employs presentation-manipulation strategies requiring no real experiments. We develop a rigorous evaluation framework with formal error guarantees (concentration bounds and calibration analysis), calibrated on real data. Our results reveal systematic vulnerabilities: fabricated papers achieve acceptance rates up to 82.0 % 82.0\% . Critically, we identify concern-acceptance conflict —reviewers frequently flag integrity issues yet assign acceptance-level scores. Our mitigation strategies show only marginal improvements, with detection accuracy barely exceeding random chance. Despite provably sound aggregation mathematics, integrity checking systematically fails, exposing fundamental limitations in current AI-driven review systems and underscoring the urgent need for defense-in-depth safeguards in scientific publishing. † † footnotetext: \dagger Equal Contribution 1 Introduction Large Language Models (LLMs) are fundamentally transforming the scientific research ecosystem, automating tasks once exclusive to human experts. LLM-powered agents are increasingly deployed as end-to-end research assistants, automating ideation, experimentation, and manuscript drafting ( Lu et al. 2024b ; Liu et al. 2025 ; Kon et al. 2025 ; Chan et al. 2024 ) . Simultaneously, LLMs are being explored to alleviate review burdens, serving as reviewers or review assistants ( Checco et al. 2021 ; Liu and Shah 2023 ; Liang et al. 2024b ; Tyser et al. 2024 ) . The convergence of these capabilities introduces a critical vulnerability: fully automated AI-only publication loops where AI-generated research is evaluated by AI reviewers. This raises profound questions about research integrity ( Vasconcelos and Marušić 2025 ; Arar et al. 2025 ) . Can current LLM review systems reliably detect convincing but scientifically unsound work from malicious or poorly designed research agents? Emerging evidence suggests concerning vulnerabilities: LLM reviewers amplify human biases ( Hosseini and Horbach 2023 ) , miss critical flaws, and remain susceptible to adversarial attacks such as prompt injection ( Ye et al. 2024 ; Taylor 2025 ; Zika 2025 ) . While AI-generated text detection is actively studied ( Gao et al. 2023 ; Mitchell et al. 2023 ; Crothers et al. 2023 ) , the adversarial interplay between fabricating and reviewing agents remains critically underexplored. Figure 1: Overview of the BadScientist framework. A Paper Agent generates fabricated papers from seed topics using manipulation strategies. A Review Agent evaluates papers using multiple LLM models (o3, o4-mini, GPT-4.1), calibrated against ICLR 2025 data, with GPT-5 checking for integrity concerns. We investigate this dynamic by asking: Can research agents write convincing but unsound papers that fool LLM reviewers? We introduce BadScientist , a framework that pits fabrication-oriented paper generation against multi-model LLM review systems. Our generator conducts no real experiments, instead employing five presentation-manipulation strategies: exaggerating performance gains ( TooGoodGains ), cherry-picking comparisons ( BaselineSelect ), constructing statistical facades ( StatTheater ), polishing presentation ( CoherencePolish ), and concealing proof gaps ( ProofGap ). We evaluate fabricated papers using LLM reviewers calibrated on ICLR 2025 data to mirror realistic acceptance thresholds. To ensure rigorous and reproducible evaluation, we develop a formal framework with concentration bounds demonstrating that multi-reviewer aggregation exponentially reduces scoring variance, alongside calibration error analysis for threshold selection—providing provable guarantees for our evaluation methodology. Our findings are stark. Fabricated papers achieve acceptance rates up to 82.0\% across strategies. More critically, we observe pervasive concern-acceptance conflict : reviewers frequently flag integrity issues yet assign acceptance-level scores. Our mitigation strategies— Review-with-Detection (ReD) and Detection-Only (DetOnly)—yield only marginal improvements, with detection barely exceeding random chance. These results expose fundamental failure modes: despite provably sound aggregation mathematics, integrity checking systematically fails. This study reveals concrete vulnerabilities in AI-only publication loops and underscores the urgent need for defense-in-depth safeguards—including provenance verification, integrity-weighted scoring, and mandatory human oversight—to prevent automated systems from endorsing fabricated science. 2 Related Work Agents for Scientific Discovery. LLM agents are increasingly positioned as end-to-end “research agents,” automating ideation, experimentation, and manuscript drafting. Systems such as the AI Scientist ( Lu et al. 2024b ) and Auto Research ( Liu et al. 2025 ) report credible, minimally supervised pipelines; complementary benchmarks probe specific stages like ML experimentation and engineering ( Kon et al. 2025 ; Chan et al. 2024 ) . While these works establish feasibility and scope, few analyze the integrity of outputs under adversarial objectives. Agents for Peer Review. LLMs have been explored as reviewers and review assistants, from early feasibility studies ( Liu and Shah 2023 ; Checco et al. 2021 ) to larger evaluations showing partial alignment with human feedback ( Liang et al. 2024b ; Liang et al. 2024a ) . Emerging platforms simulate or standardize review processes and propose bias-aware pipelines ( Tyser et al. 2024 ; Jin et al. 2024 ; Yu et al. 2024 ) , yet concerns persist LLM reviewers can amplify biases or miss deep flaws ( Hosseini and Horbach 2023 ) . Challenges in Agent-vs-Agent Settings. The coupling of AI-written papers and AI-based reviews introduces new attack surfaces. Prompt-injection into manuscripts can tilt LLM verdicts ( Ye et al. 2024 ) , and reports suggest covert instructions have appeared in real preprints ( Taylor 2025 ) . Parallel efforts assess detection and governance: holistic and red-teaming evaluations ( Liang et al. 2022 ; Perez et al. 2022 ) ; detectors and audits for AI-generated scientific text and artifacts ( Gao et al. 2023 ; Mitchell et al. 2023 ; Flitcroft et al. 2024 ; Liu et al. 2024 ; Gosselin 2025 ; Andreev et al. 2024 ; Gritsai et al. 2024 ; Crothers et al. 2023 ) ; and policy guidance on safeguarding research integrity ( Vasconcelos and Marušić 2025 ; Arar et al. 2025 ) . Our Focus. We study the adversarial interplay between an AI paper-writing agent and an AI reviewer: can a fabrication-oriented agent produce “convincing but unsound” papers that fool LLM review pipelines, and what mitigations help? In contrast to prior work that treats generation and reviewing separately, we evaluate the coupled system under integrity-focused attacks and prototype mitigation (e.g., injection-aware defenses ( Zika 2025 ) ). 3 Design of BadScientist 3.1 Preliminary We study whether AI agents can generate convincing fabricated scientific papers that deceive reviewer agents, and how reliably reviewer agents detect such fabrications. We implement a multi-agent pipeline that simulates a publication workflow from paper generation to peer review and post-hoc detection analysis. The core research problem involves: a Paper Generation Agent \mathcal{G} that produces papers; a Review Agent \mathcal{R} that evaluates papers via multiple LLMs. There is also an Analysis System \mathcal{A} that aggregates outcomes and measures detection. Notation. Let \mathcal{X} denote the space of paper artifacts. A paper is x\in\mathcal{X} . Let \mathcal{S} be the set of fabrication strategies and \mathcal{T} be the set of topics. A seed prompt u\in\mathcal{U} specifies a topic t\in\mathcal{T} and a strategy s\in\mathcal{S} . The Review Agent employs models \mathcal{M}=\{m_{1},\dots,m_{M}\} . Each model m produces a K