ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks
Minghao Li, Ying Zeng, Zhihao Cheng, Cong Ma, Kai Jia · 2025
Read ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks by Minghao Li, Ying Zeng, Zhihao Cheng, Cong Ma, Kai Jia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15804.txt.
Introduction
Minghao Li and co-authors build a benchmark that grades the research reports written by deep research agents, in a 2025 paper. Such agents cut literature work from days to minutes, but their reports must be accurate and complete before anyone should trust them. ReportBench grades two things: whether the cited literature is good and relevant, and whether the report's statements are faithful and true.
The yardstick is built backward from published survey papers on arXiv: their topics become the assignments, and the surveys themselves set the standard. An automated checker extracts citations and claims, compares cited content with the originals, and verifies uncited claims against the web. Commercial deep research agents from OpenAI and Google beat search-augmented chat models, yet coverage and factual consistency still leave wide room to grow. Code and data are promised openly.
Catalog record
| CREATOR | Minghao Li, Ying Zeng, Zhihao Cheng, Cong Ma, Kai Jia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15804
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks Minghao Li, Ying Zeng, Zhihao Cheng, Cong Ma, Kai Jia Abstract The advent of Deep Research agents has substantially reduced the time required for conducting extensive research tasks. However, these tasks inherently demand rigorous standards of factual accuracy and comprehensiveness, necessitating thorough evaluation before widespread adoption. In this paper, we propose ReportBench, a systematic benchmark designed to evaluate the content quality of research reports generated by large language models (LLMs). Our evaluation focuses on two critical dimensions: (1) the quality and relevance of cited literature, and (2) the faithfulness and veracity of the statements within the generated reports. ReportBench leverages high-quality published survey papers available on arXiv as gold-standard references, from which we apply reverse prompt engineering to derive domain-specific prompts and establish a comprehensive evaluation corpus. Furthermore, we develop an agent-based automated framework within ReportBench that systematically analyzes generated reports by extracting citations and statements, checking the faithfulness of cited content against original sources, and validating non-cited claims using web-based resources. Empirical evaluations demonstrate that commercial Deep Research agents such as those developed by OpenAI and Google consistently generate more comprehensive and reliable reports than standalone LLMs augmented with search or browsing tools. However, there remains substantial room for improvement in terms of the breadth and depth of research coverage, as well as factual consistency. The complete code and data will be released at the following link: https://github.com/ByteDance-BandAI/ReportBench . 1 Introduction The rapid development of LLM-powered Deep Research agents has revolutionized the process of knowledge synthesis by enabling autonomous execution of extensive research tasks, including academic literature surveys, industry analyses, and market assessments Chen et al. 2025 ; Gottweis et al. 2025 ; Lu et al. 2024 ; Tang et al. 2025 ; Yamada et al. 2025 ; Zheng et al. 2025 ; Li et al. 2025 . Tasks that traditionally required days or weeks of manual effort can now be completed within minutes. Notable examples include advanced systems such as OpenAI OpenAI 2025 and Google’s Gemini Deep Research Google 2025 , which effectively integrate various external tools and perform multiple rounds of deep reasoning. Despite their promising capabilities, widespread practical adoption critically depends on their ability to consistently deliver research reports with high factual accuracy and comprehensive content quality. Therefore, it is essential to monitor and ensure the quality of generated reports through evaluation. However, defining what constitutes a good report is challenging and lacks broad consensus, resulting in the current absence of mature evaluation methodologies for research report generation. In addressing this challenge, we decompose the evaluation of research reports generated by LLMs into two core dimensions: writing quality and report content. Due to the subjectivity of writing-style evaluation, while the criteria for assessing content quality can be more clearly defined, this work focuses primarily on the evaluation of report content, leaving the assessment of writing quality to future work. Specifically, we assert that the content quality of research reports hinges on two critical factors: (1) the quality and relevance of cited literature, and (2) the faithfulness and veracity of generated statements, whether derived from cited references or produced by the model. To establish a high-quality benchmark capable of rigorously assessing research reports, we propose ReportBench, a novel evaluation framework leveraging expert-generated literature reviews. Given the constraints of relying on human annotators, who typically vary in expertise, we propose using published survey papers available on arXiv as gold-standard references. Published survey papers are typically written by domain experts and have undergone a peer review process that provides additional expert-level validation, considered among the highest-quality research reports currently available. In practice, our methodology unfolds in two phases. First, we generate domain-specific retrieval prompts directly from expert-authored survey papers on arXiv: by analyzing each paper’s publication date and full text, we generate three granularity levels of prompts (sentence-level, paragraph-level, and richly detailed versions) that precisely capture the scope, methods, and temporal constraints of the original research. These prompts form the backbone of our evaluation corpus, ensuring that downstream agents search and synthesize information within the exact topical and chronological boundaries of each survey. We extract the list of cited references from the arXiv surveys as the ground truth. Given the synthesized prompts as test inputs, Deep Research agents conduct research and generate reports, which are then evaluated based on the reference overlap with the ground truth, serving as a measure of the research skills. In the second phase of our validation pipeline, we design two different verification procedures based on whether a statement includes an explicit citation to external literature. Specifically, for cited statements, the system identifies all in-text citations within the report, maps each citation to its corresponding source document, and employs semantic matching to ensure factual support from the cited literature. For non-cited statements, the framework employs a voting mechanism across multiple web-connected models to verify the factuality of these statements. By combining these complementary validation procedures, ReportBench delivers a systematic and detailed assessment of AI-generated research reports, ensuring the relevance and quality of cited literature and the factual accuracy of all claims through citation-based and web-based validation. Our contributions can be summarized as follows: • We present ReportBench , a systematic benchmark designed to evaluate the quality of research reports generated by Deep Research agents, with a focus on the quality of references and the factual accuracy of all statements presented in the report. • We propose an automated and scalable data synthesis method for constructing academic survey tasks, including prompts and ground truth, from expert-authored survey papers on arXiv. Besides, we introduce an automatic agentic evaluation framework that evaluates the precision and recall of the generated report with respect to a ground-truth reference and performs factual verification of individual claims made within the report. • We release a comprehensive benchmark suite—datasets, prompts, and evaluation scripts—to support reproducible research and community-driven progress in evaluating LLM-based knowledge synthesis. Figure 1: Overall benchmark data construction workflow. 2 Methodology We introduce ReportBench , a comprehensive evaluation framework designed to rigorously assess Deep Research agents through two interconnected components: (i) the automated construction of high-quality benchmark datasets derived from expert-authored survey papers, and (ii) a systematic validation pipeline that evaluates the quality and factual consistency of AI-generated research reports. In the following sections, we detail the processes that underlie the synthesis of the dataset and the design of our evaluation workflow. 2.1 Dataset Construction In this section, we detail the end-to-end pipeline to construct high–quality deep research questions along with ground-truth answers based on published survey papers. This workflow comprises three consecutive phases: (i) survey paper identification and reference extraction, (ii) prompt generation, and (iii) application domain distribution. A diagram illustrating the data construction process is presented in Figure 1 . 2.1.1 Phase I: Survey Paper Identification and Reference Extraction The first step is to identify high-quality survey papers to create problems. We start from the complete arXiv metadata snapshot arXiv.org submitters 2024 and only reserve papers with a submission date later than 2020-01-01. To ensure the quality of papers, we only select those that have undergone peer review and have been formally published. We achieve this by using regular expressions, i.e. , querying over titles to match “survey” or “review”