DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis
Liana Patel † , Negar Arabzadeh ‡ , Harshit Gupta † , Ankita Sundar ‡ , Ion Stoica ‡ , Matei Zaharia ‡ , Carlos Guestrin † · 2025
Read DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis by Liana Patel † , Negar Arabzadeh ‡ , Harshit Gupta † , Ankita Sundar ‡ , Ion Stoica ‡ , Matei Zaharia ‡ , Carlos Guestrin † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20033.txt.
Introduction
Liana Patel, Negar Arabzadeh, Harshit Gupta, Ankita Sundar, Ion Stoica, Matei Zaharia, and Carlos Guestrin build a live benchmark for AI systems that write research summaries, in a 2025 paper. Such systems search the web and produce long, cited reports. Judging them is an open problem. Old benchmarks reward short factual answers. Curated datasets go stale and leak into training. Their benchmark draws fresh questions from recent papers. The task is to write a related-work section with real citations.
An automated grader scores three dimensions. Knowledge synthesis, retrieval quality, and verifiability are measured apart. The team also releases a simple reference system. It posts a strong baseline score. Leading deep research products were tested too. No system passed about thirty-one percent overall. The authors call the task far from solved. They release all code and data openly. The benchmark is meant to steer the field's progress.
Catalog record
| CREATOR | Liana Patel † , Negar Arabzadeh ‡ , Harshit Gupta † , Ankita Sundar ‡ , Ion Stoica ‡ , Matei Zaharia ‡ , Carlos Guestrin † |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-20033
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis Liana Patel † , Negar Arabzadeh ‡ , Harshit Gupta † , Ankita Sundar ‡ , Ion Stoica ‡ , Matei Zaharia ‡ , Carlos Guestrin † Abstract . The ability to research and synthesize knowledge is central to human expertise and progress. An emerging class of systems promises these exciting capabilities through generative research synthesis , performing retrieval over the live web and synthesizing many discovered sources into long-form, cited summaries. However, evaluating such systems remains an open challenge: existing question-answering benchmarks focus on short-form factual responses, while expert-curated datasets risk staleness and data contamination. Both fail to capture the complexity and evolving nature of real research synthesis tasks. In this work, we introduce DeepScholar-bench, a live benchmark and holistic, automated evaluation framework designed to evaluate generative research synthesis. DeepScholar-bench draws queries from recent, high-quality ArXiv papers and focuses on a real research synthesis task: generating the related work sections of a paper by retrieving, synthesizing, and citing prior research. We develop an automated evaluation framework that holistically assesses performance across three key dimensions, knowledge synthesis , retrieval quality , and verifiability , using metrics that show strong agreement with expert human judgments. We also develop DeepScholar-base, a reference pipeline for generative research synthesis, implemented efficiently using the LOTUS API. Using the DeepScholar-bench framework, we perform a systematic evaluation of prior open-source systems, search AI’s with open-source and strong proprietary models, OpenAI’s DeepResearch, and DeepScholar-base. We find that DeepScholar-base establishes a strong baseline, attaining competitive or higher performance than prior open-source systems, search AI’s and OpenAI’s DeepResearch. We also find that DeepScholar-bench remains far from saturated, with no system exceeding a score of 19 % 19\% across all metrics. These results underscore both the difficulty and the importance of DeepScholar-bench as a foundation for progress toward AI systems capable of generative research synthesis. We make our benchmark code and data available at https://github.com/guestrin-lab/deepscholar-bench . (a) Open-source systems, Search AIs, and DeepScholar-base, each with Llama-4-Scout-17B-16E-Instruct model. (b) Search AIs with proprietary models (o3, Claude-opus-4, Gemini-2.5-pro), OpenAI DeepResearch, and DeepScholar-base (GPT4.1, Claude-opus-4) Figure 1 . Performance comparison of generative research synthesis systems on DeepScholar-Bench. On the left (a), we show the performance of prior open-source systems, including DeepScholar, STORM, and OpenScholar, our DeepScholar baseline, and a Search AI, each run with the open-source Llama-4-Scout-17B-16E-Instruct model. On the right (b), we show performance of Search AIs with strong proprietary models—o3, Claude-opus-4, and Gemini-2.5-pro—as well as OpenAI’s commercial DeepResearch and DeepScholar-base (GPT4.1 + Claude-opus-4), one of our reference pipelines. Full evaluation results appear in Section 5 . 1. Introduction A crucial underpinning of human knowledge and innovation is the ability of human experts to research and synthesize known facts and new findings to support others in comprehending, verifying and building upon existing work. Recently, new systems designed for generative research synthesis offer exciting capabilities and promises to automate challenging research synthesis tasks, which often require laborious hours of literature search, reading, and writing from human experts. These systems include a wide array of both commercial DeepResearch offerings, including ones from OpenAI ( noa [n.d.]g ) , Gemini ( noa [n.d.]c ) , Anthropic ( noa [n.d.]b ) , Grok ( noa [n.d.]e ) , and Perplexity ( noa [n.d.]h ) , as well as open-source methods, such as STORM ( Shao et al. 2024 ) , DeepResearcher ( Zheng et al. 2025 ) , and OpenScholar ( Asai et al. 2024 ) . These systems push the frontier of AI capabilities, demonstrating promising performance on existing factuality and question-answering benchmarks ( Wei et al. 2024 ; Krishna et al. 2025 ; Mialon et al. 2023 ; Wei et al. 2025 ) . Yet, as this new class of systems emerges, a key question remains: how should we benchmark and evaluate generative research synthesis? The progress of these systems requires benchmarks that carefully evaluate their critical capabilities. Specifically, these systems provide three core functions: (1) retrieval , typically from a large, complex, and constantly-evolving corpus, such as the live web, to collect key information (2) knowledge synthesis , to generate coherent long-form answers that surface key facts, integrating general knowledge and findings from many retrieved sources, and (3) verifiability , providing citations that allow readers to trace each stated claim in the synthesized answer to a reputable source from the retrieved set. The ideal benchmark must holistically evaluate across all three of these dimensions, while providing a realistic and challenging research synthesis task, which the community can reliably and repeatedly benchmark on over time. Unfortunately, existing benchmarks fall-short of these goals. Many prior works evaluate generative research synthesis systems using existing question answering benchmarks ( Wei et al. 2025 ; Wei et al. 2024 ; Mialon et al. 2023 ; Krishna et al. 2025 ; Wu et al. 2025 ; Wadden et al. 2020 ; Jin et al. 2019 ; Yang et al. 2018 ; Joshi et al. 2017 ; Kwiatkowski et al. 2019 ; Ho et al. 2020 ; Trivedi et al. 2022 ; Lee et al. 2023 ) , which do not reflect realistic research synthesis tasks and instead focus on questions with short-form, easily-verifiable answers, making them severely limited for this setting. These question-answering benchmarks do not capture the complexity of long-form answers synthesized from many sources, a key component of research synthesis. To address this limitation, several recent works ( Asai et al. 2024 ; Zheng et al. 2024 ; noa [n.d.]k ) instead leverage expert-curated datasets with open-ended research questions and exemplar answers. Unfortunately, these benchmarks quickly become stale and outdated as new information emerges. Furthermore, these datasets risk data contamination as new models are trained on snapshots of the web, including public datasets. The prohibitive expense of curating, maintaining, and updating expert-curated benchmarks further limits their utility towards realistic, scalable evaluation. To address this gap, we introduce DeepScholar-Bench, a live benchmark dataset and holistic evaluation framework tailored to tracking progress on generative research synthesis, as well as DeepScholar-Base, a reference method, which we hope will further future research. Our benchmark provides a live dataset with real research synthesis tasks, as well as a holistic, automated evaluation strategy. We design our benchmark dataset based on a fundamental research synthesis task reflected in thousands of research papers each year, where peer-reviewed manuscripts must concisely summarize existing work within a subfield, detailing and citing key prior works, before discussing new contributions. Specifically, the DeepScholar-bench task requires a system to generate a related work section for a given academic paper by retrieving sources from the Web and summarizing key prior work within a field of study. This task allows us to assess the key capabilities of research synthesis while ensuring that our evaluation reflects real, high-quality research. To source realistic and challenging queries, we develop an automated data pipeline that curates high-quality, recent ArXiv papers across diverse scientific domains. We plan to continuously re-run our data pipeline to provide an evolving dataset of live queries. To assess performance on DeepScholar-bench, we develop an automated evaluation framework, which holistically measures the performance of system answers across the three key dimensions: knowledge synthesis, retrieval quality and verifiability. Evaluating the accuracy of our long-form