DR 3 -Eval: Towards Realistic and Reproducible Deep Research Evaluation
Qianqian Xie 1 , Qingheng Xiong 1 , He Zhu 2 , Tiantian Xia 5 , Xueming Han 3 , Fanyu Meng 3 , Jiakai Wang 2 , Zhiqi Bai 2 , Chengkang Jiang 1 , Zhaohui Wang 1 , Yubin Guo 1 , Yuqing Wen 4 , Jiayang Mao 1 , Zijie Zhang 1 , Shihao Li 1 , Yanghai Wang 1 , Yuxiang Ren 1 , Junlan Feng 3 , Jiaheng Liu 1 1 Nanjing University 2 M-A-P 3 Jiutian Research 4 National University of Singapore 5 Nanjing University of Science and Technology [email protected] [email protected] · 2026
Read DR 3 -Eval: Towards Realistic and Reproducible Deep Research Evaluation by Qianqian Xie 1 , Qingheng Xiong 1 , He Zhu 2 , Tiantian Xia 5 , Xueming Han 3 , Fanyu Meng 3 , Jiakai Wang 2 , Zhiqi Bai 2 , Chengkang Jiang 1 , Zhaohui Wang 1 , Yubin Guo 1 , Yuqing Wen 4 , Jiayang Mao 1 , Zijie Zhang 1 , Shihao Li 1 , Yanghai Wang 1 , Yuxiang Ren 1 , Junlan Feng 3 , Jiaheng Liu 1 1 Nanjing University 2 M-A-P 3 Jiutian Research 4 National University of Singapore 5 Nanjing University of Science and Technology [email protected] [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14683.txt.
Introduction
Xie and 18 colleagues built DR3-Eval, a benchmark that tests deep research agents on long, multi-file report tasks without letting the shifting live web spoil reproducibility. Each task pairs authentic user materials with a fixed sandbox corpus mixing supporting documents, distractors, and noise, and scoring spans recall, factual accuracy, citation coverage, instruction following, and depth. Trials with their multi-agent DR3-Agent showed the benchmark is hard, exposing weak retrieval and poor hallucination control.
Catalog record
| CREATOR | Qianqian Xie 1 , Qingheng Xiong 1 , He Zhu 2 , Tiantian Xia 5 , Xueming Han 3 , Fanyu Meng 3 , Jiakai Wang 2 , Zhiqi Bai 2 , Chengkang Jiang 1 , Zhaohui Wang 1 , Yubin Guo 1 , Yuqing Wen 4 , Jiayang Mao 1 , Zijie Zhang 1 , Shihao Li 1 , Yanghai Wang 1 , Yuxiang Ren 1 , Junlan Feng 3 , Jiaheng Liu 1 1 Nanjing University 2 M-A-P 3 Jiutian Research 4 National University of Singapore 5 Nanjing University of Science and Technology [email protected] [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58661 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14683
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DR 3 -Eval: Towards Realistic and Reproducible Deep Research Evaluation Qianqian Xie 1 , Qingheng Xiong 1 , He Zhu 2 , Tiantian Xia 5 , Xueming Han 3 , Fanyu Meng 3 , Jiakai Wang 2 , Zhiqi Bai 2 , Chengkang Jiang 1 , Zhaohui Wang 1 , Yubin Guo 1 , Yuqing Wen 4 , Jiayang Mao 1 , Zijie Zhang 1 , Shihao Li 1 , Yanghai Wang 1 , Yuxiang Ren 1 , Junlan Feng 3 , Jiaheng Liu 1 1 Nanjing University 2 M-A-P 3 Jiutian Research 4 National University of Singapore 5 Nanjing University of Science and Technology [email protected] [email protected] Abstract Abstract Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation remains challenging due to dynamic web environments and ambiguous task definitions. We propose DR 3 -Eval , a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation. DR 3 -Eval is constructed from authentic user-provided materials and paired with a per-task static research sandbox corpus that simulates open-web complexity while remaining fully verifiable, containing supportive documents, distractors, and noise. Moreover, we introduce a multi-dimensional evaluation framework measuring Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and validate its alignment with human judgments. Experiments with our developed multi-agent system DR 3 -Agent based on multiple state-of-the-art language models demonstrate that DR 3 -Eval is highly challenging and reveals critical failure modes in retrieval robustness and hallucination control. Our code and data are publicly available. 1 1 1 https://github.com/NJU-LINK/DR3-Eval 1 Introduction Recent advances in large language models have enabled the development of Deep Research Agents (DRAs), which aim to autonomously perform complex, long-horizon research tasks involving planning, iterative information retrieval, multimodal understanding, and synthesis of structured, citation-grounded reports ( 20 ; 46 ; 33 ; 44 ; 27 ; 14 ; 36 ; 2 ; 43 ; 10 ; 4 ; 6 ) . Unlike traditional question-answering systems, DRAs must operate under uncertainty, reason over heterogeneous and noisy information sources, and integrate evidence into coherent analytical outputs. As these capabilities rapidly improve, establishing realistic and reproducible evaluation protocols has become increasingly important. Evaluating deep research poses challenges that go beyond short-form reasoning or single-answer tasks. In realistic research settings, agents must infer implicit user intent from incomplete context, formulate effective search strategies, filter relevant evidence from large volumes of noise, and avoid hallucination while generating long-form reports ( 1 ; 40 ; 16 ; 34 ; 12 ; 17 ; 41 ; 28 ; 9 ; 19 ; 32 ) . As a result, evaluation outcomes are highly sensitive to the construction of tasks and information environments, revealing a fundamental tension between realism, controllability, and evaluability. Recently, as shown in Figure 1 , several benchmarks have been proposed to evaluate deep research agents. For example, DeepResearch Bench ( 12 ) emphasizes open-ended report generation and long-horizon reasoning, but typically relies on live web access, making results difficult to reproduce and prone to evaluation ambiguity. DRBench ( 1 ) improves structure by focusing on enterprise-style report generation from curated documents, yet largely omits explicit modeling of noisy or misleading information that is common in real-world research. More recently, DeepResearchGym ( 9 ) introduces a sandbox-based framework that replaces live web access with fixed local corpora, improving reproducibility, but its tasks are purely text queries and lack grounding in authentic user research workflows. As a result, there remains a gap between real-world research complexity—which involves multimodal user-provided materials, noisy and misleading information, and implicit research intent—and the environments used to evaluate DRA performance. Figure 1: Comparison of deep research benchmarks. Given raw text queries, Deep Research Bench executes text queries via real-time search, and DeepResearchGym retrieves from a global offline database. DRBench incorporates user files (text modality) as input but relies on real-time search and focuses on the enterprise domain. In contrast, our DR 3 -Eval processes both queries and files within a controlled sandbox corpus on diverse domains. To address these limitations, we introduce DR 3 -Eval , a benchmark designed to reconcile realism, controllability, and reproducibility for deep research evaluation. DR 3 -Eval targets report-generation tasks grounded in real user needs, constructed from authentic multimodal files that users have encountered in practice. Following the evaluation paradigm of DeepResearchGym ( 9 ) and BrowseComp-Plus ( 8 ) , we localize the retrieval corpus into a static, controlled sandbox rather than directly evaluating agents on the live web. Each task is paired with a per-case research sandbox corpus that simulates the open web while remaining fully static and verifiable. Within this sandbox, documents are carefully curated to include evidential sources, confounding documents, and ambient noise, enabling systematic analysis of an agent’s retrieval strategy, critical judgment, and robustness to distraction. Besides, a key feature of DR 3 -Eval is its reverse-construction methodology: instead of posing open-ended questions with uncertain answerability, we derive each query from verified evidential documents, ensuring that every task admits a single, well-defined solution path. This design eliminates evaluation ambiguity while preserving the complexity of real research workflows. To support fine-grained assessment, we propose a multi-dimensional evaluation framework that measures Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality. Moreover, to demonstrate the utility of DR 3 -Eval, we have developed DR 3 -Agent , a multi-agent research system adapted to the benchmark’s closed-world setting, which can also take text user queries and corresponding user files (including text, image, video, audio, etc.) as input. Extensive experiments across state-of-the-art language models reveal that DR 3 -Eval is highly challenging and exposes failure modes that are obscured by existing benchmarks. Overall, the contributions of our work are as follows: • We propose DR 3 -Eval, a realistic, reproducible, and multimodal benchmark for evaluating deep research agents for the report generation setting. • We introduce a controlled sandbox-based task construction pipeline that balances real-world complexity with verifiable evaluation. • We provide comprehensive experimental analysis and diagnostics that offer new insights into the strengths and limitations of current LLMs as DRAs. Figure 2: Overview of the DR 3 -Eval framework. (1) Data construction synthesizes search paths from real-world multimodal files via a divergent-convergent mechanism, establishing a static sandbox with controlled signal-to-noise ratios and backward-derived queries. (2) Our DR 3 -Agent adopts a hierarchical multi-agent architecture where a perception-enhanced Main Agent coordinates global reasoning while specialized sub-agents execute iterative sandbox retrieval and file parsing. (3) Evaluation protocol utilizes a multidimensional metric suite to comprehensively assess performance in both evidence acquisition and analytical report generation. 2 Related Work 2.1 Deep Research Agent Deep Research Agents (DRAs) are specialized systems designed for complex, multi-stage research tasks. Their capabilities have evolved beyond the traditional question-answering paradigm ( 7 ) to autonomously planning long-horizon workflows, navigating heterogeneous web sources, and ultimately synthesizing information into structured, citation-grounded, expert-level reports ( 27 ; 14 ; 36 ; 2 ; 43 ; 5 ) . Currently, mainstream deep research systems have demonstrated powerful capabilities in handling complex research tasks, but they are mostly available as closed-source commercial products; open-source efforts, in contrast, emphasize modularity and reproducibility ( 20 ; 46 ; 31 ;