Characterizing Deep Research: A Benchmark and Formal Definition
Abhinav Java, Ashmit Khandelwal, Sukruta Midigeshi, Aaron Halfaker, Amit Deshpande, Navin Goyal, Ankur Gupta, Nagarajan Natarajan, Amit Sharma · 2025
Read Characterizing Deep Research: A Benchmark and Formal Definition by Abhinav Java, Ashmit Khandelwal, Sukruta Midigeshi, Aaron Halfaker, Amit Deshpande, Navin Goyal, Ankur Gupta, Nagarajan Natarajan, Amit Sharma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04183.txt.
Introduction
Abhinav Java and co-authors define and test a new AI task in a 2025 paper: deep research. Jobs such as writing surveys or analytic reports need wide search and reasoning. Recent models claim this skill, but the task itself was never clearly defined.
The team argues the core mark is fan-out: a search must branch across many concepts, not merely produce a long report. They define the task through the key claims a search uncovers, apart from final writing style. Their benchmark holds 100 hard tasks on science and public events. Top systems scored unevenly and often poorly, which shows how much room the field still has to grow.
Catalog record
| CREATOR | Abhinav Java, Ashmit Khandelwal, Sukruta Midigeshi, Aaron Halfaker, Amit Deshpande, Navin Goyal, Ankur Gupta, Nagarajan Natarajan, Amit Sharma |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-04183
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Characterizing Deep Research: A Benchmark and Formal Definition Abhinav Java, Ashmit Khandelwal, Sukruta Midigeshi, Aaron Halfaker, Amit Deshpande, Navin Goyal, Ankur Gupta, Nagarajan Natarajan, Amit Sharma Abstract Information tasks such as writing surveys or analytical reports require complex search and reasoning, and have recently been grouped under the umbrella of deep research — a term also adopted by recent models targeting these capabilities. Despite growing interest, the scope of the deep research task remains underdefined and its distinction from other reasoning-intensive problems is poorly understood. In this paper, we propose a formal characterization of the deep research (DR) task and introduce a benchmark to evaluate the performance of DR systems. We argue that the core defining feature of deep research is not the production of lengthy report-style outputs, but rather the high fan-out over concepts required during the search process, i.e., broad and reasoning-intensive exploration. To enable objective evaluation, we define DR using an intermediate output representation that encodes key claims uncovered during search—separating the reasoning challenge from surface-level report generation. Based on this formulation, we propose a diverse, challenging benchmark LiveDRBench with 100 challenging tasks over scientific topics (e.g., datasets, materials discovery, prior art search) and public interest events (e.g., flight incidents, movie awards). Across state-of-the-art DR systems, F1 score ranges between 0.02 and 0.72 for any sub-category. OpenAI’s model performs the best with an overall F1 score of 0.55. Analysis of reasoning traces reveals the distribution over the number of referenced sources, branching, and backtracking events executed by current DR systems, motivating future directions for improving their search mechanisms and grounding capabilities. The benchmark is available at https://github.com/microsoft/LiveDRBench . 1 Introduction Deep Research (DR) has emerged as a popular paradigm for applying the reasoning capabilities of AI systems. DR systems have been shown to be useful for writing research reports ( Li et al. 2025 ) , business reports ( Microsoft and Spataro 2025 ) , and answering needle-in-a-haystack queries ( OpenAI 2025 ) . These tasks involve non-trivial reasoning that may take a human expert hours to complete and the expectation is that DR systems can significantly reduce that time ( FutureSearch et al. 2025 ; OpenAI 2025 ) . Figure 1: Characterizing the deep research task. Left (a) shows the landscape of various multi-hop reasoning tasks. Compared to existing tasks, the deep research task involves both high search and high reasoning intensity. Right (b) shows a stylized process of generating an answer to a DR query: DR query \rightarrow Claims \rightarrow Long form report. LiveDRBench focuses on the precision and completeness of the intermediate but crucial step of claim generation. Figure 2: Average F1 score for deep research models and search-enabled LLMs on LiveDRBench . Figure 3: Average F1 scores of deep research models split by the eight categories of LiveDRBench . OpenAI DR obtains the highest F1. Beyond demonstrative examples, however, no characterization of the problem exists. Typically, DR tasks are understood to be report-writing tasks that take non-trivial time for a human to complete. They are considered similar to reasoning-intensive search tasks, but expected to be much harder than those studied in the literature such as multi-hop QA ( Yang et al. 2018 ; Hongjin et al. ) . Interestingly, the hardness of a task also depends on the available document corpus. For example, “provide me a list of all Oscar-winning movies that were adapted from books with women authors” seems to be a hard, research-oriented task, but may not qualify as DR if there’s a webpage providing exactly this information. Overall, the lack of a formal definition makes it hard to evaluate DR models and measure progress. In this paper, we characterize the deep research task and provide a benchmark that can evaluate progress objectively. We propose that the DR task can be split into two subtasks: 1) synthesizing claims that collectively answer the user’s query, given a document corpus; and 2) writing a report based on these claims. By a claim, we refer to any piece of information that is relevant for answering a user’s query, wherein each claim may consist of sub-claims that support or provide evidence for the said information. Rather than the common expectation of a long report-like output, we posit that the defining element of a DR problem is the first sub-task: synthesis of relevant information that could answer a user’s question, given a document corpus. The information synthesis problem can be conceptualized as a directed acyclic graph (DAG), where the nodes represent information such as original query, retrieved documents and synthesized information while the edges represent actions such as issuing search queries and reasoning over retrieved documents (see Figure 1 ). Consequently, we can characterize a task by the structure of its search and reasoning DAG from the perspective of a human expert. Given a document corpus, we call a query a deep research query if 1) the number of information units to be processed for obtaining the final answer is high ( search intensity ); and 2) at least one of the tasks—finding these information units, processing them or combining them to form the final claims—requires non-trivial reasoning ( reasoning intensity ). An information unit roughly corresponds to an atomic piece of information, such as a paragraph or a chunk in a retrieval corpus. While quantifying search or reasoning intensity is subjective, we posit that DR corresponds to any query that takes more than 10 minutes for an ideal human expert. Assuming the entire web as the retrieval corpus, example problems include “all oscar-winning movies adapted from books by women authors with runtime 80-95 minutes” and “find a material having the following properties” . Examples of non-DR queries include “properties of the material used in this paper” (low search intensity) and “list of papers accepted at NeurIPS 2024” (low reasoning intensity). The above definition allows us to distinguish DR from commonly studied reasoning-intensive information tasks ( Krishna et al. 2024 ; Yang et al. 2018 ; Ho et al. 2020 ; Hongjin et al. ) in terms of the two axes: search and reasoning intensity (see Figure 1 ). Typical multi-hop QA benchmarks such as HotpotQA ( Yang et al. 2018 ) contain queries with low search and low reasoning intensity. Tasks such as scientific or legal QA given a document such as Curie ( Cui et al. ) and CUAD ( Hendrycks et al. 2021 ) respectively are reasoning-intensive but do not require a non-trivial search process. Other tasks such as explain-like-im-5 (ELI5) ( Fan et al. 2019 ) involve non-trivial reasoning but the search intensity may be limited since the question asks to explain a specific concept. In contrast, most DR tasks are the ones that are both search- and reasoning-intensive. For example, reasoning benchmarks such as GAIA ( Mialon et al. 2023 ) and humanity’s last exam ( Phan et al. 2025 ) qualify as both high search and high reasoning intensity DR tasks. Besides these task-completion benchmarks that are evaluated on a single final answer, typical information-synthesis DR applications such as writing a policy proposal, an academic survey, or an analytical report involve complex output with many aspects. We focus on the latter in this paper. Given the complex output, evaluating a