When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search
Yiling Tao, Shihan Deng, Meiling Tao, Pengzhi Wei, Zhichao Hu, Zhihao Zhu · 2026
Read When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search by Yiling Tao, Shihan Deng, Meiling Tao, Pengzhi Wei, Zhichao Hu, Zhihao Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27669.txt.
Introduction
Search agents answer complex questions by retrieving and reasoning in steps. Real user requests are often vague or partly wrong, though. Ambiguity can send a long search chain down the wrong path. Yiling Tao and five colleagues built DiscoBench in 2026 to test how agents handle that. The benchmark checks whether agents notice ambiguity and ask for clarification.
DiscoBench holds 211 samples with 463 ambiguity instances. They span eleven real-world domains and four ambiguity types. A user simulator supports multi-turn interaction during tests. Scoring covers task utility, ambiguity detection, interaction strategy, and cost efficiency. Experiments on representative large language models reveal a split. Detecting ambiguity and clarifying it well are distinct capabilities. Repeated searching instead of asking often did worse than direct guessing. Proactive clarification proved far more effective in the reported comparisons. Retrieval skill alone does not make an interactive search agent.
Catalog record
| CREATOR | Yiling Tao, Shihan Deng, Meiling Tao, Pengzhi Wei, Zhichao Hu, Zhihao Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 99 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27669
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search Yiling Tao, Shihan Deng, Meiling Tao, Pengzhi Wei, Zhichao Hu, Zhihao Zhu Abstract Search agents powered by large language models (LLMs) are increasingly used to solve complex information-seeking tasks, requiring multi-step retrieval and reasoning to fulfill user goals. However, existing benchmarks often assume that user queries are complete and explicit, overlooking the fact that real-world search requests are frequently vague, underspecified, or even factually incorrect. In deep search scenarios, such ambiguity can propagate along multi-step reasoning chains and lead agents toward incorrect search trajectories. To address this gap, we introduce DiscoBench , a benchmark for clarification-aware deep search, designed to evaluate whether search agents can proactively identify ambiguity, ask effective clarification questions, and recover correct reasoning paths through user interaction. DiscoBench contains 211 samples and 463 ambiguity instances across 11 real-world domains, covering four ambiguity types. We further design a user simulator for multi-turn interaction and evaluate model performance from four perspectives: task utility, ambiguity detection, interaction strategy, and cost efficiency. Experiments on representative LLMs show that ambiguity detection and effective clarification are distinct capabilities, and that repeatedly searching instead of asking for clarification often performs worse than direct guessing, highlighting a critical gap between retrieval ability and interactive problem-solving in current search agents. 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Corresponding authors. Correspondence to [email protected] 4 4 footnotetext: Work done during an internship at Tencent Hunyuan. 1 Introduction In recent years, search agents based on Large Language Models (LLMs) have made significant progress in the field of information retrieval Mialon et al. 2023 ; Wei et al. 2025 ; Wong et al. 2025 . The search paradigm is shifting from traditional static corpus retrieval to autonomous Web Search Agents capable of handling complex goals Xi et al. 2025 ; OpenAI 2025 ; Google 2025 . These agents can simulate human navigation and browsing behaviors, achieving multi-step reasoning and information integration in dynamic and complex internet environments Wu et al. 2025 ; Zhou et al. 2024 . Figure 1: A motivating example of ambiguity propagation in interactive deep search. However, current paradigms often presuppose that the user’s initial query is complete and explicit. This assumption deviates significantly from real-world information-seeking behavior, where users often provide vague or fragmented queries due to blurred memories or cognitive load limitations Aliannejadi et al. 2019 ; Zamani et al. 2020 . In deep search scenarios, the impact of this discrepancy is further amplified. Unlike traditional single-hop retrieval Kwiatkowski et al. 2019 , deep search involves complex multi-step reasoning chains Trivedi et al. 2022 , meaning any subtle ambiguity in the initial query can lead to cascading errors in subsequent navigation and information integration, wasting expensive computational resources on the wrong path. As illustrated in Fig. 1 , failing to proactively clarify ambiguous checkpoints further propagates errors throughout the remaining search process. Consequently, introducing interactive clarification mechanisms to resolve ambiguity has become particularly important. Meanwhile, search tasks naturally provide strong factual grounding, allowing both interaction quality and retrieval correctness to be objectively verified through external evidence. This property offers a reliable signal for evaluating an agent’s ability to identify and resolve ambiguity in complex interactive settings. While the academic community has recognized the importance of query ambiguity and interaction, existing benchmarks still struggle to evaluate the disambiguation capabilities of search agents. Mainstream retrieval benchmarks (e.g., GAIA Mialon et al. 2023 , BrowseComp Wei et al. 2025 , AgentBench Liu et al. 2024 ) mostly assume explicit queries and focus on multi-hop reasoning while neglecting proactive interaction. Ambiguity-focused datasets (e.g., AmbigQA Min et al. 2020 , DEEPAMBIGQA Ji et al. 2025 ) primarily consist of static scenarios and lack dynamic interaction simulation, whereas interaction-based benchmarks (e.g., IN3 Qian et al. 2024 , UserBench Qian et al. 2025 ) are often confined to closed sandbox environments, falling short in the depth and breadth of Web-scale open-domain knowledge. The recent INTERACTCOMP Deng et al. 2025 has begun to address search interaction, yet it remains limited in terms of task authenticity, the amplification effects of ambiguity within long-chain reasoning, and the naturalness of interaction modalities. To bridge these gaps, we introduce DiscoBench ( D eep I nteractive S earch with C larificati O n Benchmark) 1 1 1 Our code and data will be publicly released soon. , a benchmark for evaluating whether search agents can proactively clarify and resolve ambiguity during multi-step search. Unlike prior benchmarks that mainly focus on static query understanding, DiscoBench models ambiguity as a dynamic phenomenon arising during multi-step search trajectories. At each ambiguous checkpoint, agents must proactively identify underspecified information and interact with the user to obtain discriminative clues, rather than relying on direct guessing or closed-form option selection strategies. We conduct experiments on DiscoBench across a set of representative LLMs. The results show that current search agents still struggle to determine when clarification is needed: even stronger models often fail to recognize ambiguity during the search process or ask effective clarification questions. This suggests that deep interactive search requires not only stronger retrieval and reasoning abilities, but also better ambiguity awareness and proactive clarification strategies. Our main contributions are as follows: • We construct DiscoBench , a benchmark that models ambiguity as a dynamic phenomenon propagating along multi-step reasoning chains rather than a static property of individual queries, covering 211 samples with 463 ambiguity instances across 11 real-world domains and four ambiguity types. • We propose an ambiguity-aware evaluation framework for multi-turn interactive deep search, together with a user simulator that progressively reveals discriminative clues, enabling unified evaluation of ambiguity detection, clarification effectiveness, and interaction cost. • Through extensive experiments, we reveal that ambiguity detection and effective clarification are distinct capabilities. We further identify a dominant failure mode in which models repeatedly continue searching instead of asking for clarification, leading to lower success rates than direct guessing. This finding highlights the need for mechanisms that explicitly bridge retrieval uncertainty and user interaction. Figure 2: Overview of the proposed interactive retrieval framework and evaluation protocol. 2 Related Work 2.1 Web Search Benchmark Efforts to benchmark search agents often bifurcate into two dimensions. One branch focuses on reasoning depth, challenging agents to navigate complex web hierarchies for multi-hop tasks, such as GAIA Mialon et al. 2023 and the BrowseComp series Wei et al. 2025 ; Zhou et al. 2025a ; Chen et al. 2025 . In parallel, other benchmarks explore information width, necessitating the synthesis of vast horizontal data, as seen in PaSa He et al. 2025 , SPAR Shi et al. 2025 , and WideSearch Wong et al. 2025 . Complementary agent benchmarks like WebArena Zhou et al. 2024 , VisualWebArena Koh et al. 2024 , Mind2Web Deng et al. 2023 , and WebShop Yao et al. 2022 further evaluate web navigation capabilities in realistic environments. Recent work, such as DeepWideSearch Lan et al. 2025 , has further begun to encompass both dimensions. Constrained by the assumption of complete queries, these benchmarks leave query ambiguity unexplored, which remains an essential aspect of autonomous search. 2.2 Ambiguity Benchmark Research on query ambiguity has evolved into several taxonomies. Semantic and structural ambiguity benchmarks represented by AmbiEnt Liu et al. 2023 investigate logical divergences arising