DeepAmbigQA : Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness

Jiabao Ji, Min Li, Priyanshu Kumar, Shiyu Chang, Saloni Potdar · 2025

Read DeepAmbigQA : Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness by Jiabao Ji, Min Li, Priyanshu Kumar, Shiyu Chang, Saloni Potdar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01323.txt.

Introduction

Jiabao Ji and colleagues benchmark whether language models with search tools give complete answers to ambiguous, multi step questions. A question like which actor from the film Heat won an Academy Award hides traps: several films share the title. The full cast must be reasoned over to gather every qualifying name.

Their data generation pipeline builds questions from text corpora linked to a knowledge graph, embedding name ambiguity and multi hop reasoning in a natural, verifiable way. The resulting data set holds thirty six hundred questions, half with explicit ambiguity. Even the strongest current model answered poorly, with low exact match scores on both ambiguous and unambiguous questions. The authors argue answer completeness is an under measured weakness of search augmented systems.

Catalog record

CREATORJiabao Ji, Min Li, Priyanshu Kumar, Shiyu Chang, Saloni Potdar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01323

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DeepAmbigQA : Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness Jiabao Ji, Min Li, Priyanshu Kumar, Shiyu Chang, Saloni Potdar Abstract Large language models (LLMs) with integrated search tools show strong promise in open-domain question answering (QA), yet they often struggle to produce complete answer set to complex questions such as “Which actor from the film Heat won at least one Academy Award?” , which requires ❶ distinguishing between multiple films sharing the same title and ❷ reasoning across a large set of actors to gather and integrate evidence. Existing QA benchmarks rarely evaluate both challenges jointly. To address this, we introduce DeepAmbigQAGen , an automatic data generation pipeline that constructs QA tasks grounded in text corpora and linked knowledge graph, generating natural and verifiable questions that systematically embed name ambiguity and multi-step reasoning. Based on this, we build DeepAmbigQA , a dataset of 3,600 questions requiring multi-hop reasoning and half of them explicit name ambiguity resolving. Experiments reveal that, even state-of-the-art GPT-5 show incomplete answers, achieving only 0.13 exact match on ambiguous questions and 0.21 on non-ambiguous questions. These findings highlight the need for more robust QA systems aimed at information gathering and answer completeness. \lst@InstallKeywords kattributesattributestyle attributestyleld 1 Introduction Large language models (LLMs) are increasingly used for complex question answering (QA), especially when paired with external search tools. Given an open-ended query, such systems decompose it into sub-steps, query the search tool, retrieve evidence, and aggregate information from multiple sources to form the final answer OpenAI 2025 ; Jin et al. 2025 ; Team et al. 2025 . Unlike simple factoid QA, which often requires a single lookup, complex queries are inherently multi-hop and demand multiple rounds of reasoning and evidence gathering Yang et al. 2018 ; Trivedi et al. 2022 ; Wolfson et al. 2025 . Figure 1 : Comparison between DeepAmbigQA questions (bottom), SimpleQA (top left), and AmbigQA (top right). Takeaway: Unlike SimpleQA and AmbigQA, DeepAmbigQA questions require both name disambiguation and multi-hop reasoning over a large intermediate evidence set, presenting a significant challenge for current LLM-based QA systems. Two core capabilities are crucial for QA systems to solve such questions: ❶ Complete evidence coverage i.e. , they must retrieve all relevant facts by issuing effective queries, disambiguating ambiguous inputs, and gathering the full set of related information rather than relying on a few salient snippets. ❷ Correct reasoning and information aggregation i.e. , they must then reason accurately over the collected evidence and perform reasoning operations such as filtering and grouping, while preserving contextual nuances. For example, the query: “Which actors in the film titled The Heat have won at least one Academy Award?” illustrates both challenges: the title The Heat is ambiguous, as multiple films share this name, and solving the query requires enumerating all candidate films, retrieving their casts, and verifying each actor’s award status. Thus, it demands both robust disambiguation and multi-hop reasoning over complete evidence. However, existing LLM-based QA systems often fall short, frequently overlooking valid interpretations or producing incomplete actor lists as shown in Figure 1 . Although such queries are common, they remain underrepresented in existing QA benchmarks. Multi-hop QA datasets such as MuSiQue and SimpleQA Trivedi et al. 2022 ; Wei et al. 2024 generally follow a single reasoning path and produce single short answer. In contrast, ambiguity-oriented datasets such as AmbigQA and CondAmbigQA Min et al. 2020 ; Li et al. 2025 capture multiple valid interpretations but involve limited multi-hop reasoning. Figure 1 illustrates how our target questions differ in both reasoning complexity and name ambiguity from those in prior benchmarks. To address this gap, we introduce DeepAmbigQAGen , an automatic pipeline for generating ambiguous multi-hop questions. Unlike prior QA dataset construction efforts that rely on manual annotation Trivedi et al. 2022 ; Wolfson et al. 2025 , DeepAmbigQAGen leverages knowledge graphs aligned with a text corpora such as Wikidata and Wikipedia, which provide structured relations among entities that can be directly converted into executable reasoning steps and verified against factual connections. DeepAmbigQAGen first identifies ambiguous names and their corresponding distinct entities, then builds executable reasoning plans with structured sequences of operations that model user information-seeking behaviors. These plans simulate multi-step reasoning by chaining operations such as entity retrieval ( e.g. , obtaining actors of a film) and filter application ( e.g. , selecting actors who have won an award). For ambiguous names, the same plan can be re-executed with each possible entity to obtain corresponding answers, allowing the dataset to capture multiple valid interpretations in a verifiable way. Finally, an LLM converts these plans into natural-language questions. We also integrates verification steps to ensure naturalness and question-answer correspondence. Building on DeepAmbigQAGen , we introduce DeepAmbigQA , a benchmark of 3,600 QA pairs. Among them, 1,800 questions contain ambiguous names leading to multiple valid reasoning paths, while the rest are non-ambiguous but still yield large answer sets (up to 43 entities). Each question requires at least two reasoning steps, with some involving up to eight, posing substantial challenges for LLM QA systems on complete answer recall and intermediate reasoning. We evaluate state-of-the-art LLMs equipped dense retrieval tool on Wikipedia corpus on DeepAmbigQA . The results show that: ❶ LLMs struggle with answer completeness, achieving low exact-match accuracy, especially for ambiguous queries, even the SOTA GPT-5 attains only 0.13 exact match on the ambiguous subset and 0.21 exact match on the non-ambiguous subset. ❷ While precision remains high (typically above 70%), recall and exact match are much lower, indicating their inability to find complete answers. Adding modules such as query expansion for disambiguation and evidence extraction for retrieved passage brings only modest improvements and cannot fully address these challenging questions. In summary, our contribution are as follows: • We propose DeepAmbigQAGen , an automatic QA synthesis pipeline grounded in knowledge graphs that generates natural questions and verifiable gold answers. • We introduce DeepAmbigQA , a benchmark of multi-hop, ambiguity-rich questions requiring reasoning and disambiguation to ensure answer completeness for QA systems. • We evaluate SOTA LLMs and demonstrate that they struggle with answer recall and completeness, highlighting the need for more robust QA systems. 2 Related Work Figure 2 : Illustration of the DeepAmbigQAGen algorithm. Beginning with a large collection of entities and relations, a complex query is constructed by iteratively sampling atomic entities and new operations. The resulting query is then converted into natural language by prompting an LLM. Ambiguous QA Ambiguous QA datasets focus on questions that can be interpreted in multiple valid ways. Existing benchmarks capture different types of ambiguity: AmbigQA, ASQA, and CondAmbigQA mainly address name entity and intent ambiguities Min et al. 2020 ; Stelmakh et al. 2022 ; Li et al. 2025 , while TempAmbigQA targets temporal ambiguity Piryani et al. 2024 . Although these datasets cover diverse ambiguity types, most of their questions require only shallow evidence collection without deep, multi-step reasoning. In contrast, our QA tasks involve complex, multi-step reasoning and information aggregation for various name entity interpretations in the question, which substantially increases the difficulty and richness of the questions. Multi-hop QA Multi-hop QA datasets typically contains questions requiring strong reasoning and evidence aggregation across documents. Datasets such as HotpotQA, 2WikiMultihopQA, and MuSiQue study reasoning over a limited number

Read the complete text