Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions
Jinyan Su, Claire Cardie · 2026
Read Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions by Jinyan Su, Claire Cardie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25284.txt.
Introduction
Jinyan Su and Claire Cardie probe a gap between spotting ambiguity and acting on it in language-model assistants. Models often recognize that a question admits several readings when asked directly. Yet in ordinary question answering they mostly answer straight away instead of seeking clarification. Added retrieved context widens the split: it makes questions easier to answer while making clarifying questions even rarer. The study tests ambiguous, clear, and disambiguated questions across three settings, with a judge model sorting replies into answers, refusals, and clarifications.
Catalog record
| CREATOR | Jinyan Su, Claire Cardie |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36370 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.25284
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions Jinyan Su, Claire Cardie Abstract User queries are often underspecified and may admit multiple valid interpretations. Rather than silently making assumptions about the user’s intent, a helpful assistant should surface such ambiguity by asking a clarifying question. Doing so requires two abilities: recognizing that a query is ambiguous, and acting on that recognition by seeking clarification instead of answering directly. To study these abilities, we evaluate models on ambiguous, unambiguous, and disambiguated questions in three settings: standard question answering, explicit ambiguity judgment, and behavioral analysis, where a judge model classifies responses as direct answers, refusals, or clarifying questions. We find a clear gap between recognition and behavior: models often identify ambiguity when explicitly asked to judge it, yet in the QA setting they overwhelmingly default to direct answers. Retrieved context further widens this gap by improving answerability while making models even less likely to ask clarifying questions. 1 Introduction Large language models (LLMs) are now used at scale to answer questions, provide advice, and support decision making for a wide range of users ( 27 ; 25 ; 1 ; 23 ) . In these interactions, user queries are often not fully specified: they may leave out crucial details, rely on unstated context, or be compatible with multiple plausible interpretations ( 26 ) . For such ambiguous queries, simply producing a single “best guess” answer forces the model to make hidden assumptions about the user’s intent, which can misrepresent the underlying task and propagate errors into downstream decisions. Instead, an effective assistant should actively seek missing information. To do so, the model first needs ambiguity awareness: the ability to recognize that a query is ambiguous or incomplete and therefore unsafe to answer directly. It should then convert this awareness into clarification behavior, by making its uncertainty visible, either by asking a clarifying question or by explicitly refusing to answer unless more information is provided, rather than silently committing to one assumed interpretation. Category Concise Description Example Temporal Unspecified Time “Who was the president?” (Depends entirely on the year ) Identity Multiple entities share the same name. “Who is Ben Stone?” (Several unrelated characters share this name.) Version Multiple instantiated forms of the same entity “Who played Annie?” (Different actresses in the 1982, 1999, and 2014 versions.) Scope Ambiguity involving multiple levels of granularity within the same domain or entity. “Where did fighting happen in WWI?” (Europe? Western Front? A single battle?) Semantic Language allows multiple interpretations “What is the flower of the dead in Spanish?” (Literal translation vs. cultural symbol.) Locale Unspecified geographic region “When does MasterChef Junior start in 2018?” (Different air dates in US vs. UK.) Other - - Table 1: Concise descriptions and representative examples for the ambiguity categories. In this paper, we investigate ambiguity awareness and clarification behavior in LLMs. To probe ambiguity awareness, we ask LLMs to make explicit judgments about the query itself with or without retrieved context: decide whether a question is ambiguous and assign it to a category from our ambiguity taxonomy. On the behavior side, we examine whether this awareness carries over when model are directly prompted with the query: given the same questions, does the model actually act on its ambiguity judgments by asking for clarification or indicating that more information is needed. Figure 1 : Overall performance in QA setting for unambiguous (left column) and ambiguous queries (right column) and also with (top row) and without (bottom row) retrieved contexts. We find three consistent effects. First, when the model is directly prompted with the query, the presence of retrieved context substantially improves QA accuracy across all three question categories (ambiguous, unambiguous, and disambiguated). Second, when explicitly asked to judge ambiguity, models exhibit non-trivial ambiguity awareness: they can often recognize that a query is ambiguous, and additional context sometimes helps these judgments, though the gains are modest and less uniform than the more reliable improvements observed for QA accuracy. Nevertheless, this awareness remains largely latent: even when models recognize that a query is ambiguous in this explicit judgment setting, they do not reveal that uncertainty unless they are asked to. Third, the presence of retrieved context makes models less likely to ask clarifying questions or indicate that more information is needed, regardless of whether the question itself is ambiguous or not. Taken together, these findings instantiate a “knowing but not showing” pattern: models can detect ambiguity to some extent, but this awareness does not reliably surface in their default answering behavior, and is further suppressed when context is provided. From a reinforcement learning perspective ( 18 ) , current training pipelines incentivize LLMs to reveal whatever yields high reward: producing responses that human annotators rate as helpful and useful. When expressing ambiguity awareness is not aligned with high reward, models learn to hide that awareness in their default behavior.This has two important implications. First, to make models more truthful and reliable under ambiguity, we need training objectives that positively reward expressing uncertainty, asking clarification questions, and explicitly acknowledging under-specification, rather than only encouraging answer accuracy. Second, when probing what LLMs “know” or estimating their capabilities, we must be careful about how we formulate the task: the very same model can display quite different levels of ambiguity awareness depending on whether it is prompted to answer the question or to analyze the question itself. Our study makes this discrepancy visible and points toward training and evaluation schemes that better align a model’s internal awareness with its outward behavior. Although we focus on ambiguity awareness, the same reasoning applies more broadly to other model behaviors we care about, wherever truthfully expressing what the model knows is not perfectly aligned with reward. 2 Related Work Figure 2 : Performance gap of unambiguous query and ambiguous query in QA setting: model generally have a better performance on unambiguous questions with or without context. Figure 3 : The impact on the performance with and without retrieved contexts: having retrieved context improves the performance for both ambiguous and unambiguous queries. Ambiguous Question Answering Early work has highlighted the prevalence of ambiguity in open-domain QA. ( 17 ) introduce AmbigQA, showing that over half of the questions in Natural Questions are underspecified and have multiple plausible answers. Follow-up studies confirm that even state-of-the-art models struggle on such ambiguous queries. For instance, ( 24 ) and ( 16 ) find that LLMs often underperform when questions admit multiple interpretations. ( 22 ) present ASQA, extending AmbigQA by providing each ambiguous question with a disambiguating context and long-form answers. In the conversational domain, ( 8 ) presented a dataset for detecting ambiguity in multi-turn QA and generating clarifying questions. Other datasets like CambigNQ ( 15 ) target single-turn ambiguous questions, while multi-hop ambiguity has been addressed by MIRAGE ( 19 ) for questions requiring reasoning across multiple documents. Clarifying Questions One prominent approach to handle ambiguity is to ask clarifying questions. In information-seeking dialogues, researchers have built datasets where systems must pose a follow-up question to clarify user intent. ( 3 ) extend the Qulac dataset ( 4 ) with ClariQ ( 2 ) , crowdsourcing single-turn conversational queries along with appropriate clarifying questions and answers. Similarly,