QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?

Belinda Z. Li, Been Kim, Zi Wang · 2025

Read QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks? by Belinda Z. Li, Been Kim, Zi Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22674.txt.

Introduction

Real questions rarely come with all the facts attached, so a good reasoner must know what to ask. Belinda Li, Been Kim, and Zi Wang test that skill in a 2025 benchmark where language models face problems missing exactly one piece of information.

The tasks span logic puzzles, planning problems, and grade-school math, each with several candidate questions to choose from. Models did well on the math sets. On logic and planning, though, accuracy sat at only forty to fifty percent. Strikingly, models that could solve the complete versions still failed to spot the one question that mattered. Asking well, the authors argue, is a separate skill that needs direct training.

Catalog record

CREATORBelinda Z. Li, Been Kim, Zi Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE119 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22674

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks? Belinda Z. Li, Been Kim, Zi Wang Abstract Large language models (LLMs) have shown impressive performance on reasoning benchmarks like math and logic. While many works have largely assumed well-defined tasks, real-world queries are often underspecified and only solvable by acquiring missing information. We formalize this information-gathering problem as a constraint satisfaction problem (CSP) with missing variable assignments. Using a special case where only one necessary variable assignment is missing, we can evaluate an LLM’s ability to identify the minimal necessary question to ask. We present QuestBench , a set of underspecified reasoning tasks solvable by asking at most one question, which includes: (1) Logic-Q: logical reasoning tasks with one missing proposition, (2) Planning-Q: PDDL planning problems with partially-observed initial states, (3) GSM-Q: human-annotated grade school math problems with one unknown variable, and (4) GSME-Q: equation-based version of GSM-Q. The LLM must select the correct clarification question from multiple options. While current models excel at GSM-Q and GSME-Q, they achieve only 40-50% accuracy on Logic-Q and Planning-Q. Analysis shows that the ability to solve well-specified reasoning problems is not sufficient for success on our benchmark: models struggle to identify the right question even when they can solve the fully specified version. This highlights the need for specifically optimizing models’ information acquisition capabilities. Code and dataset are available at https://github.com/google-deepmind/questbench . \reportnumber 001 1 Introduction Figure 1: A multi-choice question-asking task in QuestBench with ground truth answers for accuracy evaluation. We construct question choices using the CSP translated from the verbal problem. Large language models (LLMs) are increasingly being applied to reasoning tasks such as math ( Cobbe et al., 2021 ; Hendrycks et al., 2021 ; Li et al., 2024a ) , logic ( Zhang et al., 2023 ; Chen et al., 2024b ; Creswell et al., 2022 ) and planning/coding ( Curtis et al., 2024a ; Silver et al., 2024 ; Wang et al., 2024a ; Austin et al., 2021 ; Chen et al., 2021 ) . These applications typically assume all necessary information has been provided. However, real-world scenarios often violate this assumption. Users may omit crucial details in math problems, and robots might operate in environments with partial observability. In such cases, LLMs need the ability to proactively gather missing information, for instance, by asking clarifying questions. This work investigates whether LLMs can identify and acquire the missing information necessary to solve reasoning tasks. We formalize this information-gathering problem as an underspecified Constraint Satisfaction Problem (CSP). Our key idea is that many reasoning tasks can be modeled as determining the value of a target variable given a set of variables and constraints. A problem is underspecified if and only if the value of the target variable cannot be inferred from the given information. This formalization helps pinpoint the difference between semantic ambiguity (where multiple valid interpretations exist, but each yields a solvable answer) and underspecification (where the problem is unsolvable without additional information). We address the latter in this work. Figure 1 shows an example of an underspecified CSP derived from a grade school math problem. We focus on a simple special case of underspecified CSPs where we only need to acquire one unknown variable’s value in order to solve for the target variable. We call these “1-sufficient CSPs” and construct QuestBench : a benchmark of question asking problems based on 1-sufficient CSPs. The model is given an underspecified problem and multiple choices for which variable to ask about. It must pick a relevant variable, one whose value isn’t already known (or derivable from the existing assigned variables), and is sufficient for computing the target variable. QuestBench has the following multi-choice problems, along with the correct solutions for each: • Logic-Q : Logical reasoning tasks where the truth value of a missing proposition is needed to determine the correctness of a claim. • Planning-Q : Blocks world planning problems in Planning Domain Definition Language (PDDL) ( Ghallab et al., 1998 ) , with partially observed initial states, where one additional observation is needed to disambiguate the shortest path to a goal. • GSM-Q/GSME-Q : Grade-school math problems that are missing conditions needed to derive the solution. GSM-Q consists of verbalized forms of problems, while GSME-Q consists of equation forms of problems. Both are annotated by humans. We benchmark state-of-the-art LLMs including GPT-4o ( OpenAI, 2023 ) , GPT-4-o1 Preview, Claude 3.5 Sonnet, Gemini 1.5 Pro and Gemini 2.0 Flash Thinking Experimental ( Gemini Team Google, 2023 ; Gemini Team Google, 2024 ) on QuestBench . While these models demonstrate strong performance on GSM-Q and GSME-Q (over 80% accuracy), their performance on Logic-Q and Planning-Q is significantly lower (under 50% accuracy). To investigate these discrepancies, we analyze the correlation between model accuracy and four “axes of difficulty” in QuestBench (the number of variables in the problem, the number of constraints in the problem, the search depth required by a backwards search solution to the problem, and the expected number of guesses required by brute-force search solution to the problem), finding differing trends between domains. We also conduct ablation studies to assess LLMs’ ability to 1) reason in the presence of sufficient information and 2) detect whether the problem is underspecified. We find that success on QuestBench requires more than just the ability to reason with sufficient information. Our contributions include (1) a constraint satisfaction-based problem formulation of asking clarification questions for reasoning, (2) QuestBench , a novel benchmark with multi-choice problems for evaluating information-gathering abilities in reasoning tasks, (3) empirical evaluations and detailed analyses of SOTA or near-SOTA LLMs 1 1 1 As of 03/2025. on QuestBench . 2 Related work Importance of Information Gathering. Actively gathering information is a critical skill for both humans and autonomous systems when solving problems under uncertainty. For humans, Chouinard et al. (2007) demonstrates that asking questions plays a crucial role in cognitive development, helping children resolve ambiguity and learn more effectively. Similarly, AI/ML researchers have extensively explored strategies for sequential data acquisition in various domains, including concept learning ( Sammut and Banerji, 1986 ; Angluin, 1988 ) , active learning ( Cohn et al., 1996 ; Settles, 2009 ; Houlsby et al., 2011 ; Gal et al., 2017 ; Ren et al., 2021 ) , Bayesian optimization ( Kushner, 1962 ; Kushner, 1964 ; Moc̆kus, 1974 ; Auer, 2002 ; Srinivas et al., 2010 ; Hennig and Schuler, 2012 ; Wang and Jegelka, 2017 ; Garnett, 2023 ; Chen et al., 2022 ; Fan et al., 2024 ; Wang et al., 2024b ; Feurer et al., 2018 ) , reinforcement learning ( Kaelbling et al., 1996 ; Ghavamzadeh et al., 2015 ; Sutton, 2018 ) , and robot planning with partially observable states ( Kaelbling et al., 1998 ; Kaelbling and Lozano-Pérez, 2013 ; Phiquepal and Toussaint, 2019 ; Curtis et al., 2024b ) or unknown models ( Wang et al., 2017 ; Wang et al., 2018 ; Wang et al., 2021 ) . Wu (2023) emphasizes the need for AI assistants to ask questions in the face of uncertainty, specifically in the context of code generation. Our work follows this tradition by evaluating how well large language

Read the complete text