Asking What Matters: Reward-Driven Clarification for Software Engineering Tasks

Sanidhya Vijayvargiya, Vijay Viswanathan, Graham Neubig · 2026

Read Asking What Matters: Reward-Driven Clarification for Software Engineering Tasks by Sanidhya Vijayvargiya, Vijay Viswanathan, Graham Neubig (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14624.txt.

Introduction

Vijayvargiya, Viswanathan, and Neubig study clarification in real software engineering tasks, where assistants must ask what users left unsaid. Not all missing information matters equally, and questions must target what users can realistically provide. Using Shapley values and distributional comparisons, they pin down two traits of good clarification: task relevance and user answerability. These properties become multi-stage reinforcement learning rewards training CLARITI, an 8-billion-parameter module matching GPT-5's issue-resolution rate on underspecified issues while asking 41 percent fewer questions.

Catalog record

CREATORSanidhya Vijayvargiya, Vijay Viswanathan, Graham Neubig
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87969
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14624

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Asking What Matters: Reward-Driven Clarification for Software Engineering Tasks Sanidhya Vijayvargiya, Vijay Viswanathan, Graham Neubig Abstract Humans often specify tasks incompletely, so assistants must know when and how to ask clarifying questions. However, effective clarification remains challenging in software engineering tasks as not all missing information is equally valuable, and questions must target information users can realistically provide. We study clarification in real software engineering tasks by quantifying which types of information most affect task success and which questions elicit useful responses from simulated users. Using Shapley attribution and distributional comparisons, we identify two key properties of effective clarification: task relevance (which information predicts success) and user answerability (what users can realistically provide). We operationalize these properties as multi-stage reinforcement learning rewards to train CLARITI, an 8B-parameter clarification module, that matches GPT-5’s resolution rate on underspecified issues while generating 41% fewer questions. Our results suggest that grounding reward design in empirical analysis of information impact and user answerability improves clarification efficiency. 1 Introduction Figure 1 : Our trained clarification model, CLARITI, achieves GPT-5-level performance (36.80%) with 41% fewer average questions (3.0 vs 5.1) by prioritizing task relevance and user answerability, demonstrating effective clarification at low user burden. Real-world user requests are frequently underspecified. Despite rapid improvements, LLM-based agents remain brittle when user instructions are underspecified or ambiguous, leading to wasted computation or task failure. Recent empirical studies highlight ambiguity and missing information as dominant failure modes for agents ( Zhang et al. 2024 ; Vijayvargiya et al. 2025 ) , underscoring the need for systems that can detect uncertainty and proactively seek additional information. A growing body of work demonstrates that clarification questions can substantially improve downstream outcomes ( Zhang & Choi 2025 ; Chen et al. 2025 ) , motivating recent work on user modeling that personalizes questions to individuals ( Sun et al. 2025 ) , or infers mental states ( Zhou et al. 2025 ) . We take a complementary, model-centric perspective: given typical user knowledge boundaries, how should models optimize clarification to maximize task success? Prior work provides limited understanding of which missing information is most valuable or how agents should prioritize clarification under interaction constraints. The analyses primarily target linguistic ambiguity—often involving a single point of intent-level uncertainty (e.g., resolving a referent or disambiguating an intent)—which does not capture the broader underspecification found in task-oriented settings where multiple types of information may be missing with varying importance ( Madge et al. 2025 ) . As a result, despite evidence that clarification helps, we lack a principled understanding of what information to ask about, whether users can answer the questions posed, and how these factors impact task success. We study clarification in software engineering (SWE), a domain where tasks require precise specifications and objective success can be measured via test suites. SWE issues often omit critical details such as environment configuration, expected behavior, or implementation constraints. An effective agent must prioritize among many possible information needs and identify which gaps genuinely block progress. We consider a controlled, single-turn setting where the agent poses clarifying questions to a simulated user before beginning implementation. Using separate modules for clarification generation and task execution, we isolate clarification quality from agent problem-solving capabilities allowing us to evaluate clarification independent of code-generation ability. This design exposes two fundamental dimensions: effective clarifications must target information likely to improve task success ( task relevance ) while remaining answerable by typical users given their knowledge about the issues they report. We demonstrate that the identified qualities for effective clarification are learnable and predictive of downstream success by training a clarification module, CLARITI (CLARIficiation with TIered rewards) 1 1 1 Code and data can be accessed at https://github.com/sani903/Teaching-Effective-Clarification , guided by these qualities that improves downstream agent performance (Figure 1 ). We first characterize the information needs common to underspecified SWE tasks, then investigate three research questions. Concretely, our contributions are: • RQ1: What information most impacts task success? We quantify the association between information category availability and downstream agent performance through Shapley value analysis across 700 underspecified SWE instances. Error information exhibits the strongest association, followed by implementation details and environment configurations, revealing a hierarchy where concrete diagnostic information contributes more to agent performance than abstract goal specifications. • RQ2: What makes questions answerable? We investigate characteristics that distinguish answerable from unanswerable clarification questions through distributional analysis of the two classes. We identify that answerable questions tend to ground in observable behaviors, maintain appropriate technical depth, and avoid requesting internal state users would not know. Analyzing clarification composition, we find that performance often plateaus or declines with more questions while imposing higher user burden. Excess questions can waste user effort without improving outcomes, and any informational gains from additional questions are offset by the introduction of irrelevant or unusable information into the context. • RQ3: Training with empirically-grounded rewards. We design a multi-stage reward pipeline targeting both task relevance (from RQ1’s impact hierarchy) and answerability (from RQ2’s distributional analysis), plus auxiliary criteria of non-redundancy and diversity. Our trained 8B-parameter module matches GPT-5’s performance on our task (88% of fully specified upper bound) while generating 41% fewer questions, demonstrating that effective clarification can be learned through principled, empirically-grounded reward design. Our study connects clarification quality directly to empirical task-solving outcomes, providing a methodology for identifying what information matters, understanding what makes questions answerable, and training models to optimize both dimensions. While our work focuses on software engineering, the framework combines empirical impact analysis, answerability characterization, and reward-driven training that can generalize to domains where agents rely on user-provided specifications. 2 Information Categorization Effective clarification requires understanding what information agents lack. While prior work studies linguistic ambiguity (e.g., referential or syntactic ambiguity), it does not categorize the types of missing information that affect agent-based task resolution. We address this gap by analyzing underspecified issues in SWE-Bench. We define an information need as information absent from the issue description that is required for an agent to produce a correct patch without relying on unverifiable assumptions. Table 1: We identify six categories of information needs from 112 underspecified SWE-Bench Verified annotations, ranging in frequency from occurring in as many as 65% of issues to only 3% of issues. Category Frequency Core Question Example Missing Information Error Information 65% What specific failure is occurring? Missing stack trace showing where code crashes; absent error message revealing constraint violation; unclear description of incorrect output format. Expected Behavior 33% What behavior should occur instead? Missing specification of correct return value; absent description of intended output format; unstated desired system state after operation completes. Implementation Details 37% What approach or constraints should guide the implementation? Missing step-by-step guidance on achieving expected behavior; lack of exact functions to modify and how they should be modified; unclear whether to change existing logic or implement from scratch. External References 15% What external context is needed to understand requirements? Inaccessible API documentation link; missing reference to upstream library behavior; absent dataset schema or format specification. Reproduction Steps 12% Under what conditions does the issue manifest? Missing command-line invocation that triggers the bug; absent minimal code example demonstrating the failure; unclear input that produces incorrect behavior. Version/Environment 3% What configuration details affect the issue? Missing dependency version that

Read the complete text