Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions

Harsh Darji, Thibaud Lutellier · 2025

Read Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions by Harsh Darji, Thibaud Lutellier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21285.txt.

Introduction

Harsh Darji and Thibaud Lutellier build a coding helper that asks before it guesses. Their 2025 paper notes that vague prompts lead coding models astray, since intent hides behind few words. Their system pairs two parts: a classifier that flags unclear coding asks, and a tuned language model that poses clarifying questions, much as a human reviewer would.

The tuned model beat plain prompting at useful questions. In a user study, its questions also beat the baseline's, and the full helper yielded truer, more helpful code than stock assistants. Asking first beats silent guessing, they conclude.

Catalog record

CREATORHarsh Darji, Thibaud Lutellier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-21285

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions Harsh Darji, Thibaud Lutellier Abstract Large Language Models (LLMs) are increasingly used as coding assistants. However, the ambiguity of the developer’s prompt often leads to incorrect code generation, as current models struggle to infer user intent without extensive prompt engineering or external context. This work aims to build an LLM-based coding assistant that mimics the human code review process by asking clarification questions when faced with ambiguous or under-specified queries. Our end-to-end system includes (1) a query classifier trained to detect unclear programming-related queries and (2) a fine-tuned LLM that generates clarification questions. Our evaluation shows that the fine-tuned LLM outperforms standard zero-shot prompting in generating useful clarification questions. Furthermore, our user study indicates that users find the clarification questions generated by our model to outperform the baseline, demonstrating that our coding assistant produces more accurate and helpful code responses compared to baseline coding assistants. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable potential as coding assistants by translating informal natural language (NL) intents into functional code (code generation) [ 38 ] . However, the inherent ambiguity of NL makes it challenging to ensure that the generated code fully aligns with the user’s intended functionality. Much work highlights the current limitations of LLMs, especially regarding their intent understanding capacities, which may lead to vague, incorrect, or hallucinated answers [ 31 , 49 ] . Specifically, prior efforts to improve intent specification in coding assistants include prompt engineering techniques and the inclusion of external structured information, such as test cases [ 13 ] . While these strategies enhance the clarity of the prompt and the quality of the AI-generated answer, they place the burden on the user, who needs to guess how much input, test cases, or documentation to supply—an approach that can be both costly (due to prompt length constraints) and impractical. Moreover, such strategies do not fully resolve ambiguity and may introduce irrelevant or distracting information. A more natural approach mirrors how human assistants and pair programming work: by asking follow-up questions when presented with unclear input. Prior research suggests that dialog-style refinement—where an LLM asks clarification questions—can improve outcomes in code generation tasks [ 31 , 5 , 36 ] . This mirrors real-world interactions on platforms like StackOverflow, where clarification often precedes useful answers. However, existing systems rely on zero-shot or few-shot prompting strategies to simulate such behavior. These methods lack robustness since they are not explicitly trained to detect and handle ambiguous input. 1.1 Motivating Example Figure 1 is a typical example of a prompt that a JavaScript software developer might submit to an LLM for code generation. While the prompt appears reasonable, it is under-specified. The developer implicitly expects the LLM to address the TODO comment by generating the code required to fetch user data. However, when this prompt is sent to our baseline model, Google’s Gemma-3-1B-IT [ 42 ] , the model fails to produce the expected code. Instead, it explains in English the purpose of this function. This highlights that the LLM did not correctly capture the user’s intent. Instead of generating an unhelpful description of the code, it would have been more effective if the LLM recognized the ambiguity of the prompt and asked the user for additional information. Figure 1 : Example of under-specified prompt to an LLM. 1.2 Our Work We propose a new end-to-end coding assistant that asks clarification questions when input is ambiguous or under-specified, mirroring how an experienced human developer might seek to better understand the intent behind a vague coding request. Our solution has two key components; (1) an intent clarity classifier that detects under-specified coding prompts; and (2)a fine-tuned LLM that generates clarification questions before producing a final code answer. These two components are combined into a single pipeline. When a user submits a natural language coding prompt, the system first applies the intent clarity classifier to determine whether the prompt is sufficiently clear for immediate code generation. If the classifier predicts the prompt is under-specified, the system defers code generation and instead uses the fine-tuned LLM to generate a clarification question. Only after the user provides a follow-up clarification does the assistant proceed to generate the final code output. If the original input is deemed clear, the assistant skips the clarification step and generates code directly. Our evaluation aims to answer two research questions: RQ1: Can our new fine-tuned clarification module generate better clarification questions than a standard LLM baseline? RQ2: Does our approach yield more precise and useful final code responses than standard LLM pipelines? Our paper makes the following contributions: Contribution 1. We fine-tune two new AI models with two new datasets: (1) a coding intent clarity classifier and (2) a coding clarification question generator tailored to under-specified coding queries. Contribution 2. We integrate these models into a coding assistant that prompts the user for clarification before answering coding questions. Contribution 3. Our evaluation shows that users prefer the final answer provided by our model over the baseline in the vast majority of cases (82% for precision and focus, 78% for contextual fit, and 80% for answer faithfulness). Our code, datasets, user study templates, and models are publicly available in supplementary material. 2 Related Work Our work is at the intersection of natural language processing and software engineering. We group related work into three different categories: General Conversational Assistants, Coding Assistants, and Intent Clarification. 2.1 General Conversational Assistant Conversational assistants leveraged LLMs to deliver more engaging, adaptive, and context-aware interactions in multiple domains [ 9 , 34 , 10 , 30 ] . Gu et al. [ 16 ] explored how prompt-based control over personality dimensions can yield distinct conversational styles. Similarly, Zhang et al. [ 48 ] surveyed the evolution of conversational search systems, highlighting how LLMs enable complex, multi-turn interactions through modules like query reformulation, clarification, and response generation—laying the groundwork for naturalistic and informative dialogue systems. While such assistants has been well research and are becoming common use, the challenge of unclear or underspecified queries and user intent has risen as a limitation of such assistants. 2.2 Coding Assistants Much work explores how LLMs can be integrated into programming environments to support tasks such as code generation, understanding, debugging, and planning [ 44 , 23 , 51 , 26 , 20 , 6 , 33 ] . PairCoder [ 47 , 36 ] draws from pair programming to coordinate two LLM agents. Similarly, ROBIN [ 4 ] introduces a specialized debugging assistant that uses an investigate-and-respond interaction model within GitHub Copilot Chat. Others aim to ground LLM assistants in the user’s workflow [ 39 , 14 , 41 , 28 , 19 , 38 ] . TiCoder [ 14 ] introduces a test-driven, interactive loop that supports intent clarification via tests. In a similar spirit, Mathews et al. [ 28 ] show that providing test cases upfront can improve the accuracy of LLM outputs. Meanwhile, CodeLutra [ 41 ] emphasizes learning from both correct and incorrect attempts using preference-guided refinement, improving smaller open-source models’ performance without extensive retraining. Our work differs from prior research by focusing on training LLMs to ask clarification questions, rather than assuming user intent

Read the complete text