Teaching Language Models To Gather Information Proactively
Tenghao Huang, Sihao Chen, Muhao Chen, Jonathan May Longqi Yang Mengting Wan Pei Zhou · 2025
Read Teaching Language Models To Gather Information Proactively by Tenghao Huang, Sihao Chen, Muhao Chen, Jonathan May Longqi Yang Mengting Wan Pei Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21389.txt.
Introduction
Huang and six co-authors taught language models to ask questions before answering in twenty twenty-five. Real requests are often ambiguous or missing key facts, yet today's models tend to answer passively or ask shallow clarifications instead of gathering what they need.
The team defines a new task, proactive information gathering, and generates partially specified real-world tasks by masking key information. Their training strategy uses reinforcement learning that rewards questions uncovering genuinely new, unspoken user knowledge, such as hidden expertise or fine-grained requirements. A trained seven-billion-parameter model beat a strong commercial mini model by eighteen percent on automatic metrics. Human judges preferred its clarification questions by forty-two percent and its final outlines by twenty-eight percent. The authors argue proactive questioning turns models into collaborative partners.
Catalog record
| CREATOR | Tenghao Huang, Sihao Chen, Muhao Chen, Jonathan May Longqi Yang Mengting Wan Pei Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-21389
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Teaching Language Models To Gather Information Proactively Tenghao Huang, Sihao Chen, Muhao Chen, Jonathan May Longqi Yang Mengting Wan Pei Zhou Abstract Large language models (LLMs) are increasingly expected to function as collaborative partners, engaging in back-and-forth dialogue to solve complex, ambiguous problems. However, current LLMs often falter in real-world settings, defaulting to passive responses or narrow clarifications when faced with incomplete or under-specified prompts—falling short of proactively gathering the missing information that is crucial for high-quality solutions. In this work, we introduce a new task paradigm: proactive information gathering, where LLMs must identify gaps in the provided context and strategically elicit implicit user knowledge through targeted questions. To systematically study and train this capability, we design a scalable framework that generates partially specified, real-world tasks, masking key information and simulating authentic ambiguity. Within this setup, our core innovation is a reinforcement finetuning strategy rewards questions that elicit genuinely new, implicit user information—such as hidden domain expertise or fine-grained requirements—that would otherwise remain unspoken. Experiments demonstrate that our trained Qwen-2.5-7B model significantly outperforms o3-mini by 18% on automatic evaluation metrics. More importantly, human evaluation reveals that clarification questions and final outlines generated by our model are favored by human annotators by 42% and 28% respectively. Together, these results highlight the value of proactive clarification in elevating LLMs from passive text generators to genuinely collaborative thought partners. Figure 1: Distribution of LLM Interaction Styles Across Disciplines. Collaborative exchanges account for over half of the interactions in all three fields, indicating a widespread shift from one-shot prompting to iterative, multi-turn dialogue Handa et al. 2025 . Figure 2: Proactive Clarification Enables Optimal LLM Responses.(top) the traditional "one-pass" approach, where the model attempts to respond with limited information, often leading to suboptimal results; (bottom) a proactive approach, where the model detects missing details and engages the user in clarification, ultimately producing a more accurate and helpful response. 1 Introduction Large-language models (LLMs) are now indispensable partners for solving diverse reasoning tasks—drafting proofs, debugging code, or writing essays—often excelling when provided with well-structured prompts and sufficient information Lightman et al. 2023 ; Chen et al. 2024a ; Luo et al. 2024 ; Wang et al. 2024 ; Brown et al. 2024 . Yet, as LLMs become more tightly woven into real-world workflows, the demands on their conversational abilities are evolving. Instead of simply answering queries, users increasingly expect LLMs to act as collaborative partners—engaging in multi-turn, back-and-forth dialogues to tackle complex, ambiguous, and open-ended problems Bommasani et al. 2022 ; Spangher et al. 2025 ; Handa et al. 2025 ( Figure 1 ). A core challenge in these collaborative settings is information asymmetry: users often provide incomplete or under-specified prompts (“lazy prompting”), and expect the LLM to fill in gaps or steer the conversation productively. As we show in Table 2 , off-the-shelf models typically respond with clarifying questions that focus narrowly on what is already present in the current context, or fall back to generic queries that rarely drive the conversation forward. This approach falls short when the model needs to proactively gather information that is missing, unknown, or unspoken—especially in domains like social science or business, where crucial details are often left unsaid, and no single “correct” answer exists. Proactive information gathering marks a crucial departure from conventional question generation. Rather than merely seeking clarification of ambiguities in existing input, proactive questions poke into missing dimensions, soliciting new, complementary information from the user—details that have not yet surfaced in the conversation. The goal is to transform the LLM from a passive responder into a genuine thought partner that can elicit relevant, actionable knowledge, anticipate what is needed, and drive the dialogue toward more productive outcomes ( Figure 2 ). However, designing LLMs that excel at proactive information gathering poses several obstacles. High-quality, collaborative dialogue data is scarce and difficult to scale; organic user logs are often noisy and proprietary, while crowd-sourcing nuanced, domain-specific exchanges is expensive and hard to control Malaviya et al. 2024 ; Spangher et al. 2025 . Even more fundamentally, the qualities that define a “good” proactive question—helpfulness, novelty, contextual complementarity—are subjective and hard to capture with standard reward signals or simple heuristics. In this work, we address these challenges by designing a framework that specifically rewards pioneering, context-complementary questions—those that reach beyond what is already provided, proactively soliciting critical information from users. Our approach has three core innovations: 1. Task Formulation: We introduce proactive information gathering as a new task for LLMs, formalizing the ability to identify and elicit targeted, contextually missing information from users through dialogue. 2. Synthetic Conversation Engine: Leveraging the Dolomites dataset Malaviya et al. 2024 , we construct a simulation pipeline that creates ambiguous prompts and rich clarification trajectories by systematically masking critical information—ensuring that proactive questioning becomes necessary for task completion. 3. Reinforcement Fine-Tuning: We propose a reward structure that rewards questions reaching beyond the provided context, and fine-tune LLMs using proximal policy optimization. Experiments demonstrate that our trained Qwen-2.5-7B model significantly outperforms o3-mini by 18% on automatic evaluation metrics. More importantly, human evaluation reveals that clarification questions and final outlines generated by our model are favored by human annotators by 42% and 28% respectively. Together, these contributions chart a path toward LLMs that are not just compliant responders, but proactive, collaborative partners—able to drive richer, more effective conversations by actively seeking the information that matters most. 2 Related Work Proactive Agent. The term “ proactive agent ” has long been correlated with agents that ask follow - up questions to resolve ambiguities Bi et al. 2021 ; Ren et al. 2021 . However, these efforts focus on addressing slot ambiguities, which aims to clarify ambiguous information present in corpus Guo et al. 2021 ; Deng et al. 2022 ; Pang et al. 2024 ; Chen et al. 2024b . Recent works extend the notion of a ‘proactive agent’, encompassing agents that predict user tasks Lu et al. 2024 . In this work, our definition of proactive agents entails actively identifying incomplete knowledge regarding domain procedures and user preferences through iterative clarification. Recent works emphasize proactive assistance for everyday conversation Chen et al. 2024b within fully unobservable environments and reasoning tasks Wu et al. 2025 . In contrast, our work targets open-ended writing tasks and employs a partially observable environment, reflecting real-world scenarios where users inputs imperfect prompt and agents only have access to partial information up front and must strategically clarify hidden details through iterative dialogue. Reinforcement Learning for LLM Alignment. Previous works focus on reasoning scenarios where step-level rewards are available or the outcome is easy to verify, such as mathematics Chen et al. 2024a ; Luo et al. 2024 ; Lightman et al. 2023 ; Wang et al. 2024 , and coding Brown et al. 2024 tasks. However, these methods are not generalizable to open-ended tasks, where supervision signals are sparse. Our work proposes a framework that masks user domain knowledge and preferences, providing sufficient learnable reward signals for models at train time. Another line of works uses reward signals from interactive settings Zhou et al. 2023 ; Roth et al. 2025 , typically relying on simulated user feedback to shape the model’s behavior. Figure 3: An example of our