InfoQuest: Evaluating Multi-Turn Dialogue Agents for Open-Ended Conversations with Hidden Context

Bryan L. M. de Oliveira, Luana G. B. Martins, Bruno Brandão, Luckeciano C. Melo · 2025

Read InfoQuest: Evaluating Multi-Turn Dialogue Agents for Open-Ended Conversations with Hidden Context by Bryan L. M. de Oliveira, Luana G. B. Martins, Bruno Brandão, Luckeciano C. Melo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12257.txt.

Introduction

Bryan de Oliveira and three co-authors introduce InfoQuest in 2025, a test for chat agents that face vague requests. Language models follow clear orders well, but they often answer fuzzy requests with long, generic replies instead of asking what the user means. InfoQuest hides key facts that the agent must draw out through questions.

The test uses open-ended scenes where the right answer depends on facts the user has not given. The team ran both open and closed models through it. Even the best paid models struggled to gather the facts they needed, and weaker models fell back on generic replies. The authors share their method for building such scenes, so others can train and test better question-asking agents.

Catalog record

CREATORBryan L. M. de Oliveira, Luana G. B. Martins, Bruno Brandão, Luckeciano C. Melo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12257

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InfoQuest: Evaluating Multi-Turn Dialogue Agents for Open-Ended Conversations with Hidden Context Bryan L. M. de Oliveira, Luana G. B. Martins, Bruno Brandão, Luckeciano C. Melo Abstract Large language models excel at following explicit instructions, but they often struggle with ambiguous or incomplete user requests, defaulting to verbose, generic responses instead of seeking clarification. We introduce InfoQuest, a multi-turn chat benchmark designed to evaluate how dialogue agents handle hidden context in open-ended user requests. This benchmark presents intentionally ambiguous scenarios that require models to engage in information-seeking dialogue by asking clarifying questions before providing appropriate responses. Our evaluation of both open and closed models reveals that, while proprietary models generally perform better, all current assistants struggle to gather critical information effectively. They often require multiple turns to infer user intent and frequently default to generic responses without proper clarification. We provide a systematic methodology for generating diverse scenarios and evaluating models’ information-seeking capabilities, which can be leveraged to automatically generate data for self-improvement. We also offer insights into the current limitations of language models in handling ambiguous requests through multi-turn interactions. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities in following explicit instructions and engaging in task-oriented dialogue. However, when users provide incomplete or ambiguous requests – a common occurrence in real-world interactions – these models often default to producing verbose, generic responses rather than seeking clarification ( Kuhn et al., 2022 ; Kim et al., 2023 ; Chi et al., 2024 ; Zhou et al., 2024b ; Rahmani et al., 2023 ) . The ability to ask clarification questions is crucial for conversational systems to understand a user’s underlying needs, especially with limited input ( Rahmani et al., 2023 ) . Furthermore, in order to maximize the expected quality of a conversation, dialogue agents should reason about the stochastic transitions within a conversation to select the optimal response at each turn ( Chen et al., 2025 ) . As a motivating example, consider the conversation shown in Figure 1 . The left block provides the conversation context. The center block illustrates how a naive agent responds with a lengthy, generic explanation, making assumptions about the user’s context. In contrast, the right block shows an information-seeking agent engaging in focused dialogue, asking targeted clarifying questions to understand the specific scenario before providing a tailored response. This balanced approach of gathering key information while maintaining conversation flow is crucial – excessive questioning can frustrate users, while insufficient context-gathering leads to unhelpful generic responses. The information-seeking behavior enables agents to provide more accurate, personalized assistance by understanding the user’s unique situation, constraints, and goals. However, current LLMs often fail to strike this balance, defaulting to overly broad responses that may waste the user’s time without addressing their actual needs. Effective information-seeking requires carefully weighing when additional context is truly needed versus when enough information exists to provide meaningful help. Figure 1: Naive vs. information-seeking agents handling ambiguous user requests . Left: context. Center: naive agent’s verbose response. Right: information-seeking agent’s targeted questions. Evaluating dialogue agents in open-ended conversations presents unique challenges. In constrained environments like games or structured tasks, quantifying how well a model collects relevant context is relatively straightforward ( Hausknecht et al., 2020 ; Yao et al., 2022 ; Abdulhai et al., 2023 ; Zhou et al., 2024b ) . However, measuring this capability becomes significantly more complex in open-ended, multi-turn settings that better reflect everyday human-AI interactions. The key challenge lies in the unbounded nature of these interactions; there is no predefined ”correct” sequence of questions or complete set of information to gather, making it difficult to establish ground truth or quantitative metrics. While existing benchmarks have focused on structured tasks or specific dialogue abilities ( Bai et al., 2024 ; Kwan et al., 2024 ) , an evaluation framework that can assess how models handle the inherent ambiguity and contextual complexity of natural conversations remains necessary. To address this gap, we introduce InfoQuest , a multi-turn chat benchmark designed to evaluate how dialogue agents handle hidden context in user requests. It simulates conversational settings with intentionally open-ended and ambiguous requests by generating seed messages that could plausibly come from multiple distinct personas, each with their own goals and constraints. This controlled ambiguity requires agents to demonstrate effective information-seeking behavior through clarifying questions and progressively infer the user’s specific context before providing a satisfactory answer. The benchmark uses a combination of LLMs to simulate realistic user responses. We leverage other LLMs for role-playing the user in our dynamic scenarios ( Zhou et al., 2024a ) and evaluate the assistant’s ability to gather critical information through targeted questioning. Our evaluation reveals that, while proprietary models generally outperform open models, all current assistants struggle to effectively handle hidden information across diverse user scenarios. Notably, we find that models require multiple turns to infer user intent and address latent requests, demonstrating poor turn efficiency. Qualitatively, we observe that models often default to generic responses without asking clarification questions, highlighting a key area for improvement in developing more interactive and context-aware conversational agents. Contributions. This work contributes to the field by introducing InfoQuest, a novel benchmark for evaluating how language models handle open-ended conversations with hidden context. We develop a systematic methodology for generating diverse, ambiguous scenarios that require models to engage in multi-turn dialogue to uncover critical information. Furthermore, the data generated through InfoQuest’s automated pipeline can be used for self-improvement, enabling models to learn from their interactions and enhance their ability to handle ambiguous queries. Additionally, we conduct comprehensive experiments comparing both open and proprietary models on their ability to handle ambiguous requests through multi-turn interactions, providing insights into the current limitations of LLMs in this area and identifying key challenges in information-seeking dialogue. 2 Related Work Our work builds on prior research in clarification questions and multi-turn dialogue evaluation. Recent work has explored various approaches to handling ambiguous queries. Kuhn et al. (2022) proposed a two-stage framework that first classifies whether a question is ambiguous and then generates appropriate clarifying questions. However, their evaluation focused on single-turn interactions using paired ambiguous and unambiguous questions. Kim et al. (2023) developed a method to generate comprehensive trees of disambiguations to address ambiguous queries in a single response, in contrast to our focus on interactive, multi-turn clarification. Chi et al. (2024) introduced an approach for selecting clarifying questions that maximize certainty in book search tasks, though their work was limited to this specific domain rather than open-ended dialogue. Several benchmarks have been developed to evaluate multi-turn dialogue capabilities in constrained settings. Hausknecht et al. (2020) created interactive fiction games that require information gathering through a text interface. Yao et al. (2022) developed a shopping website simulation where agents must navigate constraints to complete purchases. Abdulhai et al. (2023) proposed environments for training and evaluating reinforcement learning with language models. While these works provide valuable insights into structured information gathering, they rely on rigid rules and predefined success criteria that may not reflect the complexity of open-ended dialogue. Recent work has explored fine-grained evaluation of multi-turn dialogue abilities. Bai et al. (2024) used GPT-4 to generate evaluation conversations with fixed user messages, providing detailed assessment across multiple capabilities. Kwan et al. (2024) extended

Read the complete text