YIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation Agents

Victor De Lima, Grace Hui Yang · 2026

Read YIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation Agents by Victor De Lima, Grace Hui Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10968.txt.

Introduction

De Lima and Yang introduce information elicitation agents, whose goal is to draw information from users for institutional or task goals. It fits academic interviews, court proceedings, and journalism, where most chat agents follow user direction. They present YIELD, a dataset of two thousand two hundred eighty-one ethically sourced human-to-human dialogues. They model elicitation as a finite-horizon decision process with hidden state and propose metrics for it. Pilot tests show training on YIELD improves alignment with real elicitation behaviour, confirmed by human review.

Catalog record

CREATORVictor De Lima, Grace Hui Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50987
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10968

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

YIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation Agents Victor De Lima, Grace Hui Yang Abstract Most conversational agents (CAs) are designed to satisfy user needs through user-driven interactions. However, many real-world settings, such as academic interviewing, judicial proceedings, and journalistic investigations, involve broader institutional decision-making processes and require agents that can elicit information from users. In this paper, we introduce Information Elicitation Agents (IEAs) in which the agent’s goal is to elicit information from users to support the agent’s institutional or task-oriented objectives. To enable systematic research on this setting, we present YIELD , a 26M-token dataset of 2,281 ethically sourced, human-to-human dialogues. Moreover, we formalize information elicitation as a finite-horizon POMDP and propose novel metrics tailored to IEAs. Pilot experiments on multiple foundation LLMs show that training on YIELD improves their alignment with real elicitation behavior and findings are corroborated by human evaluation. We release YIELD under CC BY 4.0. The dataset, project code, evaluation tools, and fine-tuned model adapters are available at: https://github.com/infosenselab/yield . Figure 1: Traditional vs. information elicitation agents. 1 Introduction Most conversational agents (CAs) are designed to satisfy user needs through user-driven interactions. However, many real-world settings, such as academic interviewing 30 , judicial proceedings 28 , and journalistic investigations 40 , involve broader institutional decision-making processes and require agents that can elicit information from users. In this paper, we propose to study and support Information Elicitation Agents (IEAs), a new class of conversational agents designed to meet these goals. Figure 1 illustrates the core distinctions between traditional CAs and IEAs across three dimensions: conversational initiatives, agent’s objectives, and common use cases. Table 1 shows sample dialogues between human elicitors and human respondents in various domains, including oral history and judicial proceedings. The goal of this paper is to support the study and creation of agents that can behave like human elicitors. However, little research has been done for this purpose. Oral History 18 Judicial Proceeding 28 Elicitor : "OK, then think back to when you went into the Peace Corps. What were you doing the year before you went in?" Respondent : "I was in high school when I went in. I was one of the eighteen year olds that went into the Peace Corps." Elicitor : "Did you have any particular country or any place that you wanted to go?" Respondent : "South America always had a calling for me. I didn’t have any particular country at all." Elicitor : "Is there any aspect of social media that you think is protected by the First Amendment?" Respondent : "Yes, Your Honor. I can certainly imagine platforms that would be subject to this law that indeed have First Amendment rights." Elicitor : "But why is it different?" Respondent : "Well – well, there’s a lot in there, Your Honor. Maybe I can start with the Hurley case." Academic Interview 30 Journalistic Investigation 40 Elicitor : "I was just wondering your opinion on whether biodiversity offsetting is likely to lead to more stringent or less stringent regulations around infrastructure projects?" Respondent : "It’s ill-informed people not understanding the system properly because at the end of the day biodiversity offsetting doesn’t affect any existing protections." Elicitor : "What’s been the reception of businesses to biodiversity offsetting extending the mitigation hierarchy?" Respondent : "Well, the reception that I’ve had when I’ve been speaking to them has been really quite positive." Elicitor : "This is your second trip to Mexico, but why are you here now in Mexico?" Respondent : "So, as the new government makes this transition beginning on Dec. 1, we’re working diligently to make sure we have a solid foreign policy relationship with them." Elicitor : "That caravan that is coming up from Guatemala heading to Mexico and then presumably maybe some place else, maybe the United States: What’s Mexico say about that?" Respondent : "So Mexico agrees that it is not constructive to have unlawful migration transit from these Northern Triangle countries through Mexico into the United States." Table 1: Shortened examples of information elicitation dialogues. Sources are included next to domain names. IEAs differ fundamentally from other conversational agents in both whose goals they serve and how success is defined. Conversational agents are generally designed to help users accomplish user-initiated tasks, with the user controlling the agenda and direction of the interaction. IEAs, by contrast, act on behalf of an external institutional objective and use conversation as a means to extract information from the respondent. Crucially, at any point in the dialogue there is not a single optimal question or expected answer; instead, there are many possible directions, each potentially yielding valuable information. The agent’s task is therefore to choose questions that are likely to surface informative and relevant facts, maximizing the cumulative value of what is uncovered rather than converging on a single correct outcome. The task of creating IEAs is challenging for several reasons. First, the agent must act in an extremely high-dimensional action space, where every utterance can shift the trajectory of the dialogue. Second, defining what makes information “informative” is inherently nuanced and context-dependent, bringing subjectivity into both training and evaluation. These difficulties are further amplified in long dialogues, where agents must maintain topic coherence over many turns while responding adaptively to unpredictable or even adversarial user replies. Additionally, the amount of publicly available data for training IEAs is limited. To the best of our knowledge, no existing dataset is available to support the development of IEAs. In this paper, we make the following contributions: Dataset # Domains # Dialogues Avg. Turns Per Dialogue Avg. Tokens Per Turn # Turns # Tokens DSTC2 1 1,612 14.49 8.54 23,354 199,431 MultiWOZ 7 8,438 13.46 13.13 113,556 1,490,615 SGD 16 16,142 20.44 9.75 329,964 3,217,149 YIELD 4 2,281 171.07 67.27 390,205 26,249,014 Table 2: Commonly used dialogue dataset statistics. • We introduce IEAs as a distinct conversational paradigm and release YIELD: Information-Elicitation Learning Dialogues (YIELD), the first large-scale, ethically sourced IEA dataset. YIELD consists of real, human-to-human, English information-elicitation dialogues sourced exclusively from repositories with clear public domain or Creative Commons licenses. YIELD contains four domains and is larger than existing task-oriented datasets (See Table 2 ). We publish both the standard, full dialogue version of the dataset and a rolling-window segmented version used for the experiments in this study. • We propose and operationalize a few IEA-specific evaluation metrics: Conformity , Progression , and Turn-Length Ratio . • We further formalize information elicitation as a finite-horizon Partially Observable Markov Decision Process (POMDP) and use offline reinforcement learning (ORL) with a minimal proxy reward to show that dialogue-level optimization with YIELD is competitive with supervised fine-tuning (SFT). • Pilot experiments across multiple foundation LLMs show that fine-tuning on YIELD improves distributional and behavioral alignment with human elicitation dialogues. These findings are confirmed by human evaluations. The remainder of the paper is organized as follows. Section 2 reviews related work. Section 3 describes the YIELD dataset. Section 4 introduces the formalization of the information elicitation problem. Section 5 presents IEA evaluation metrics. Section 6 details the pilot experiments and human evaluation; and Section 7 concludes the paper. 2 Related Work Dialogue datasets are instrumental in advancing research in conversational

Read the complete text