CIG: Measuring Conversational Information Gain in Deliberative Dialogues with Semantic Memory Dynamics

Ming-Bin Chen, Jey Han Lau, Lea Frermann · 2026

Read CIG: Measuring Conversational Information Gain in Deliberative Dialogues with Semantic Memory Dynamics by Ming-Bin Chen, Jey Han Lau, Lea Frermann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15647.txt.

Introduction

Chen, Lau, and Frermann present Conversational Information Gain, a way to judge talk by how much new useful info it adds, not by tone or shape. A live store of claims drawn from what each speaker says rates each claim for newness, fit to the topic, and how much it implies. On 80 bits of debate and forum talk, this store's ups and downs match what people rate as gain far better than length or word-count tricks. The same scores can power tools that guess gain from text.

Catalog record

CREATORMing-Bin Chen, Jey Han Lau, Lea Frermann
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84207
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15647

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CIG: Measuring Conversational Information Gain in Deliberative Dialogues with Semantic Memory Dynamics Ming-Bin Chen, Jey Han Lau, Lea Frermann Abstract Measuring the quality of public deliberation requires evaluating not only civility or argument structure, but also the informational progress of a conversation. We introduce a framework for Conversational Information Gain (CIG) that evaluates each utterance in terms of how it advances collective understanding of the target topic. To operationalize CIG, we model an evolving semantic memory of the discussion: the system extracts atomic claims from utterances and incrementally consolidates them into a structured memory state. Using this memory, we score each utterance along three interpretable dimensions: Novelty , Relevance , and Implication Scope . We annotate 80 segments from two moderated deliberative settings (TV debates and community discussions) with these dimensions and show that memory-derived dynamics (e.g., the number of claim updates) correlate more strongly with human-perceived CIG than traditional heuristics such as utterance length or TF--IDF. We develop effective LLM-based CIG predictors paving the way for information-focused conversation quality analysis in dialogues and deliberative success. 1 1 1 The code and annotation data are available at https://github.com/mrknight21/memcig-analysis/tree/master/data . 1 Introduction Figure 1: Overview of the CIG pipeline. Each utterance is evaluated with the Semantic Memory as knowledge context for Novelty, Relevance, Implication Scope, and overall CIG (1–4). The Semantic Memory is maintained through two modules: Extraction , which converts utterances into atomic claims; and Consolidation , which matches extracted claims against the retrieved memory, triggering ADD , UPDATE , or NONE operations. Public deliberation—reasoned dialogue aimed at collective understanding and decision-making ( 15 ) for public interest ( 8 ) —is fundamental to democratic societies. Yet, the quality of these exchanges, from community forums ( 32 ) to public debates ( 26 ) , can vary widely from stagnation to productive collaboration. Metrics for evaluating dialogue quality ( 14 ) have largely focused on structural formality over substantive content. Approaches based on schemes like the Deliberative Quality Index (DQI; 35 ) or computational proxies for civility ( 1 ; 29 ) and argument structure ( 37 ) struggle to capture true informational progress. These surface-level signals can mislead: civil phrasing can conceal malicious intent ( 19 ) , and valuable insights might emerge from informal exchanges ( 39 ; 3 ) . Such methods fail to distinguish constructive progress from bureaucratic talk ( 26 ; 38 ) . Grounded in Meadow and Yuan’s notion of information impact as “A change, or the nature or magnitude of change, in the knowledge base of a subject domain of the recipient” ( 25 , p. 710) , we define Conversational Information Gain (CIG) as the degree to which an utterance advances collective understanding toward the goal/topic. We decompose CIG into three interpretable aspects— Novelty , Relevance , and Implication Scope —capturing whether a contribution introduces new information, connects to the shared goal, and extends its implications to the public community beyond individual cases. To operationalize CIG, we require a representation of the evolving collective knowledge state against which each new utterance can be evaluated: both annotators and models must know what has already been said. As shown in Figure 1 , we implement this through a lightweight semantic memory that maintains a consolidated set of claims. We first validate CIG through human annotation of 80 dialogue segments drawn from two moderated group-discussion settings—TV debates and community discussions—achieving moderate-to-high inter-annotator agreement for CIG and its aspects. We then validate automation by using an LLM with access to the same information as the human annotators (topic, short context, and a prior-memory summary), and show that LLM predictions closely track aggregated human judgments. Finally, by varying the model’s prior context, we find that predictions based on retrieved memory summaries are highly correlated with those based on the full preceding transcript, indicating that memory summaries provide a compact yet faithful substitute for full-history context in automated CIG assessment. We then analyze how memory dynamics relate to perceived informational progress, finding that simple memory-state signals (e.g., claim update counts) track human CIG ratings more consistently than common heuristics such as sentence entropy or TF–IDF. An unsupervised aggregation analysis reveals a “conjunctive bottleneck”: an utterance’s CIG is effectively limited by its weakest aspect. Finally, we provide a case study illustrating how CIG can be used to analyze downstream interaction dynamics in moderated discussions. 2 Related work Measuring Informativeness in Conversation While definitions of informativeness vary across disciplines, they share a common conceptual core: the change a message induces in the recipient—whether in certainty ( 17 ) , utility ( 13 ) , or knowledge state ( 25 ) . Particularly, dialogue evaluation has largely converged on two necessary conditions required to trigger this change: novelty (the presence of new signal) and relevance (the alignment of that signal to the goal context) ( 11 ; 23 ) . However, these two dimensions alone are insufficient to capture the magnitude or worthiness of a contribution ( 16 ) . While metrics like “impact” or “usefulness” attempt to proxy this third dimension, they are often vaguely defined or rely on subjective Likert scales that conflate personal preference with collective value ( 20 ; 30 ) . A complementary line of work extends classical information theory ( 33 ) , leveraging surprisal or information-density to trace information flow ( 22 ; 12 ; 36 ) . However, such measures only partially align with human-perceived salience ( 43 ; 22 ) , limiting their precision in localizing information exchanges. Agent Semantic Memory Agent memory modules provide a way to measure knowledge acquisition during conversation ( 45 ) . Originally developed to sustain coherence and personalization in conversational agents over long conversation sessions, these modules function by maintaining a selective, persistent state. In frameworks such as Mem0 ( 5 ) , salient claims are extracted from utterances and an LLM applies update operations to a knowledge store. This approach improves temporal and multi-hop reasoning on long-dialogue benchmarks ( 24 ) and reduces latency and token cost relative to full-history baselines. Aspect Label Definition / Anchor Conversational Information Gain 1. No gain Repeats or obstructs; no meaningful advance beyond the existing knowledge. 2. Minimal gain Small clarification or slight nuance that is noticeable but limited. 3. Incremental Adds new details/mechanisms within the same conceptual frame or ideas within the topic. 4. Insightful Reframes or introduce new ideas under the topic; shifts the conversation in a new valuable way. Novelty 1. Not novel Repetition/paraphrase of prior content or non/common-sense content. 2. Minimally novel Minor or mostly predictable detail added to an existing idea. 3. Moderately novel New evidence, concrete example, or supporting detail expanding an existing idea. 4. Highly novel New framework, principle, idea, or line of reasoning that opens a new direction. Relevance 1. Not relevant Off-topic; no connection to the conversation goal. 2. Minimally relevant Loose or indirect link; requires inference to connect. 3. Moderately relevant Substantially related but not central (e.g., side issue or counterpoint). 4. Highly relevant Directly and explicitly addresses the core topic or goal. Implication Scope 1. Local Manages the immediate moment; implication limited to participants/procedures. 2. Bounded Self-contained fact, feeling, or stance; no generalization beyond the case. 3. Generalizing

Read the complete text