Enabling Conversational Behavior Reasoning Capabilities in Full-Duplex Speech

Shuchang Pan, Siddharth Banerjee, Dhruv Hebbar, Siddhant Patel, Akshaj Gupta, Kan Jen Cheng, Hanjo Kim, Zeyi Austin Li, Martin Q. Ma, Tingle Li, Gopala Anumanchipalli, Jiachen Lian, [0.5em] Zhejiang University, University of California, Berkeley · 2025

Read Enabling Conversational Behavior Reasoning Capabilities in Full-Duplex Speech by Shuchang Pan, Siddharth Banerjee, Dhruv Hebbar, Siddhant Patel, Akshaj Gupta, Kan Jen Cheng, Hanjo Kim, Zeyi Austin Li, Martin Q. Ma, Tingle Li, Gopala Anumanchipalli, Jiachen Lian, [0.5em] Zhejiang University, University of California, Berkeley (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-21706.txt.

Introduction

Shuchang Pan, Gopala Anumanchipalli, Jiachen Lian, and their co-authors teach machines to reason about conversational behavior in full-duplex speech, where both sides can talk and listen at once. Human conversation runs on an implicit chain of intentions that surfaces as timed speech acts: interruptions, backchannels, turn-taking. Interactive systems need to foresee those acts, not merely transcribe words.

The method treats the intent-to-action pathway as causal inference inside a graph of thoughts. A labeling scheme predicts high-level intents and low-level speech acts together, learning their causal and temporal links. Training data mixes event-rich simulations with human-annotated rationales and recorded conversations. During a live dialogue, the system keeps an evolving graph of its streaming predictions. A multimodal transformer forecasts the next speech act, gives a reason, and revises its reasoning as talk unfolds. Tests show robust behavior detection and readable reasoning chains.

Catalog record

CREATORShuchang Pan, Siddharth Banerjee, Dhruv Hebbar, Siddhant Patel, Akshaj Gupta, Kan Jen Cheng, Hanjo Kim, Zeyi Austin Li, Martin Q. Ma, Tingle Li, Gopala Anumanchipalli, Jiachen Lian, [0.5em] Zhejiang University, University of California, Berkeley
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-21706

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enabling Conversational Behavior Reasoning Capabilities in Full-Duplex Speech Shuchang Pan, Siddharth Banerjee, Dhruv Hebbar, Siddhant Patel, Akshaj Gupta, Kan Jen Cheng, Hanjo Kim, Zeyi Austin Li, Martin Q. Ma, Tingle Li, Gopala Anumanchipalli, Jiachen Lian, [0.5em] Zhejiang University, University of California, Berkeley Abstract Human conversation is organized by an implicit chain of thoughts that manifests as timed speech acts. Capturing this causal pathway is key to building natural full-duplex interactive systems. We introduce a framework that enables reasoning over conversational behaviors by modeling this process as causal inference within a Graph-of-Thoughts (GoT). Our approach formalizes the intent-to-action pathway with a hierarchical labeling scheme, predicting high-level communicative intents and low-level speech acts to learn their causal and temporal dependencies. To train this system, we develop a hybrid corpus that pairs controllable, event-rich simulations with human-annotated rationales and real conversational speech. The GoT framework structures streaming predictions as an evolving graph, enabling a multimodal transformer to forecast the next speech act, generate concise justifications for its decisions, and dynamically refine its reasoning. Experiments on both synthetic and real duplex dialogues show that the framework delivers robust behavior detection, produces interpretable reasoning chains, and establishes a foundation for benchmarking conversational reasoning in full duplex spoken dialogue systems. Project page: https://got-duplex.github.io/ † † footnotetext: ∗ Equal contribution. † Corresponding author: [email protected] 1 Introduction Recent advances in spoken dialogue systems have shifted from turn-based, half-duplex models to full-duplex systems capable of simultaneous listening and speaking ( 4 ; 40 ; 21 ) . As illustrated in Figure 1 (left), the dominant paradigms frame this task as prediction. The first approach, Next Segment Prediction, models the agent’s response as a complete turn ( 18 ; 27 ; 25 ) . A more recent approach, Next Dual-Token Prediction, generates simultaneous token streams for both speakers to better handle overlap and real-time interaction ( 39 ; 11 ) . While these methods have improved system responsiveness, they treat conversation as a sequence generation problem, bypassing the cognitive layer of reasoning that governs human interaction. Human conversation, however, operates on a more abstract and causal level. We argue for a paradigm shift from black-box prediction to an explicit process of Next Behavior Perception and Prediction, as depicted in Figure 1 (right). When Speaker 1 produces an utterance, Speaker 2 does not simply predict the next sequence of words. Instead, they first perceive the behavior (e.g., recognizing a constative speech act), which triggers an internal chain of thought (e.g., deciding not to interrupt and to remain silent). This reasoning process culminates in a generated action (e.g., an acknowledgement). This gap between pattern matching and causal reasoning is a fundamental barrier to creating truly natural AI agents. Our work addresses the core scientific question: How can a machine model this perception-reasoning-generation loop to make principled, interpretable decisions in real time? Figure 1: Comparison of dialogue paradigms. (Left) Traditional duplex systems frame conversation as a direct sequence prediction task. (Right) We propose a framework based on next-behavior perception and reasoning, where an agent perceives the speaker’s act, reasons using a Graph-of-Thoughts, and then generates a response. To tackle this challenge, we introduce a framework that operationalizes the process. Our approach is twofold. First, we formalize the Perception stage with a hierarchical conversational behavior detection model. This module learns to identify conversational behaviors at two dimensions: high-level speech acts (e.g., constative, directive ) ( 23 ) that capture communicative intent, and low-level acts (e.g., turn-taking, backchannel ) that describe interaction mechanics ( 54 ; 16 ; 13 ; 49 ; 24 ; 38 ; 34 ; 4 ; 40 ) . This provides the system with a structured understanding of the ongoing dialogue. Second, we model the explicit reasoning process with a Graph-of-Thoughts (GoT) system ( 62 ) . This system constructs a dynamic causal graph from the sequence of perceived speech acts, capturing the evolving chain of thought within the conversation. By performing inference over this graph, our model can not only predict the most appropriate subsequent behavior but also generate a natural language rationale explaining its decision. This transforms the opaque prediction task into an auditable reasoning process, which provides a unified benchmark for evaluating conversational behavior in duplex speech systems. To train our framework, we developed a hybrid corpus that combines behavior-rich simulated dialogues with real-world conversational data annotated with human rationales. Our analysis confirms that the synthetic data reproduces key interactional structures of human conversation, such as turn-taking dynamics. In summary, our contributions are: • A conceptual reframing of full-duplex interaction from next-token prediction to next-behavior reasoning, arguing that modeling the causal chain from intent to action is critical for natural dialogue. • A hierarchical speech act detection model that perceives conversational behaviors at both high (intent) and low (action) levels, serving as a foundational module for reasoning-driven dialogue systems. • A GoT framework for conversational reasoning that models intent-act dynamics as a causal graph, enabling real-time, interpretable decision-making and rationale generation. • A comprehensive empirical validation demonstrating that our system effectively detects conversational behaviors, generates plausible rationales, and successfully transfers its reasoning capabilities from simulated to real-world full-duplex audio. 2 Related Work Duplex Models. Recent work in spoken dialogue systems (SDMs) increasingly draws on human conversational behaviors. Building on insights from human conversation, recent SDMs have progressed toward duplex capabilities—systems that listen and speak concurrently. SDMs are commonly built in two modes. Half-duplex models follow a turn-by-turn protocol, waiting for explicit end-of-turn signals (e.g., end-pointing/VAD) before responding, which simplifies streaming but adds latency and suppresses natural overlap. They commonly adopt next segment prediction ( 18 ; 27 ; 25 ) where the system predicts the agent’s response as a segment. Full-duplex models listen and speak simultaneously, modeling overlapping speech, micro-pauses, and background noise to sustain context and deliver timely cues (e.g., continuers, barge-in handling). They either employ the next segment prediction paradigm ( 4 ; 40 ; 21 ) or introduce the next dual token prediction ( 39 ; 11 ) scheme to incorporate the listener’s branch. Conversational Behaviors. Interactive behaviors in SDMs have been extensively examined to foster mutual understanding, user engagement, and social connection between human and computer. However, recent studies in SDMs only focus on low-level behaviors (backchannel ( 54 ; 33 ; 34 ) , turn-taking ( 16 ; 13 ; 49 ) , interruption ( 24 ; 38 ) , continuation ( 33 ; 34 ; 3 ; 41 ) , emotion ( 35 ; 44 ) , and disfluency ( 28 ; 29 ; 30 ; 31 ) , which overlook the importance of discourse-level intent that drives these phenomena. To resolve this, we also introduce high-level speech acts (constative, directive, commissive, acknowledgment) ( 23 ) to restore this layer, enabling more interpretable modeling and evaluation for conversational behaviors. Graph of Thought Reasoning. Reasoning techniques such as Chain-of-Thought ( 60 ) , Tree of Thought ( 61 ) , and Graph of Thought ( 6 ) have help guide LLMs through intermediate steps to boost reasoning accuracy. Particularly, Graph of Thought enables arbitrary reasoning dependencies by modeling an LLM thought as a vertex, and dependencies among thoughts as edges, offering increased reasoning capabilities while reducing inference cost ( 6 ; 7 )

Read the complete text