Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systems

2025

Read Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systems (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23938.txt.

Introduction

Knowing when to speak is the hidden skill of a natural voice assistant. Guojian Li and 10 co-authors release Easy Turn, an open-source model that detects dialogue turn states in full-duplex spoken systems. It fuses acoustic and linguistic cues to call four states: complete, incomplete, backchannel, and wait. Existing tools are often closed, huge, or single-modality, and full-duplex training data are scarce.

Alongside the model comes a training set of more than a thousand hours of speech built for this task. On their open test set, Easy Turn beat existing open models such as TEN Turn Detection and Smart Turn V2. Data and model are promised on GitHub.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23938

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systems Abstract Full-duplex interaction is crucial for natural human–machine communication, yet remains challenging as it requires robust turn-taking detection to decide when the system should speak, listen, or remain silent. Existing solutions either rely on dedicated turn-taking models, most of which are not open-sourced. The few available ones are limited by their large parameter size or by supporting only a single modality, such as acoustic or linguistic. Alternatively, some approaches finetune LLM backbones to enable full-duplex capability, but this requires large amounts of full-duplex data, which remain scarce in open-source form. To address these issues, we propose Easy Turn —an open-source, modular turn-taking detection model that integrates acoustic and linguistic bimodal information to predict four dialogue turn states: complete , incomplete , backchannel , and wait , accompanied by the release of Easy Turn trainset , a 1,145-hour speech dataset designed for training turn-taking detection models. Compared to existing open-source models like TEN Turn Detection and Smart Turn V2, our model achieves state-of-the-art turn-taking detection accuracy on our open-source Easy Turn testset . The data and model will be made publicly available on GitHub. 1 1 1 https://github.com/ASLP-lab/Easy-Turn † † address: 1 Audio, Speech and Language Processing Group (ASLP@NPU), School of Computer Science, Northwestern Polytechnical University, Xi’an, China 2 Huawei Technologies, China [email protected], [email protected], [email protected] * * footnotetext: Corresponding authors. 1 Introduction In human-machine speech interaction, full-duplex interaction enables simultaneous bidirectional information flow, allowing participants to switch smoothly and naturally between speaking and listening roles [ 1 ] . This capability is critical for achieving real-time, efficient interaction [ 2 ] . While full-duplex exchange occurs effortlessly for humans, it remains a major challenge for spoken dialogue systems [ 3 ] . To achieve equally natural communication, such systems require robust turn-taking detection, which involves accurately identifying user intent to determine when the system should speak, listen, or remain silent. With the rapid advancement of LLMs, spoken dialogue systems have made remarkable progress in recent years [ 4 , 5 , 6 , 7 ] , quickly entering daily life and exerting a profound impact. The turn-based (half-duplex) approaches already deliver excellent interactive experiences. Research on full-duplex spoken dialogue systems has become a major focus, with current efforts falling into two main categories: 1) integrating an additional turn-taking detection model independent of the LLM backbone, such as TEN Turn Detection 2 2 2 https://github.com/ten-framework/ten-turn-detection and Smart Turn V2 3 3 3 https://github.com/pipecat-ai/smart-turn . TEN Turn Detection utilizes the Qwen2.5-7B LLM to perform semantic analysis on text inputs, predicting one of three states: finished , unfinished , or wait . Smart Turn V2 is the latest turn-taking detection model of the Smart Turn series. It employs Wav2Vec2 [ 8 ] as the acoustic encoder with a linear classifier layer to directly conduct turn-taking detection from speech, outputting one of two states: complete or incomplete . 2) endowing the LLM backbone with full-duplex capability through fine-tuning on large-scale full-duplex dialogue data, as seen in Moshi [ 7 ] , Freeze-Omni [ 9 ] , and GLM-4-Voice [ 10 ] , etc. Although full-duplex spoken dialogue systems have achieved notable progress, they continue to face substantial challenges. For the approach that integrates a standalone turn-taking detection model, most implementations remain closed-source. Only a few available open-source models include TEN Turn Detection and Smart Turn V2. However, both models have significant limitations: 1) TEN Turn Detection suffers from an excessively large 7B parameter size and lacks direct speech input support, requiring an upstream automatic speech recognition (ASR) model that risks losing critical acoustic information. 2) Smart Turn V2 is fast and lightweight but relies solely on a linear classifier layer, making it difficult to capture rich semantic information. For the approach that directly trains the LLM backbone to acquire full-duplex capabilities, it is even more demanding, requiring extensive high-quality full-duplex dialogue data. Yet, most studies either withhold full-duplex training datasets entirely or provide only a brief description of their processing pipelines [ 11 ] . To accelerate research in this field, we introduce Easy Turn , an open-source and modular turn-taking detection model. The model accepts user’s speech as input and outputs both the corresponding ASR transcription and the dialogue turn state, effectively integrating acoustic and linguistic information. More importantly, we concurrently release Easy Turn trainset , a large-scale speech dataset specifically designed for training turn-taking detection models. This dataset spans all four dialogue turn states ( complete , incomplete , backchannel , wait ) with a total duration of 1,145 hours. Compared with the existing open-source turn-taking detection models TEN Turn Detection and Smart Turn V2, Easy Turn supports the most comprehensive set of dialogue turn state categories: complete (semantically complete), incomplete (semantically incomplete), backchannel (brief feedback), and wait (request to pause or end the dialogue). On our open-source Easy Turn testset , Easy Turn achieves state-of-the-art state classification accuracy while maintaining competitive inference speed and memory efficiency. The data and model will be made publicly available on GitHub. Figure 1: The detailed processing pipeline of the Easy Turn Corpus. 2 Easy Turn Corpus Our Easy Turn corpus consists of two parts: the Easy Turn trainset and the Easy Turn testset, covering four dialogue turn states: complete , incomplete , backchannel , and wait . Each speech recording is paired with a transcription and labeled with one of these states. In the following sections, we detail the definition of each turn state and the construction process of both the trainset and testset. 2.1 Turn States Complete The complete state indicates that the user has fully expressed their intent and expects an immediate response from the spoken dialogue system, as in the utterance, “Can you tell me a story?”. A high-performance full-duplex spoken dialogue system should respond swiftly and accurately to such complete expressions. Incomplete The incomplete state occurs when a user pauses but clearly has not finished speaking, as in “I want to ask…” or “Do you know how it…”. An ideal full-duplex spoken dialogue system continues listening until the user’s semantic expression is complete, rather than interrupting prematurely. Backchannel In natural conversations, backchannel signals refer to brief listener responses (e.g., “Uh-huh.”, “Right.”) that indicate active engagement and comprehension while the speaker is talking [ 12 ] . In full-duplex spoken dialogue systems, backchannel signals from users should not interrupt the system’s speech output, which is critical for maintaining interaction fluency and enhancing user experience. Wait The wait state refers to cases where users explicitly request to pause or terminate the interaction with the spoken dialogue system (e.g., “shut up”, “please stop”), serving as an efficient and concise way to end the system’s current turn or completely halt the dialogue [ 13 ] . 2.2 Trainset The Easy Turn trainset is a large-scale speech dataset designed for training turn-taking detection models, comprising both real and synthetic data. It comprises four dialogue turn states ( complete , incomplete , backchannel , wait ), totaling approximately 1,145 hours. Detailed statistics are provided in Table 1 . The detailed data processing pipeline is shown in Figure 1 . The following sections detail the procedures used to construct them. Table 1: The statistics

Read the complete text