Full-Duplex-Bench v1.5 : Evaluating Overlap Handling for Full - Duplex Speech Models
Guan-Ting Lin, Shih-Yun Shan Kuan, Qirui Wang, Jiachen Lian, Tingle Li Hung-yi Lee · 2025
Read Full-Duplex-Bench v1.5 : Evaluating Overlap Handling for Full - Duplex Speech Models by Guan-Ting Lin, Shih-Yun Shan Kuan, Qirui Wang, Jiachen Lian, Tingle Li Hung-yi Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23159.txt.
Introduction
Guan-Ting Lin and co-authors release a benchmark for speech systems that talk and listen at once in a 2025 paper. Natural conversation includes interruptions and overlapping voices. Most dialogue software still takes strict turns, and overlap handling is rarely tested in a systematic way.
Their benchmark, now in version 1.5, simulates four overlap scenes. A user may interrupt, add a brief backchannel, talk to someone else, or speak over background voices. Scores cover response choices, stop and reply delays, and voice adaptation. Tests of five leading agents reveal two styles: quick responders and floor holders that filter overlaps. The open framework lets builders compare systems and improve overlap behavior.
Catalog record
| CREATOR | Guan-Ting Lin, Shih-Yun Shan Kuan, Qirui Wang, Jiachen Lian, Tingle Li Hung-yi Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-23159
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Full-Duplex-Bench v1.5 : Evaluating Overlap Handling for Full - Duplex Speech Models Guan-Ting Lin, Shih-Yun Shan Kuan, Qirui Wang, Jiachen Lian, Tingle Li Hung-yi Lee Abstract While full - duplex speech agents promise natural, low - latency human–machine interaction by concurrently processing input and output speech, overlap management remains under - evaluated. We introduce Full - Duplex - Bench v1.5 , a modular, fully automated benchmark that simulates four overlap scenarios—user interruption, listener backchannel, side conversation, and ambient speech. Our framework supports both open - source and commercial models, offering a comprehensive, extensible metric suite—categorical dialogue behaviors, stop and response latency, prosodic adaptation, and perceived speech quality—that can be tailored to application - specific criteria. Benchmarking five state - of - the - art agents reveals two principal strategies— repair - first rapid yielding versus continuity - first sustained flow—and highlights scenario - dependent performance trends. The open - source design enables seamless extension with new audio assets, languages, and deployment contexts, empowering practitioners to customize and accelerate the evaluation of robust full - duplex speech systems 1 1 1 Code and data will be available at https://github.com/DanielLin94144/Full-Duplex-Bench . Figure 1 : Overview of the proposed evaluation framework for full - duplex speech models. User speech (top waveform) overlaps with the model’s output (bottom waveform) under controlled scenarios—Interruption, Backchannel, and Multi - party conversation—and the model is assessed along three dimensions: Behaviors (Resume, Respond, Uncertain, Unknown), Timing metrics (Stop latency, Response latency), and Speech Features (Pitch, Intensity, Speaking rate, Quality). 1 Introduction Full - duplex speech models—systems that can speak and listen concurrently—are reshaping the landscape of spoken human–computer interaction. By enabling simultaneous speech production and perception, they eliminate the rigid wait–listen–respond cycle, allowing for more fluid exchanges that support backchannels, mid - utterance corrections, and dynamic user engagement [ 1 ] . These capabilities are particularly valuable in settings such as in - car navigation, real - time translation, clinical triage, and hands - free environments, where responsiveness and natural interaction are essential. However, overlapping speech is not an edge case but a defining characteristic of natural dialogue: over 40% of turns involve some degree of overlap [ 2 , 3 , 4 ] . These overlaps arise from a variety of sources [ 5 , 6 , 7 , 8 ] : (i) user interruptions or barge-ins, where someone speaks over others, violating their turn to speak; (ii) listener backchannels (e.g., “uh - huh”, “right”) that signal attention but do not claim the floor; (iii) side conversations , where the user briefly addresses someone else while the model is speaking; and (iv) ambient or broadcast speech that shares the acoustic channel but is not directed at the system. If poorly managed, these phenomena can result in truncated responses, repeated outputs, or awkward silences—undermining interaction quality and user experience. Despite the prevalence and importance of such phenomena, most existing benchmarks target single - turn or strictly half - duplex paradigms and fail to probe overlapping behaviors [ 9 , 10 , 11 , 12 , 13 , 14 ] . Human evaluation offers contextual insight but is labor-intensive and not scalable. Corpus-level methods based on voice activity detection (VAD), inter-pausal units (IPUs), pause statistics, and floor-transfer offsets (FTO) [ 15 ] scale well but mask fine-grained, scenario-specific behaviors and overlook semantic appropriateness. Classifier-based frameworks such as Talking Turns [ 16 ] offer automation but suffer from limited generalizability due to corpus-specific training and reliance on user study annotations. The original Full - Duplex - Bench [ 17 ] took a first step toward systematic overlap evaluation by focusing on user interruptions. We extend this effort in Full - Duplex - Bench v1.5 , a modular benchmarking suite that evaluates four overlap scenarios: user interruption , user backchannel , user talking to others , and background speech . Each interaction is conducted through the model’s production API, with automatic alignment and segmentation of pre-overlap event and post-overlap event regions. The framework is designed for extensibility: users can swap in different audio conditions, real human speech, or new languages by simply replacing assets and rerunning the pipeline. All code, metrics, and annotation schemas are open-sourced for reproducibility and community-driven extensions. Our evaluation captures both content and prosodic dimensions. We label system responses with dialogue behaviors— Respond , Resume , Uncertain , Off-target , and Silent —and compute signal-level features including pitch , intensity , perceived naturalness , as well as stop latency and response latency . This multidimensional view surfaces model trade-offs between responsiveness and coherence, highlights stylistic adaptations under different overlap types, and reveals robustness patterns that static, turn-by-turn benchmarks fail to expose. Figure 2 : Illustration of the four controlled overlap scenarios used in Full - Duplex - Bench v1.5 . For each case, the user’s speech (top waveform) and the model’s speech (bottom waveform) are shown on a shared timeline. (1) User Interruption —the user barges in with a new request; (2) User Backchanneling —a brief acknowledgement ( uh - huh ); (3) Talking to Other —the user briefly turns to speak to another person while the model is still talking; (4) Background Speech —far - field third - party talk that is acoustically present but not intended for the model. These scenarios probe a model’s ability to detect addressee shifts, distinguish intentional versus incidental overlap, and resume coherently after interference. 2 Related Works 2.1 Model Architectures Full - duplex spoken dialogue models (SDMs) can be grouped into three paradigms. Cascaded architectures decompose the pipeline into ASR, dialogue management (typically an LLM), and TTS modules; notable examples include FSM [ 18 ] , which introduces control, speak, and listen tokens, and MiniCPM - Duplex/Duo [ 19 , 20 ] , which synchronize components via time - sliced token windows. End - to - end models, by contrast, process input and output speech jointly to internalize turn - taking and prosodic cues: dGSLM [ 15 ] uses a Siamese cross - attention network, SyncLLM [ 21 ] fragments audio into time - synchronous frames, and Moshi [ 22 ] employs parallel inference for overlap handling. More recent research—SALMONN - omni [ 23 ] , MinMo [ 24 ] , Parrot [ 25 ] , RTTL - DG [ 26 ] —augments these designs with explicit state tokens or dialogue managers to refine turn - taking decisions. Despite this rich design space, most academic and open - source projects publish only project pages without releasing pretrained weights. To date, only Freeze - Omni and Moshi offer openly accessible checkpoints; all other state - of - the - art systems—e.g. Gemini Live 2 2 2 https://ai.google.dev/api/live , GPT - 4o realtime 3 3 3 https://platform.openai.com/docs/models/gpt-4o-realtime-preview , and Amazon Nova Sonic [ 27 ] —are distributed exclusively as closed - source APIs. This split between open - weight and API - only models poses significant reproducibility and comparability challenges, motivating our first - of - its - kind effort to uniformly evaluate both categories under a common framework. 2.2 Evaluation Frameworks for Full - Duplex Spoken Dialogue Systems Most existing speech benchmarks [ 28 , 29 , 11 , 12 , 14 ] focus on single - turn responses with minimal contextual