MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models
He Zhang Wenqian Cui Haoning Xu Xiaohui Li Lei Zhu Shaohua Ma Irwin King · 2025
Read MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models by He Zhang Wenqian Cui Haoning Xu Xiaohui Li Lei Zhu Shaohua Ma Irwin King (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10262.txt.
Introduction
He Zhang and seven co-authors introduce MTR-DuplexBench, a benchmark for full-duplex speech models that can talk and listen at once across many conversational rounds. Existing benchmarks test single exchanges, while real full-duplex dialogue has blurred turn boundaries and context that drifts during inference. The new benchmark segments continuous dialogue into discrete turns so each round can be scored separately.
Evaluation covers conversational features, dialogue quality, instruction following, and safety, rather than fluency alone. Current full-duplex models struggled to hold performance steady across rounds and dimensions, which the authors take as evidence the benchmark is needed. Code and data are public.
Catalog record
| CREATOR | He Zhang Wenqian Cui Haoning Xu Xiaohui Li Lei Zhu Shaohua Ma Irwin King |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-10262
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models He Zhang Wenqian Cui Haoning Xu Xiaohui Li Lei Zhu Shaohua Ma Irwin King Abstract Full-Duplex Speech Language Models (FD-SLMs) enable real-time, overlapping conversational interactions, offering a more dynamic user experience compared to traditional half-duplex models. However, existing benchmarks primarily focus on evaluating single-round interactions and conversational features, neglecting the complexities of multi-round communication and critical capabilities such as instruction following and safety. Evaluating FD-SLMs in multi-round settings poses significant challenges, including blurred turn boundaries in communication and context inconsistency during model inference. To address these gaps, we introduce MTR-DuplexBench, a novel benchmark that segments continuous full-duplex dialogues into discrete turns, enabling comprehensive, turn-by-turn evaluation of FD-SLMs across dialogue quality, conversational dynamics, instruction following, and safety. Experimental results reveal that current FD-SLMs face difficulties in maintaining consistent performance across multiple rounds and evaluation dimensions, highlighting the necessity and effectiveness of our proposed benchmark. 1 Introduction Speech Language Models (SLMs) are foundation models designed for seamless, end-to-end interaction with users through speech ( 5 ) . These models excel in comprehending various aspects of user speech, including meaning, prosody, voice characteristics, and in generating appropriate responses. Most SLMs are built to support half-duplex communication (HD-SLMs), wherein the model first “listens” to a complete user query and then “speaks” the corresponding response. A more advanced subset of SLMs, known as Full-Duplex Speech Language Models (FD-SLMs), specializes in enabling full-duplex communication . This capability allows the model to concurrently listen and speak during real-time interactions, fostering complex conversational features, such as interruptions and backchannels, thereby creating a more dynamic and engaging user experience. When evaluating FD-SLMs, existing benchmarks primarily assess the model’s ability to interact in a single round of interaction. For instance, Full-Duplex-Bench ( 19 ) generates scenarios where the user engages in continuous speech for one turn, pauses mid-turn, or interrupts the assistant. Full-Duplex-Bench v1.5 ( 19 ) builds on this by introducing speech inputs with overlapping interruptions, backchannels, and similar dynamics. However, real-world conversations often unfold in a multi-round format, and evaluating FD-SLMs in such settings is crucial to ensure the model consistently delivers timely and high-quality speech even after multiple rounds of communication. Multi-round communication evaluation for FD-SLMs is challenging, mainly due to the following features in the full duplex dialogues. 1) Blurred Turn Boundary. Unlike HD-SLMs, where conversations are naturally organized in a turn-by-turn manner, full duplex communication occurs spontaneously, without adhering to a strict turn-taking structure. Consequently, there are no clear indicators marking the start or end of each speaking turn for either the user or the assistant. This lack of defined boundaries makes it difficult to assess the response quality for a specific communication round. 2) Context Inconsistency. In natural full-duplex communication datasets, the user’s speech in each round depends on the ground-truth assistant’s responses from previous rounds. However, during evaluation, the model’s responses to earlier rounds may diverge significantly from the ground truth, creating a context mismatch. This inconsistency leads the model to operate in scenarios that would never occur in real-world conversations, ultimately diminishing the reliability of the evaluation. Figure 1 illustrates the two challenges. Figure 1: Illustration of the Blurred Turn Boundary and the Context Inconsistency challenges in the multi-round full duplex evaluations. Another notable limitation of existing benchmarks is their predominant focus on evaluating FD-SLMs’ ability to manage various conversational features, while largely overlooking other critical capabilities of the model, such as instruction following ( 21 ; 40 ; 37 ) and ensuring safe outputs ( 28 ; 30 ; 15 ) . These capabilities need to be evaluated accordingly since they pose unique challenges to FD-SLMs. For example, during a full-duplex conversation, the model may be interrupted at any moment, raising an essential question: can the model maintain expected behavior after undergoing multiple rounds of interruptions? Table 1: Comparison of the supported features between MTR-DuplexBench and other full duplex benchmarks. MRD, TE, CF, DQ, IF represent Multi-round Dialogue, Turn-by-turn Evaluation, Conversational Features, Dialogue Quality, Instruction Following, respectively. Benchmarks MRD TE CF DQ IF Safety Full-Duplex-Bench ✗ ✗ ✓ ✓ ✗ ✗ Full-Duplex-Bench v1.5 ✗ ✗ ✓ ✗ ✗ ✗ Full-Duplex-Bench v2 ✓ ✗ ✓ ✗ ✓ ✓ FD-Bench ✓ ✗ ✓ ✓ ✗ ✗ Talking Turns ✓ ✗ ✓ ✗ ✗ ✗ MTR-DuplexBench ✓ ✓ ✓ ✓ ✓ ✓ To address the challenges and limitations mentioned above, we introduce a novel benchmark, MTR-DuplexBench, designed to comprehensively evaluate FD-SLMs in a multi-round communication setting. First, our benchmark employs an innovative methodology to segment continuous full-duplex dialogues into discrete turns, enabling effective evaluation of FD-SLMs for each turn separately. This approach directly tackles key challenges of the blurred turn boundary and context inconsistency. Second, our benchmark provides a more comprehensive evaluation framework compared to existing benchmarks, encompassing aspects such as dialogue quality, conversational features, instruction following, and safety. This holistic approach ensures thorough and reliable assessments of FD-SLMs. Table 1 presents a detailed comparison between MTR-DuplexBench and other established benchmarks. To summarize our contributions: 1. We propose a novel benchmark, MTR-DuplexBench, designed to enable comprehensive evaluation of the multi-round interactions of FD-SLMs. 2. We introduce a turn segmentation methodology for segmenting continuous full-duplex dialogues into discrete turns, enabling turn-by-turn evaluations of FD-SLMs. 3. We integrate datasets covering comprehensive evaluation dimensions into the benchmark, including dialogue quality, conversational dynamics, instruction following, and safety, along with their evaluation metrics tailored for full-duplex communication settings. 4. We demonstrate through experiments that existing FD-SLMs struggle to maintain consistent performance across multiple rounds and evaluation dimensions, highlighting the necessity and effectiveness of our proposed benchmark. 2 Related Works 2.1 Speech Language Models Speech Language Models (SLMs) refer to speech-based autoregressive foundation models that enable end-to-end spoken dialogue interactions ( 5 ) . As mentioned before, SLMs can be categorized into HD-SLMs and FD-SLMs. Half Duplex Speech Language Models (HD-SLMs) refer to the SLMs that enable “listening” and “speaking” in a turn-by-turn fashion. In this setup, the user provides a complete query to the model, which then generates the corresponding spoken response ( 14 ; 13 ; 38 ) . Therefore, HD-SLMs are designed to handle the concatenated sequence comprising the user query and the assistant’s response. HD-SLMs are typically continually trained upon existing Text Large Language Model (TLM) ( 39 ; 1 ) checkpoints through a two-stage process. 1) Pre-training. In this initial stage, the model uses a vast amount of speech data to learn autoregressive speech generation through a next-token prediction objective. This process may involve pure speech data for unsupervised learning ( 38 ; 32 ) or text-speech alignment pairs ( 25 ; 36 ) to transfer knowledge from text to speech. 2) Instruction Tuning. After pre-training, the model is trained on speech instruction tuning datasets, enabling it to respond effectively to spoken queries ( 35 ; 9 ) . During inference, some HD-SLMs generate speech directly ( 10 ; 25 ) , while others first generate text as a guide and then produce the corresponding speech ( 20 ; 3 ) . Full Duplex Speech Language Models (FD-SLMs) refer to the SLMs that enable “listening” and “speaking” simultaneously to support advanced conversational features such as interruption and backchannel. FD-SLMs are typically built