Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder PubID: pubid: © 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or pro
Muhammad Shakeel 1 , Yui Sudo 1 , Yifan Peng 2 , Chyi-Jiunn Lin 2 , Shinji Watanabe 2 · 2025
Read Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder PubID: pubid: © 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or pro by Muhammad Shakeel 1 , Yui Sudo 1 , Yifan Peng 2 , Chyi-Jiunn Lin 2 , Shinji Watanabe 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20474.txt.
Introduction
Muhammad Shakeel and co-authors present in a 2025 paper one encoder that serves speaker diarization, speech separation, and recognition together. The catalog title carries an IEEE notice, while the text gives the clean title. Their unified multi-speaker encoder learns inner codes for the three tasks jointly, through a shared speech encoder, rather than training three stand-alone systems.
Hidden codes from several layers are joined as a residual weighted-sum encoding. That move draws well on information from different levels of meaning and backs matching between the tasks from the bottom up. Training the tasks together catches the links they share by nature. Tests show big gains over single-task baselines on LibriMix sets for all three tasks. Readers get one model where three pipelines stood.
Catalog record
| CREATOR | Muhammad Shakeel 1 , Yui Sudo 1 , Yifan Peng 2 , Chyi-Jiunn Lin 2 , Shinji Watanabe 2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-20474
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder PubID: pubid: © 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Muhammad Shakeel 1 , Yui Sudo 1 , Yifan Peng 2 , Chyi-Jiunn Lin 2 , Shinji Watanabe 2 Abstract This paper presents a unified multi-speaker encoder (UME), a novel architecture that jointly learns representations for speaker diarization (SD), speech separation (SS), and multi-speaker automatic speech recognition (ASR) tasks using a shared speech foundational encoder. We leverage the hidden representations from multiple layers of UME as a residual weighted-sum encoding (RWSE) to effectively use information from different semantic levels, contributing to bottom-up alignment between tasks. This joint training approach captures the inherent interdependencies among the tasks, enhancing overall performance on overlapping speech data. Our evaluations demonstrate that UME substantially improves over the single-task baselines dedicated to SD, SS, and multi-speaker ASR on LibriMix evaluation sets. Notably, for SD, UME outperforms the previous studies, achieving diarization error rates of 1.37% and 2.29% on Libri2Mix and Libri3Mix evaluation sets, respectively. I Introduction Speaker diarization (SD), speech separation (SS), and multi-speaker automatic speech recognition (ASR) are tasks of great importance that aim to comprehend and answer the question “who spoke what and when,” with applications to transcribing meetings and interviews, among others. Previous studies in SD [ 1 , 2 , 3 ] , SS [ 4 , 5 , 6 ] , and multi-speaker ASR [ 7 , 8 , 9 , 10 ] have focused primarily on improving the quality of single-task models that operate independently on acoustic information to separate or label speaker segments and transcribe the text in a speech-processing system [ 11 , 12 , 13 ] . A key limitation of training tasks independently is that inter-dependencies cannot be leveraged. Most existing frameworks [ 13 , 14 , 15 ] address this limitation by unifying speech-processing architectures [ 16 , 17 ] . These architectures consist of either a joint ASR/SD [ 18 , 19 , 20 ] , SS/ASR [ 21 , 22 , 23 ] , or a SD/SS [ 24 , 25 ] task following a fixed optimal order that can vary depending on the target scene scenario [ 26 , 27 , 28 , 29 ] . These different target scenes suggest that we solve these tasks jointly, independent of the order, so all these tasks can benefit from each other. Lately, there has been a shift towards employing pre-trained speech foundation models (SFMs) [ 30 , 31 , 32 , 33 , 34 ] in end-to-end (E2E) systems, which effectively learn useful representations for various speech processing tasks [ 35 ] . However they do not work well on multi-speaker conversation recognition. Additionally, it has been demonstrated that different layers encode different types of information in SD and ASR tasks [ 35 , 32 ] . Preliminary observation from these studies shows that intermediate layers of the encoder extract a rich hierarchy of information, e.g., in WavLM large [ 32 ] , initial layers and last layers are more critical for SD and ASR tasks. Therefore, it makes sense to utilize multiple layers to jointly optimize all SD, SS, and ASR tasks effectively. The question, therefore, naturally arises: can we build a unified model that leverages all encoder layers to optimize performance across multiple tasks? Motivated by the potential of SFMs and E2E speech processing, we propose a unified multi-speaker encoder (UME), a novel E2E speech-processing framework. The proposed framework is generalizable to use any SFM, E2E SD, SS and multi-speaker ASR task and jointly optimizes all these tasks into a single network with multitask learning to minimize the error accumulation for a speech processing framework. Additionally, by extracting features from all the layers of the OWSM (open Whisper style speech model [ 36 ] ) v3.1 encoder and using it as a residual weighted-sum encoding (RWSE), we can learn better hidden representations from the encoder layers. We hypothesize that RWSE introduces information exchange and better bottom-up alignment to all the tasks from different semantic levels. We argue that UME framework should provide a shared representation space for SD, SS and multi-speaker ASR tasks and preferably have strong generalizability and learnability. We conduct extensive experiments on different design choices of UME on Libri2Mix & Libri3Mix [ 37 ] datasets. Our key contributions are: • We propose a unified speech-processing framework to jointly optimize the performance of SD, SS, and multi-speaker ASR tasks with hidden representations of the SFM encoder. • We propose using RWSE of the pre-trained SFM encoder layers to unify and optimize the performance across diverse speech processing tasks. • We demonstrate the effectiveness of our framework on two-speaker and three-speaker overlapped speech and obtain substantial performance improvement in each diarization, separation, and multi-speaker ASR task. II Unified Multi-Speaker Encoder (UME) Figure 1 shows the overall framework of UME. It leverages the hidden representations through an RWSE of intermediate layers, which act as a bridge between SD, SS, and multi-speaker ASR tasks. This enables a comprehensive and detailed interaction from each layer of the SFM encoder. Note that our goal is not to develop a new encoder or speech processing tasks; in principle, one can apply any SFM encoder, SD, SS, or multi-speaker ASR tasks. We start with the T -length single-channel input speech mixture. X=\{x_{t}\in\mathbb{R}|t=1,\cdots,T\} of C speakers. We define the input speech mixture in an anechoic condition by: X=\sum_{c=1}^{C}Y^{c}\odot S^{c}+N, (1) where, S^{c}=\{s^{c}_{t}\in\mathbb{R}|t=1,...,T\} is the clean speech signal of speaker c , Y^{c}=\{y^{c}_{t}\in[0,1]\mid t=1,\cdots,T\} is a binary speech activity sequence indicating whether speaker c is talking at time t or not, and N=\{n_{t}\in\mathbb{R}\mid t=1,\cdots,T\} is the additive noise signal. Together, the activity sequences \{Y^{c}\}_{c=1}^{C} form the ground truth speaker label for the speaker diarization (SD) task described in Section II-B . II-A Speech Foundation Model Encoder We selected OWSMv3.1 [ 36 ] as the shared encoder due to its widespread recognition, reproducibility, open-source availability, and fast, efficient encoding capabilities. We can note that OWSM was trained on single-speaker speech-to-text tasks (i.e., no speaker tasks in pre-training). However, we can still adapt it to our multi-speaker setup. The speech encoder is a stack of L E-Branchformer [ 38 ] encoder layers that transforms the input speech mixture X of C speakers into a D^{\text{enc}} -dimensional subsampled T^{\text{enc}}(<T) -length hidden state representations H_{(l)}=\{\mathbf{h}_{(t,l)}\in\mathbb{R}^{D^{\text{enc}}}|t=1,\cdots,T^{\text{enc}}\} , where l is a layer index from 1 to L . The simplified speech encoder is given by: H_{(l)}=\mathrm{SpeechEnc}_{(l)}(X). (2) To integrate into task-specific models, all encoder layers are combined into a single feature vector via a weighted sum [ 35 ] : H^{\text{ws}}=\sum_{l=1}^{L}\omega_{(l)}^{\text{task}}H_{(l)}, (3) where \omega^{\text{task}}_{(l)} are softmax-normalized learnable weights that scale the representations from different encoder layers depending on each task . Finally, RWSE is introduced by adding H^{\text{ws}} to the last encoder layer ( l=L ) to amplify the influence of the final layer according to [ 32 ] across tasks during training. H^{\text{enc}}=H^{\text{ws}}+H_{(l=L)}.