Joint Learning Global-Local Speaker Classification to Enhance End-to-End Speaker Diarization and Recognition

Yuhang, Haopeng, Jiale, Ruiqi, Hao, Hanke, Hanlin, Shunshun, Ming, Xie, Lei, Xinsheng · 2026

Read Joint Learning Global-Local Speaker Classification to Enhance End-to-End Speaker Diarization and Recognition by Yuhang, Haopeng, Jiale, Ruiqi, Hao, Hanke, Hanlin, Shunshun, Ming, Xie, Lei, Xinsheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25377.txt.

Introduction

Dai and co-authors train speaker naming and speech transcription as one joint job. Big audio-language models can diarize and recognize speech end to end. But telling voices apart lags, since chat data is scarce and no loss targets speaker traits. Their GLSC-SDR adds speaker classification to the training mix. Clustered speakers give global labels, and re-encoded cluster members give fine local ones. On three meeting sets it matches or beats simulation-fed and multi-encoder rivals, with no large real chat corpus.

Catalog record

CREATORYuhang, Haopeng, Jiale, Ruiqi, Hao, Hanke, Hanlin, Shunshun, Ming, Xie, Lei, Xinsheng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22770
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.25377

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Joint Learning Global-Local Speaker Classification to Enhance End-to-End Speaker Diarization and Recognition Yuhang, Haopeng, Jiale, Ruiqi, Hao, Hanke, Hanlin, Shunshun, Ming, Xie, Lei, Xinsheng Abstract Large Audio-Language Models (LALMs) have demonstrated remarkable performance in end-to-end speaker diarization and recognition. However, their speaker discriminability remains limited due to the scarcity of large-scale conversational data and the absence of explicit speaker representation optimization. To address this, we propose GLSC-SDR, a paradigm that jointly trains speaker classification with diarization and recognition. We further introduce a Global-Local Speaker Classification strategy, which uses clustered speakers as global labels and re-encoded intra-cluster speakers as local labels. This hierarchical design enhances fine-grained speaker discrimination while preserving semantic transcription accuracy. Experiments on AliMeeting, AISHELL-4, and AMI-SDM demonstrate that GLSC-SDR achieves competitive or superior performance compared to simulation-based and multi-encoder approaches, without relying on large-scale real conversational data. Dai Lin Qian Yan Meng Xie Wen Yin Tao Chen Xie Wang keywords joint learning, automatic speech diarization and recognition, LALMs † † address: 1 Audio, Speech and Language Processing Group (ASLP@NPU), School of Computer Science Northwestern Polytechnical University 2 Soul AI Lab, China 3 X-LANCE Lab, Shanghai Jiao Tong University, China † † email: [email protected], [email protected], [email protected] 1 Introduction Multi-speaker conversational transcription requires solving the fundamental problem of “who spoke what and when”, commonly referred to as Speaker Diarization and Recognition (SDR). Real-world conversations are characterized by rapid turn-taking, speaker overlaps, and acoustic variability, making robust speaker attribution a critical challenge [ 1 , 17 ] . Traditional SDR systems typically adopt a cascaded pipeline composed of Voice Activity Detection (VAD), Speaker Verification (SV), and Automatic Speech Recognition (ASR). Although modular and interpretable, such architectures suffer from error propagation and high system complexity. Recent advances in Large Audio-Language Models (LALMs) have demonstrated strong capabilities in unified end-to-end multi-speaker processing [ 7 , 20 ] , enabling structured sequence generation that jointly models timestamps, speaker labels, and transcripts [ 15 , 14 ] . To enhance speaker attribution, several approaches extend the basic LALM framework with auxiliary components. For example, TagSpeech [ 8 ] incorporates an additional speaker encoder together with a time-anchoring strategy to improve fine-grained timestamp and speaker label prediction. SpeakerLM [ 25 ] introduces a speaker registration mechanism to strengthen identity consistency across segments. Other works explore post-processing strategies [ 22 , 16 ] , such as leveraging large language models to refine diarization outputs and improve transcript readability [ 22 ] . While these efforts demonstrate that LALMs can be adapted to improve multi-speaker transcription, most approaches rely on architectural augmentation or inference-time refinement rather than explicitly optimizing speaker representation learning within the model. Consequently, speaker identity modeling remains largely implicit, and the learned representations may lack sufficient intra-speaker compactness and inter-speaker separability—particularly when conversational training data are limited or when acoustically similar speakers must be distinguished. Figure 1: Joint training Global-Local Speaker Classification and Speech Diarization and Recognition paradigm. To overcome the limited speaker discriminability of current end-to-end models, we draw inspiration from the evolution of the Speaker Verification (SV) systems. Over the past decade, SV has evolved from statistical modeling [ 19 , 4 ] to deep representation learning [ 21 ] . Building upon this paradigm, recent models further introduce metric learning and margin-based objectives within architectures such as ResNet and ECAPA-TDNN [ 5 ] to explicitly structure the embedding space. As a result, modern SV systems learn speaker representations with improved intra-class compactness and inter-class separability, enabling strong generalization to unseen speakers. Inspired by these advances, we argue that although LALMs achieve strong performance in speech recognition and semantic modeling due to their powerful backbones [ 27 , 12 ] , their speaker representation learning remains largely implicit. As a result, the models may struggle to distinguish acoustically similar speakers in complex conversational scenarios. To address this limitation, we introduce explicit speaker-aware supervision into LALMs by integrating speaker classification objectives into the training process. Specifically, we propose a new paradigm termed G lobal- L ocal S peaker C lassification jointly trained with S peaker D iarization and R ecognition (GLSC-SDR). By combining hierarchical speaker classification with the SDR objective, the proposed approach enhances the speaker discriminability of LALMs while preserving their end-to-end modeling capability, enabling more accurate multi-speaker transcription and tracking in conversational speech. The main contributions of this paper are summarized as follows: • We propose GLSC-SDR , a fully end-to-end joint training paradigm that tightly integrates speaker classification with the SDR task. Given multi-speaker audio as input, it directly produces structured outputs containing timestamps, speaker labels, and corresponding speech transcripts within a unified architecture. • We introduce a novel GLSC strategy tailored for LALMs. By incorporating hierarchical clustering and a speaker re-encoding mechanism, it provides both global and local speaker labels, which significantly enhance cross-segment speaker discriminability at the input representation level while requiring no architectural modification to the LLM backbone. • Extensive experiments on multi-speaker benchmarks, including AliMeeting, AISHELL-4, and AMI-SDM, demonstrate that the proposed paradigm substantially improves speaker diarization and recognition performance in complex conversational scenarios. 2 Method Effectively enhancing a model’s ability to discriminate among different speakers is critical for the SDR task. The core innovation of our approach lies in the integration of a hierarchical Global-Local Speaker Classification mechanism, which is jointly optimized with the Speaker Diarization and Recognition task. Diverging from traditional flat classification approaches, GLSC adopts a “macro-acoustic clustering followed by micro-individual identification” strategy. This compels the model to simultaneously acquire cross-segment global consistency and fine-grained local discriminability at the input representation level. Such a design not only effectively mitigates the risk of overfitting in large-scale classification through an implicit curriculum learning effect but also significantly enhances the model’s sensitivity to acoustic differences in complex multi-speaker scenarios. Notably, this paradigm achieves substantial performance gains solely through joint multi-task constraints without requiring additional network modules, as illustrated in Figure 1 . 2.1 Problem Formulation The goal of our GLSC-SDR framework is to improve the performance of SDR on multi-speaker audio. Given a raw waveform S , SDR aims to generate a structured sequence O=(o_{1},o_{2},...,o_{l}) containing timestamps, speaker labels, and transcriptions. While standard LALMs can model this sequence, their inherent speaker discriminability is often limited, particularly in complex multi-speaker scenarios. To enhance SDR, we introduce the GLSC mechanism as a specialized training task within the LLM. GLSC generates a sequence of speaker labels for a single audio segment, where each label consists of a global speaker cluster label L_{g} and a local speaker identifier L_{u} within that cluster. This task explicitly teaches the model to recognize fine-grained speaker differences and cluster-level consistency, which indirectly improves the speaker attribution capability of the SDR task. During training, the GLSC and SDR tasks are both serialized into unified sequence formats, with a special task token distinguishing them. Notably, GLSC sequences are single-utterance samples rather than full conversational data. The combined training objective is defined as: L=\alpha L_{GLSC}+(1-\alpha)L_{SDR} (1) where L_{GLSC} and L_{SDR} denote the losses for the GLSC and SDR tasks, respectively, and \alpha serves as the weighting coefficient. During inference, the model adheres to the Serialized Output Training (SOT) [ 9 ] paradigm, generating a structured sequence containing timestamps, speaker labels, and transcriptions. Table 1: Main results in Alimeeting,

Read the complete text