End-to-End Diarization utilizing Attractor Deep Clustering

David, Matthew, Eric · 2025

Read End-to-End Diarization utilizing Attractor Deep Clustering by David, Matthew, Eric (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11090.txt.

Introduction

David Palzer and co-authors wrote a 2025 paper on a compact system that tells who speaks when, in a recording of mixed speech, a task called diarization. The task is hard: it needs clean representations for each voice, efficient work, and robustness under rough conditions. Their system joins three parts. A deep listening network reads the recording. Anchor points, called attractors, are refined by a transformer. A clustering technique then sorts the anchor points by type.

The clustering technique is extended here, with label vectors built for each anchor point. The embeddings are aligned by their direction, not just their position. Set rules keep active anchors apart and suppress inactive ones, so a silent voice is not falsely called active. A final loss function, blind to the order of speakers, trains the detection of who is speaking. The system achieves low error rates while staying small, the team says.

Catalog record

CREATORDavid, Matthew, Eric
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-11090

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

End-to-End Diarization utilizing Attractor Deep Clustering David, Matthew, Eric Abstract Speaker diarization remains challenging due to the need for structured speaker representations, efficient modeling, and robustness to varying conditions. We propose a performant, compact diarization framework that integrates conformer decoders, transformer-updated attractors, and a deep clustering style angle loss. Our approach refines speaker representations with an enhanced conformer structure, incorporating cross-attention to attractors and an additional convolution module. To enforce structured embeddings, we extend deep clustering by constructing label–attractor vectors, aligning their directional structure with audio embeddings. We also impose orthogonality constraints on active attractors for better speaker separation while suppressing non-active attractors to prevent false activations. Finally, a permutation invariant training binary cross-entropy loss refines speaker detection. Experiments show that our method achieves low diarization error while maintaining parameter count. Palzer Maciejewski Fosler-Lussier The Ohio State UniversityU.S.A. The Johns Hopkins UniversityU.S.A. keywords speaker diarization, conformer, attractors, DPCL, angle loss, orthogonality constraints † † email: [email protected], [email protected], [email protected] 1 Introduction Speaker diarization, the task of segmenting an audio stream into homogeneous speaker segments, remains a challenging problem due to the inherent variability in acoustic conditions, the presence of overlapping speech, and the need to robustly handle a wide range of speakers [ 1 , 2 ] . End-to-end neural diarization systems (EEND) [ 3 ] allow for direct minimization of diarization errors during training. The original EEND included an intermediate auxiliary loss of Deep Clustering (DPCL) [ 4 ] to help discriminate speakers; DPCL itself was introduced as a method to explicitly create discriminable embedding vectors for an arbitrary number of classes and was evaluated on speech separation. Further work in the EEND line led to Encoder-Decoder Attractors (EDA) [ 5 ] where an EDA module was responsible for encoding discriminable speaker representations directly from audio embeddings, allowing for a flexible number of speakers to be separated. Neither this work nor subsequent EEND studies (e.g., [ 6 , 7 ] ) utilized DPCL. In this work, we propose a performant, compact diarization framework that synthesizes the strengths of conformer-based decoders, transformer-updated attractors, and an improved DPCL-style angle loss to promote robust, structured representations. By enhancing a conformer architecture with cross-attention to attractors and incorporating additional convolutional modules, we capture both local and global speaker information while keeping the network size manageable. We also reintroduce DPCL loss into the EEND framework (first seen in [ 3 ] but absent in subsequent EEND models) with a new extension to enforce directional alignment between label–attractor vectors and audio embeddings, leveraging orthogonality constraints on active attractors to reduce cross-speaker confusion. Finally, we include a permutation invariant training (PIT) [ 8 ] binary cross-entropy (BCE) loss term to refine speaker detection, effectively suppressing non-active attractors and minimizing false alarms. Figure 1: Example four layer EEND-CD architecture. Each block is described in subsections of Section 3 . Experiments demonstrate that our approach achieves competitive or superior diarization error rates (DER), while maintaining efficiency due to the compactness of the architecture. The key contributions of this paper are: • An enhanced conformer decoder design that integrates Latte based attention [ 9 ] and an attractor-based cross-attention for refined speaker representation. • An improved DPCL-style angle loss extension using label–attractor vectors to enforce structured embedding alignments and orthogonality constraints. • A novel speaker detection refinement strategy that explicitly works to reduce false attractor activations. The rest of the paper is organized as follows. In Section 2 , we discuss related studies in speaker diarization and representation learning. Our proposed method, including the conformer architecture, attractor updating mechanism, and extended DPCL loss, is described in Section 3 . Section 4 details experimental setups and datasets, and Section 5 presents our results, discussion, and future directions. 2 Related Work Speaker diarization involves partitioning an audio stream according to speaker identity, a task that has traditionally relied on statistical modeling and clustering approaches. Early methods often combined Gaussian Mixture Models (GMM) or i-vector [ 10 ] representations with clustering [ 1 , 11 ] , later transitioning to x-vectors [ 12 ] for more robust speaker embeddings. According to Park et al. , [ 2 ] , ”many traditional speaker diarization systems, especially clustering-based systems, have only focused on non-overlapping regions and even the overlapping regions are excluded in the evaluation metric. (pg 22)” Deep learning-based strategies have accelerated progress in speech and diarization [ 2 ] , especially with the introduction of End-to-End Neural Diarization (EEND) [ 3 ] . DPCL projects time-frequency bins (or time frames) into an embedding space, then applies k-means to cluster an arbitrary number of speakers. In contrast, EEND directly predicts speaker activities in a permutation-invariant manner, but requires speaker counts a priori. More recently, attractor-based approaches [ 5 , 7 , 6 ] introduced learnable speaker prototypes (attractors) that iteratively refine speaker estimates, enabling the EEND models to adapt to an unknown or variable number of speakers. Despite these advances, challenges remain in capturing both local and global acoustic context efficiently, as well as maintaining scalability to longer recordings. Work incorporating label dependency has been explored by [ 6 ] , where intermediate attractors are utilized. These attractors are computed as a weighted sum of intermediate diarization predictions and subsequently added to each time frame before being processed by subsequent layers. This approach has been shown to reduce the missed speech rate, albeit at the cost of an increased false alarm rate. The conformer architecture [ 13 ] has proven effective in speech processing tasks by combining local convolutions with global self-attention, improving upon pure transformer-based models in Automatic Speech Recognition (ASR). In diarization, conformer-based encoders have shown promise for generating robust speaker embeddings [ 7 , 14 , 15 ] . However, we are not aware of any diarization works that have explored deep conformer-based decoders that also integrate attractor-based cross-attention. In parallel, various forms of optimized self-attention have been proposed to reduce quadratic complexity or to focus on latent representations of the sequence [ 16 , 17 ] . Our work adopts a latent self-attention (Latte [ 9 ] ) mechanism within each conformer decoder, providing an efficient means of capturing global context across frames while maintaining task performance. Finally, angle-based losses, such as the DPCL-style angle loss, have been shown to encourage discriminable embeddings which show improvements in related fields such as speaker separation [ 4 , 18 ] . Following papers have pursued this idea further [ 19 ] and offered alternative loss formulations. 3 Proposed Method In this section, we introduce our diarization framework, which combines a CNN-based feature extractor, conformer blocks with latent self-attention (Latte), attractor-based cross-attention, depthwise pooled residual connections, and a DPCL-style angle loss for structured embeddings. Figure 1 illustrates the overall architecture. 3.1 Overview of the Architecture Our framework takes a batch of recordings as input, where the recordings are processed into 23-dimensional mel features at a 10 ms frame rate. These frames are further stacked into windows of 15 frames using a hop of 10 frames such that each window overlaps with the next by 5 frames. We represent this input as a 4D tensor of shape Batch × Time × Window × Mels . Previous models flatten these into a 3D tensor \text{Batch}\times\text{Time}\times(\text{Window}\times\text{Mels})

Read the complete text