Improving Neural Diarization through Speaker Attribute Attractors and Local Dependency Modeling

David Palzer, Matthew Maciejewski, Eric Fosler-Lussier · 2025

Read Improving Neural Diarization through Speaker Attribute Attractors and Local Dependency Modeling by David Palzer, Matthew Maciejewski, Eric Fosler-Lussier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05593.txt.

Introduction

David Palzer, Matthew Maciejewski, and Eric Fosler-Lussier published a computing paper in 2025 on sorting out who speaks when in recordings with several talkers. End-to-end methods have advanced this task, called diarization. One method builds attractor points to handle a varying count of speakers. The team extends that idea in two ways.

First, they model detailed speaker attributes through stages of intermediate forms, rather than modeling speakers directly. Second, they swap transformers for conformers, a variant that also tracks local patterns. Tests on the CALLHOME dataset showed better diarization scores. The authors present the changes as a stronger base for neural diarization.

Catalog record

CREATORDavid Palzer, Matthew Maciejewski, Eric Fosler-Lussier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05593

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Neural Diarization through Speaker Attribute Attractors and Local Dependency Modeling David Palzer, Matthew Maciejewski, Eric Fosler-Lussier Abstract In recent years, end-to-end approaches have made notable progress in addressing the challenge of speaker diarization, which involves segmenting and identifying speakers in multi-talker recordings. One such approach, Encoder-Decoder Attractors (EDA), has been proposed to handle variable speaker counts as well as better guide the network during training. In this study, we extend the attractor paradigm by moving beyond direct speaker modeling and instead focus on representing more detailed ‘speaker attributes’ through a multi-stage process of intermediate representations. Additionally, we enhance the architecture by replacing transformers with conformers, a convolution-augmented transformer, to model local dependencies. Experiments demonstrate improved diarization performance on the CALLHOME dataset. † † address: ∗ The Ohio State University, Computer Science and Engineering † The Johns Hopkins University, Human Language Technology Center of Excellence 1 Introduction Speaker diarization is the activity of labeling an audio recording with continuous segments by speaker identity. Diarization has a large range of applications, such as pre-processing audio for downstream tasks like source separation, information retrieval for pre-recorded audio, speaker-turn analysis for call recordings, and processing meetings [ 1 ] . Diarization can also improve ASR performance for multi-speaker recordings (ICSI [ 2 ] , AMI [ 3 ] ) and acoustically noisy environments such as the family home (CHiME-5 [ 4 ] ). The lack of approaches that directly minimize diarization errors prompted the development of End-to-End Speaker Diarization (EEND) by Fujita et al. [ 5 ] . EEND is designed to address this challenge by optimizing diarization errors directly while using Permutation Invariant Training (PIT) to address speaker-label ambiguity [ 6 ] . Notably, EEND, particularly the Self-Attentive EEND (SA-EEND) [ 7 ] , demonstrates the effectiveness of end-to-end training by surpassing traditional clustering-based methods. However, EEND has the limitation that the maximum number of speakers it can handle is predefined. To tackle this limitation, Horiguchi et al. introduced Encoder-Decoder Attractor (EDA) models [ 8 ] that flexibly determine how many attractors are needed during the decoding of a speech embedding sequence. This is achieved through compressing all speakers into a time-independent embedding and then separating out speaker attractors until the network predicts that all speakers have been produced. Every layer in the iterative network is required to predict a set of attractors. In this study, we introduce speaker attribute attractors as a further improvement to the EEND-EDA architecture that disconnects layer wise prediction of speaker attractors from the iterative refinement, which provides a more robust way to condition deeper layers of the model. Additionally we integrate a conformer into our foundation model to inject local temporal dependencies, an approach that has shown promise on related audio processing tasks, which displays improvements in identifying speakers and finding speech. 2 Related Work Traditional diarization solutions [ 9 , 10 ] utilize clustering on speaker embeddings such as i-vectors [ 11 ] , d-vectors [ 12 ] , and x-vectors [ 13 ] . These speaker time embeddings are clustered using traditional ML algorithms, such as GMMs [ 9 ] , or hierarchical clustering [ 14 ] . Neural diarization techniques like EEND [ 5 ] employ an end-to-end neural network architecture that takes audio features as its input and produces the collective speech activities of multiple speakers as its output. The optimization of this network utilizes the entire recording, encompassing non-speech segments and instances of speaker overlap, with a primary focus on minimizing diarization errors. Within the domain of speech separation, various techniques have been developed to handle mixtures involving a variable number of speakers. One set of methods follows the one-vs-rest approach, which is applied iteratively to separate speakers [ 15 , 16 , 17 ] . However, a significant drawback of this approach lies in the fact that calculations continue until all speakers are isolated, resulting in a linear increase in computational time as the number of speakers rises. Another set of methods adopts attractor-based strategies, one example being the Deep Attractor Network (DANet) [ 18 ] . While these approaches do not impose limits on the number of speakers during inference, they do require prior knowledge of the number of speakers. Anchored DANet [ 19 ] effectively resolves these challenges; however, it still necessitates calculating dot products for all potential combinations of anchors and extracted embeddings during the inference phase. Consequently, its scalability with respect to the number of speakers is limited. Further work on introducing label-dependency has been done by [ 20 ] with the use of intermediate attractors. These intermediate attractors are added to each time frame as a weighted sum using intermediate diarization predictions before being passed to further layers. They find that this helps lower the missed speech rate, but also results in a higher false alarm rate. 3 Methods This section describes the original EEND-EDA [ 21 ] and EEND-EDA-deep [ 20 ] architectures followed by our proposed improvements to both models. 3.1 EEND-EDA EEND-EDA modifies the original EEND network to use a stack of L transformers instead of LSTMs and to allow for an arbitrary number of speakers, which is not known a priori. E_{l}=\mathrm{Encoder}(E_{l-1})\hskip 36.135pt(1\leq l\leq L), (1) where E_{l}\in\mathbb{R}^{T\times D} is the sequence of embeddings of length T , and dimension D . The original EEND produced a static number of speakers, whereas with the inclusion of the EDA modification, EEND-EDA produces an attractor for each of S speakers through the use of an LSTM and auxiliary existence loss for each produced attractor, a_{s} where (1\leq s\leq S) . A=\mathrm{EDA}(E_{L}),\mathrm{where}\ A=[a_{1},...,a_{S}]\in\mathbb{R}^{S\times D} (2) These speaker attractors are subsequently used to predict each speaker’s frame-wise label by calculating the dot-product between each frame and each attractor. These speaker predictions are made concurrently and all frames are processed in parallel. Y=\sigma(AE_{L}^{T}) (3) Two of the downsides to this approach are the drop in performance as we produce more attractors, and the lack of label dependency between labels on our output. 3.2 Intermediate Attractors EEND-EDA with Intermediate Attractors was introduced by [ 20 ] . They try to solve the lack of label dependency issue through creating intermediate speaker attractors and conditioning further layers on these attractors. This relaxes the conditional dependence between frames. E_{l}=\mathrm{Encoder}(\hat{E}_{l-1})\hskip 36.135pt(1\leq l\leq L) (4) They produce these intermediate attractors by injecting a shared EDA module between each transformer encoder which predicts the speaker activity at each layer. These shared intermediate EDA layers have an auxilary loss attached to them using the true speaker labels to help guide learning. These predicted speaker activities are then used as the attention weights of each speaker attractor for each frame. A shared linear projection layer is then used to project the frame-wise attractor conditioning before summing it with the current frame-embeddings and passing it to the next transformer encoder. A_{l}=\mathrm{EDA}_{l}(E_{l}) (5) \hat{E}_{l}=E_{l}+\sigma(E_{l}A_{l}^{T})A_{l}^{T}W_{l} (6) 3.3 Attribute Attractors The task of directly modeling speakers, and conditioning further layers on these predictions, ties its expressiveness to how well each layer can individually solve the diarization problem. We instead propose a solution that relaxes this requirement in the form of attribute attractors, a non-autoregressive intermediate representation of our speakers that is more robust than direct speaker modeling. These

Read the complete text