Attractor-Based Speech Separation of Multiple Utterances by Unknown Number of Speakers

Yuzhu, Archontis, Konstantinos, Tuomas · 2025

Read Attractor-Based Speech Separation of Multiple Utterances by Unknown Number of Speakers by Yuzhu, Archontis, Konstantinos, Tuomas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16607.txt.

Introduction

Yuzhu Wang, Archontis Politis, Konstantinos Drossos, and Tuomas Virtanen tackle speech separation when nobody knows how many people are talking. A single recording may mix several speakers, and each may speak in more than one burst. Their model separates the voices, estimates the number of speakers on the fly, and tracks when each speaker is active, all at once.

The design centers on an attractor module that blends local and global time patterns in the sound. That blend suits recordings where the same speaker returns after pauses. To test under hard conditions, the team built a data set by mixing clean speech with real-world noise, adding echo. The system counted the sources correctly and sorted the utterances into the right outputs, whether the speaker count was given in advance or not. The authors present the model as a step toward separation tools for messy, real rooms.

Catalog record

CREATORYuzhu, Archontis, Konstantinos, Tuomas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16607

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attractor-Based Speech Separation of Multiple Utterances by Unknown Number of Speakers Yuzhu, Archontis, Konstantinos, Tuomas Abstract This paper addresses the problem of single-channel speech separation, where the number of speakers is unknown, and each speaker may speak multiple utterances. We propose a speech separation model that simultaneously performs separation, dynamically estimates the number of speakers, and detects individual speaker activities by integrating an attractor module. The proposed system outperforms existing methods by introducing an attractor-based architecture that effectively combines local and global temporal modeling for multi-utterance scenarios. To evaluate the method in reverberant and noisy conditions, a multi-speaker multi-utterance dataset was synthesized by combining Librispeech speech signals with WHAM! noise signals. The results demonstrate that the proposed system accurately estimates the number of sources. The system effectively detects source activities and separates the corresponding utterances into correct outputs in both known and unknown source count scenarios. Wang Politis Drossos Virtanen Tampere UniversityFinland Nokia TechnologiesFinland keywords speech separation, speaker counting, speaker diarization, attractors, transformers † † email: [email protected], [email protected], [email protected], [email protected] 1 Introduction Speech separation aims to isolate individual speech signals from mixtures containing multiple speakers. This task is particularly challenging in real-world environments due to noise, reverberation, and the highly time-varying nature of source activities, especially in single-channel scenarios [ 1 , 2 , 3 , 4 ] . Most existing research assumes a known and fixed number of speakers with each contributing a single utterance [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 ] . However, this assumption does not match with real-world scenarios, where the number of speakers is often unknown and varies over time, with each speaker potentially contributing multiple utterances that may overlap in variable ways. To handle unknown source numbers, one direct approach is recursive separation [ 13 , 14 ] . In contrast, deep clustering methods [ 15 , 16 ] create high-dimensional embeddings for each time-frequency bin, which are then clustered to group together the embeddings corresponding to the same speaker, thereby avoiding the dependency on the prior number of speakers. A similar approach is applied in speaker clustering [ 17 ] . Deep attractor networks [ 18 , 19 ] try to create a representation vector for each sound source in feature space as a reference point. Following the success of attractor networks, systems that combine attractors and separators have gradually emerged as the mainstream solution. RNN-based attractors were explored in joint speaker diarization and speech separation systems [ 20 , 21 ] . More recently, a transformer attractor was proposed in [ 22 ] , which demonstrated excellent performance. The aforementioned studies focus only on single-utterance scenarios. For situations with an unknown number of speakers and multiple utterances, several methods have shown potential effectiveness, including recurrent attention networks [ 23 ] , spatial filtering [ 24 ] , and dual-path model consisting of LSTM and transformers [ 25 ] . However, most of these approaches have concentrated on speaker counting and long-sequence modeling, overlooking source activity detection. Moreover, they primarily target speech recognition, with limited evaluations of separation performance and the impact of noise and reverberation. Figure 1: The illustration of the A-DCSS in multi-speaker with multi-utterance scenarios. Our work aims to address the novel problem of separating an unknown number of speakers, each with multiple utterances, a scenario that has not been evaluated in previous research, as shown in Fig. 1 . Related research exists in continuous speech separation (CSS) [ 26 , 27 ] , which focuses on producing output streams where speech signals do not overlap, typically assuming a known number of sources. In contrast, our work emphasizes separation of unknown number of speakers while ensuring all utterances from the same speaker are consistently assigned to a single output channel, rather than merely de-overlapping the signals. We propose an attractor-based joint diarization, counting, and separation system (A-DCSS) that leverages multi-task learning to simultaneously minimize errors across these tasks. Our system makes four major contributions: (1) We are the first to test an attractor design for multi-utterance separation, validating its effectiveness in source modeling and potential source identification. (2) We integrate RNN attractors with feature-wise linear modulation (FiLM) [ 28 ] and develop an improved multi-path module with transformer-LSTM blocks [ 10 ] . (3) Our ablation studies demonstrate that source activity detection significantly improves separation performance. (4) We retrain and evaluate existing methods [ 13 , 20 , 21 , 22 ] on the synthesized dataset, demonstrating their adaptability to multi-utterance scenarios. Our experimental results show that A-DCSS outperforms baseline systems in both fixed and varying speaker scenarios under anechoic, noisy, and reverberant conditions. 2 Signal Model The observed signal \mathbf{y} of length T^{*} can be represented as \mathbf{y}=\sum_{c=1}^{C}\mathbf{x}_{c}+\mathbf{n}. (1) Here, the observed signal \mathbf{y} consists of speech signals \mathbf{x}_{c} from C sources and noise \mathbf{n} . The signal \mathbf{x}_{c} from the c -th speaker, consists of multiple utterances and potential pauses between them. The speaker counting task involves estimating the number of speakers C . We assume that by extracting features from the time-domain signals, we can predict binary speaker activities \hat{\mathbf{P}} of size C\times T , where T denotes the number of time segments in which the activity is predicted. In realistic scenarios the number of speakers C is unknown, and the speech signal \mathbf{x}_{c} consists of multiple utterances. Our objective is to estimate \mathbf{x}_{c} by separating all utterances belonging to the same speaker into a single output. This process leverages the estimated speaker count \hat{C} and speaker activity \hat{\mathbf{P}} , resulting in \hat{C} separated speech signals, as illustrated in Fig. 1 . Figure 2: A-DCSS architecture. 3 Attractor-based joint diarization, counting, and separation system (A-DCSS) The proposed joint separation system consists of three core processes: feature embedding, attractor generation, and speech separation. The system architecture is illustrated in Fig. 2 . An encoder and a decoder are used to transform the signal between the time domain and the time-feature domain. The embedding module maps the time-feature representations into a latent embedding space through a dual-path transformer-based network. The attractors are generated by learning source-level information from the embeddings. As shown in Fig. 2 , the speaker diarization results and the prediction of the attractor existence probability can be obtained by the attractors, respectively. Then, the generated attractors are used as conditioning input, modulating the mixture features into speaker-specific features through a FiLM operation. Finally, we employ a triple-path separator to achieve speech separation. Encoder and decoder : The encoder applies a one-dimensional convolutional layer with rectified linear unit (ReLU) activation to transform the input \mathbf{y}\in\mathbb{R}^{T^{*}} into a time-feature representation Y\in\mathbb{R}^{T\times F} , F being the feature dimension. The kernel size is L samples and the stride is L/2 samples. The decoder reconstructs each separated signal using a transposed convolutional layer with the same hyperparameters as the encoder. Feature Embedding : The feature embedding module of A-DCSS uses a dual-path design, as shown in Fig. 3 . A linear layer is used to map the encoded features from dimension F to D . Following the Sepformer [ 9 ] , we employ a chunking operation that segments the time dimension T into S chunks of equal length K

Read the complete text