Multi-Utterance Speech Separation and Association Trained on Short Segments
2025
Read Multi-Utterance Speech Separation and Association Trained on Short Segments (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02562.txt.
Introduction
Wang, Politis, Drossos, and Virtanen present a network that separates overlapping speech across long recordings, in a paper from twenty twenty-five. Speech separation models train on short clips due to computing limits. Real recordings run far longer, with speakers pausing and resuming. Models must keep track of who is who across those gaps.
Their network has two parts, one for frequency patterns and one for time patterns. It trains on ten-second clips yet handles recordings up to two minutes long. It keeps speaker identity across gaps longer than any seen in training. The model is light, with under a million parameters. It runs on full audio with no cutting, which avoids seam distortions.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-02562
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-Utterance Speech Separation and Association Trained on Short Segments Abstract Current deep neural network (DNN) based speech separation faces a fundamental challenge — while the models need to be trained on short segments due to computational constraints, real-world applications typically require processing significantly longer recordings with multiple utterances per speaker than seen during training. In this paper, we investigate how existing approaches perform in this challenging scenario and propose a frequency-temporal recurrent neural network (FTRNN) that effectively bridges this gap. Our FTRNN employs a full-band module to model frequency dependencies within each time frame and a sub-band module that models temporal patterns in each frequency band. Despite being trained on short fixed-length segments of 10 s, our model demonstrates robust separation when processing signals significantly longer than training segments (21-121 s) and preserves speaker association across utterance gaps exceeding those seen during training. Unlike the conventional segment-separation-stitch paradigm, our lightweight approach (0.9 M parameters) performs inference on long audio without segmentation, eliminating segment boundary distortions while simplifying deployment. Experimental results demonstrate the generalization ability of FTRNN for multi-utterance speech separation and speaker association. \NoHyper 1 Introduction Speech separation aims to isolate individual speech signals from mixtures containing multiple speakers. Current deep neural network (DNN) based separation models need to be trained on short segments [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ] . However, real-world applications reveal significant performance degradation when these models process longer recordings [ 12 , 13 , 14 ] . This generalization issue remains underexplored in existing literature. Even the methods that perform excellently on short recordings struggle when applied to long speech separation — separating mixture signals consisting of multiple utterances per speaker into individual streams, ensuring all utterances from the same speaker are consistently associated to the same output stream. Current approaches typically employ a segment-separation-stitch paradigm to address long speech separation, as shown in Fig. 1 (b), where long recordings are split into short segments, processed independently, and then stitched. This processing paradigm does not leverage contextual information across segments. When implementing this paradigm, the challenge lies in determining the correct permutation of separated segments during stitching, particularly when utterances from the same speaker are separated by long intervals. Continuous speech separation (CSS) [ 13 , 14 , 15 , 16 , 17 , 18 ] follows this paradigm by stitching separated segments into output streams, with each stream potentially containing utterances from different speakers rather than isolating individual speakers, as shown in Fig. 1 (b). CSS primarily aims to reduce the overlap rate in output streams. To achieve CSS-style outputs, the approaches require very short segments, ensuring that each segment contains no more than 2 - 3 speakers [ 14 ] . This constraint leads to frequent stitching operations, increasing the risk of introducing distortions at segment boundaries. Fig. 1 : Visualization of long speech separation: (a) training phase on short segments, (b) segment–separate–stitch inference, (c) our proposed recurrent neural network inference on unsegmented mixture. An alternative solution leverages speaker-wise representation in separating long recordings, as shown in Fig. 1 (b). These methods incorporate speaker modeling techniques such as speaker clustering [ 12 , 10 , 11 ] , speaker identification [ 19 , 20 , 21 ] , or attractor-based mechanisms [ 22 , 23 , 24 , 25 , 26 ] to determine the optimal assignment of separated segments. While effective, these methods often require speaker modeling components or auxiliary speaker identification modules, increasing implementation complexity. In this paper, we investigate how speech separation models trained on short fixed-length segments perform when applied to long recordings with multiple utterances per speaker. Our main contributions are twofold. First, we evaluate the generalization performance of existing speech separation methods when processing recordings significantly longer than their training segments, an aspect previously underexplored in literature despite its importance for real-world applications. Our analysis reveals that current methods show limited performance in long speech separation, particularly when utterances from the same speaker are separated by extended silent intervals. Second, we propose a frequency-temporal recurrent neural network (FTRNN) that effectively bridges the gap between fixed-length training and flexible-length inference. As illustrated in Fig. 1 (a) and Fig. 1 (c), our proposed FTRNN offers two key advantages: (1) though trained on fixed-length segments, it generalizes to flexible-length inputs during inference without performance degradation; (2) while trained with short utterance intervals, it maintains speaker association across substantially longer intervals than seen during training. Experimental results on our generated multi-utterance dataset demonstrate that the proposed lightweight FTRNN ( 0.9 M parameters) significantly outperforms existing methods across various utterance gaps and numbers. 2 Signal Model A single-channel mixture signal can be formulated in the time domain as y(t)=\sum_{c=1}^{C}x_{c}(t)+n(t). (1) Here, the observed signal y(t) consists of speech signals x_{c}(t) from C speakers and noise n(t) . The x_{c}(t) represents the time-domain signal from the c -th speaker, which consists of multiple utterances and potential pauses between them. Our objective is to perform speaker-independent separation to estimate x_{c}(t) , separating all utterances belonging to the same speaker into a single output, where the output indices of speakers can be arbitrary, as illustrated in Fig. 1 (c). In this study, we focus on the two-speaker case where C=2 . 3 Proposed method The proposed FTRNN is trained on fixed-length short segments but generalizes to flexible-length long audio during inference without requiring any segmentation operations, as illustrated in Fig. 1 (a) and Fig. 1 (c). During training, we split long audio into 10 -second segments as input signals. The FTRNN processes these segments independently, separating each into two output channels, with each channel containing all utterances from a single speaker. We employ permutation invariant training (PIT) [ 7 ] with the scale-invariant signal-to-distortion ratio (SI-SDR) [ 27 ] loss function, where the PIT is applied to the separated segment pairs independently. Notably, there is no need to enforce consistent speaker ordering across segments during training, meaning the same speaker might appear in different output channels for different segments. The architecture of FTRNN is shown in Fig. 2 . The input mixture is transformed into the time-frequency domain via short-time Fourier transform (STFT), with real and imaginary components stacked to form a tensor of shape 2\times T\times F . Here, T is time frames and F is frequency bins. A 2D convolutional layer with a 3\times 3 kernel transforms the stacked tensor into a latent representation of shape T\times F\times D , where D is the feature dimension. This representation then passes through our full-band and sub-band modules. Along-Frequency Full-band Module: The along-frequency full-band module is shown in Fig. 2 (b). After layer normalization, a bidirectional long short-term memory (BLSTM) network operates along the frequency dimension for each time frame independently. The BLSTM captures both forward and backward dependencies across frequencies, where D_{\text{BLSTM}} is the hidden dimension. The BLSTM output is then projected back to the feature dimension D through a feed-forward network (FFN). Along-Temporal Sub-band Module: The along-temporal sub-band module in Fig. 2 (c) focuses on temporal modeling. After layer normalization, a BLSTM network operates along the temporal dimension for each frequency bin independently, where D^{\prime}_{\text{BLSTM}} is