FlowTSE: Target Speaker Extraction with Flow Matching

Aviv, Aviv, Yael, Neta, Gil, Joseph · 2025

Read FlowTSE: Target Speaker Extraction with Flow Matching by Aviv, Aviv, Yael, Neta, Gil, Joseph (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14465.txt.

Introduction

A team of researchers isolates one voice from a mixture with a flow-based generative model in a paper from 2025. The catalog creator field lists first names only, while the head byline gives six full names, so team framing is used here. The system hears an enrollment clip of the target speaker plus the mixed audio. Both are fed as mel-spectrograms. A conditional flow matching model then grows the target's clean speech.

A special vocoder rebuilds phase using the mixture's complex spectrum, where phase matters most. Benchmarks show results matching or beating strong baselines. Earlier generative approaches needed heavy pipelines and pretrained parts. The authors stress simplicity as the contribution. The method targets calls and recordings with overlapping talkers.

Catalog record

CREATORAviv, Aviv, Yael, Neta, Gil, Joseph
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14465

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FlowTSE: Target Speaker Extraction with Flow Matching Aviv, Aviv, Yael, Neta, Gil, Joseph Abstract Target speaker extraction (TSE) aims to isolate a specific speaker’s speech from a mixture using speaker enrollment as a reference. While most existing approaches are discriminative, recent generative methods for TSE achieve strong results. However, generative methods for TSE remain underexplored, with most existing approaches relying on complex pipelines and pretrained components, leading to computational overhead. In this work, we present FlowTSE, a simple yet effective TSE approach based on conditional flow matching. Our model receives an enrollment audio sample and a mixed speech signal, both represented as mel-spectrograms, with the objective of extracting the target speaker’s clean speech. Furthermore, for tasks where phase reconstruction is crucial, we propose a novel vocoder conditioned on the complex STFT of the mixed signal, enabling improved phase estimation. Experimental results on standard TSE benchmarks show that FlowTSE matches or outperforms strong baselines. 1 1 1 Audio samples are available at: https://aiola-lab.github.io/flow-tse Navon Shamsian Segal-Feldman Glazer Hetz Keshet aiOla ResearchIsrael keywords target speaker extraction, flow matching † † email: {aviv, aviv.shamsian}@aiola.com 1 Introduction Target Speaker Extraction (TSE) focuses on extracting the clean speech of a specific speaker from an audio mixture containing interfering speakers and background noise. TSE has gained significant attention for its role in improving speech clarity in communication systems and enhancing hearing aids for individuals with hearing impairments [ 1 , 2 ] . Additionally, TSE enhances Automatic Speech Recognition (ASR) by isolating the target speaker’s speech before transcription, reducing interference from noise and other speakers [ 3 ] . This enables applications like virtual assistants to operate accurately in noisy environments. Traditional approaches to TSE mainly rely on discriminative models, which directly map a noisy speech sample to a clean speech or speech mask, by optimizing signal-level metrics such as signal-to-distortion ratio [ 4 , 5 ] . While these models have shown strong performance, they often produce artifacts and may fail to reconstruct highly corrupted samples [ 6 ] . In addition, they may exhibit limited generalization to unseen noise types or speakers [ 2 , 7 , 8 , 9 ] . Figure 1: FlowTSE architecture. The model is conditioned on the mixed and enrollment mel. Recently, generative models have emerged as a powerful alternative, especially in speech enhancement, offering improved perceptual quality and robustness [ 7 , 8 , 10 ] . Despite these advantages, their potential in TSE is still relatively unexplored. In the context of TSE, generative approaches model the distribution of clean speech given a target speaker’s reference, enabling more natural and high-fidelity speech extraction [ 2 , 6 ] . However, current generative models also have their limitations. Many existing methods rely on complex pipelines involving discriminative TSE models [ 2 ] , or incorporating pretrained models such as Self-Supervised Learning (SSL) models [ 6 , 11 ] or ASRs [ 12 ] , which complicates their training and deployment. Additionally, several approaches operate directly in the complex short-time Fourier transform (STFT) domain, which, while providing fine-grained frequency resolution, can introduce computational overhead, potential artifacts, and slow convergence [ 2 , 13 ] . An alternative is to model speech in the mel-spectrogram domain, simplifying the pipeline and reducing computational complexity. However, this approach comes with its own challenges, particularly phase reconstruction. Since mel-spectrograms discard phase information, vocoder-based inversion may introduce artifacts or degrade speech quality, especially in scenarios where precise phase reconstruction is crucial. Moreover, generative models may still struggle with maintaining speaker identity in challenging conditions [ 12 ] . To address these challenges, we propose a simplified generative approach based on Flow Matching (FM) [ 14 ] operating directly on mel-spectrograms. This representation reduces training complexity while maintaining high perceptual quality. Furthermore, for applications where phase reconstruction is crucial, we propose conditioning the vocoder on the comple STFT of the mixed signal, supporting better phase estimation. We conduct extensive evaluations on both clean two-speaker mixtures and more challenging two-speaker scenarios with background noise. Our results demonstrate that FlowTSE achieves performance matches or outperforms strong baselines while maintaining a simpler and more efficient pipeline. Our main contributions are as follows: (i) We propose FlowTSE, a novel TSE approach based on conditional flow matching, offering a simple yet effective alternative to complex generative pipelines. (ii) We introduce a vocoder augmentation strategy to improve phase reconstruction in scenarios where phase estimation is critical. (iii) We provide extensive evaluations in both clean and noisy TSE conditions, demonstrating the effectiveness and robustness of our approach. 2 Related Work TSE falls under the broader category of Speech Enhancement (SE), which aims to recover clean speech from an input audio corrupted by noise. SE methods are generally categorized as discriminative or generative. Discriminative approaches directly map noisy inputs to clean outputs, operating either in the frequency domain [ 4 ] or in the time domain to mitigate phase estimation distortions [ 5 ] . Generative approaches enhance robustness and perceptual quality, with recent advances incorporating diffusion models for both time-domain [ 10 ] and complex STFT-domain processing [ 7 , 8 ] . Additionally, hybrid models, such as [ 15 ] , integrate discriminative and generative techniques to achieve improved performance. Similar to SE, TSE methods can be categorized into discriminative and generative approaches. Discriminative models either operate in the frequency domain, such as [ 16 ] , or in the time domain, as seen in [ 17 ] . However, these methods often struggle to generalize to unseen noise conditions or speakers [ 7 , 2 , 8 ] , leading to increased interest in generative methods. In order to address these limitations, generative-based TSE models have emerged. [ 13 ] proposed a diffusion model operating on the complex STFT, while [ 2 ] combined diffusion processes with discriminative objectives. [ 12 ] further improved generative models by incorporating speech embeddings and ASR loss, enhancing both perceptual quality and intelligibility. Another approach is to leverages discrete speech tokens derived from large self-supervised language models, as demonstrated by [ 6 ] and [ 11 ] . However, there is still room for improvement, as the state-of-art methods rely on complex pipelines involving discriminative TSE models [ 2 ] or large pretrained models [ 6 , 11 , 12 ] . Table 1: Performance comparison on Libri2Mix (clean and noisy) using the min configuration. We bold the best results and underline the second-best results. Libri2Mix Noisy (mix-both) Libri2Mix Clean (mix-clean) Type PESQ ESTOI OVRL DNSMOS SIM PESQ ESTOI OVRL DNSMOS SIM Mixture - 1.08 0.40 1.63 2.71 0.46 1.15 0.54 2.65 3.41 0.54 Clean - - - 3.26 3.73 - - - 3.26 3.73 - DPCCN D 1.74 0.73 2.93 3.58 0.69 2.22 0.83 3.05 3.73 0.82 NCSN++ D 1.55 0.73 3.15 3.68 0.69 2.24 0.86 3.28 3.86 0.85 DiffSep+SV G 1.32 0.60 2.78 3.63 0.62 1.85 0.79 3.14 3.83 0.83 DDTSE G 1.60 0.71 3.28 3.74 0.71 1.79 0.78 3.30 3.79 0.73 SKIM G 1.29 / 3.27 / / / / / / / FlowTSE G 1.86 0.75 3.30 3.82 0.83 2.58 0.84 3.27 3.79 0.90 3 Method In this section, we present FlowTSE, our flow matching-based approach for TSE. Our method maintains

Read the complete text