MeanFlowSE: One-Step Generative Speech Enhancement via MeanFlow
2025
Read MeanFlowSE: One-Step Generative Speech Enhancement via MeanFlow (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23299.txt.
Introduction
Cleaning noisy speech in one generative step, instead of many, is the draw of MeanFlowSE. Yike Zhu and 8 co-authors built the framework on MeanFlow, which predicts an average velocity field. One step then refines the speech latent. Self-supervised speech codes guide the model, not autoencoder latents. That choice speeds inference and anchors training.
On the Interspeech 2020 DNS blind test set, quality hit top-tier levels. Intelligibility stayed competitive with multi-step generative rivals. Real-time factor and model size both dropped a lot. Telecom and speech recognition uses need that speed. Code will be released on publication, the team say.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23299
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MeanFlowSE: One-Step Generative Speech Enhancement via MeanFlow Abstract Speech enhancement (SE) recovers clean speech from noisy signals and is vital for applications such as telecommunications and automatic speech recognition (ASR). While generative approaches achieve strong perceptual quality, they often rely on multi-step sampling (diffusion/flow-matching) or large language models, limiting real-time deployment. To mitigate these constraints, we present MeanFlowSE , a one-step generative SE framework. It adopts MeanFlow to predict an average-velocity field for one-step latent refinement and conditions the model on self-supervised learning (SSL) representations rather than VAE latents. This design accelerates inference and provides robust acoustic–semantic guidance during training. In the Interspeech 2020 DNS Challenge blind test set and simulated test set, MeanFlowSE attains state-of-the-art (SOTA) level perceptual quality and competitive intelligibility while significantly lowering both real-time factor (RTF) and model size compared with recent generative competitors, making it suitable for practical use. The code will be released upon publication at https://github.com/Hello3orld/MeanFlowSE . † † address: 1 Audio, Speech and Language Processing Group (ASLP@NPU), School of Software, Northwestern Polytechnical University, Xi’an, China 2 The Hong Kong University of Science and Technology, Hong Kong, China 3 Huawei Technologies, China [email protected], [email protected], [email protected] $\dagger$ $\dagger$ footnotetext: Equal contribution. $*$ $*$ footnotetext: Corresponding author. 1 Introduction Figure 1: Overview of the proposed MeanFlowSE architecture. The left side depicts the training pipeline, while the right side illustrates the one-step inference procedure. Speech enhancement (SE) aims to remove interference from noisy signals and is essential for applications such as telecommunications, hearing aids, and automatic speech recognition (ASR). In recent years, neural network (NN) based methods have greatly improved the performance of speech enhancement. NN-based SE methods fall broadly into two paradigms: discriminative and generative. Discriminative methods aim to directly estimate clean speech or a corresponding mask from noisy inputs. Although effective in matched environments, such approaches often exhibit limited generalizability to unseen acoustic conditions and are prone to introduce artifacts or distortions, especially in low signal-to-noise ratio (SNR) scenarios [ 1 , 2 , 3 ] . Generative methods, in contrast, aim to model the distribution of clean speech and reconstruct it through probabilistic frameworks, such as diffusion models [ 4 , 5 , 6 ] , language models (LM) [ 7 , 8 , 9 ] and masked transformer [ 10 , 11 ] based approaches. Recently, generative modeling has achieved remarkable success in SE, producing high-quality speech reconstruction and demonstrating strong robustness under low SNR conditions. Nevertheless, the substantial computational resources required by such models, particularly the multiple sampling steps in diffusion/flow-matching-based methods [ 12 , 13 ] greatly hinder their deployment on low resource devices. Moreover, current diffusion/flow-matching-based SE models typically condition only on noisy mel-spectrograms or latents of the noisy waveform from variational autoencoders (VAE) [ 14 ] , which provide irregular and noisy representations, thereby leading to suboptimal naturalness and intelligibility in the generated outputs. To mitigate the inefficiency of multi-step sampling, recent studies have explored one-step generative frameworks [ 12 , 15 , 16 ] . Among them, MeanFlow [ 17 ] introduces a principled formulation that leverages the average velocity, defined as the displacement over a time interval, instead of the instantaneous velocity in standard flow-matching. This reformulation links average and instantaneous velocities, offering a stable and efficient training target for one-step generation. Though still trailing multi-step models, MeanFlow demonstrate superior performance than previous one-step generation frameworks. Meanwhile, unlike fully generative tasks, SE benefits from the presence of a reference signal, making the problem more tractable and well-suited for one-step generative modeling. Motivated by these insights, we propose MeanFlowSE , an efficient one-step generative SE framework. MeanFlowSE integrates one-step generative modeling of MeanFlow with conditioning from SSL representations, which provide fine-grained acoustic cues and rich semantic information to guide generation. Experiments show that MeanFlowSE not only achieves SOTA-level performance on public and simulated test sets but also substantially reduces computational demands, marking a significant step toward practical deployment of generative SE. 2 Proposed Method MeanFlowSE aims to recover a clean signal x\in\mathbb{R}^{T} from a noisy signal y\in\mathbb{R}^{T} by modeling their transformation in the latent space. As illustrated in Fig. 1 , the framework contains three major components: (1) an SSL model to extract latent acoustic and semantic representations, (2) a latent Diffusion Transformer (DiT) [ 18 ] -based MeanFlow module that predicts the average velocity field, and (3) a VAE decoder for waveform reconstruction. This modular design allows us to integrate powerful pre-trained models while keeping inference simple and efficient. 2.1 MeanFlow for Speech Enhancement Conventional flow-matching learns an instantaneous velocity field v(z_{t},t) that governs the evolution of latent states through the ordinary differential equation (ODE): \frac{dz}{dt}=v(z_{t},t),\hskip 10.00002ptz_{0}=y,\,z_{1}=x, (1) where z_{t} denotes the latent representation at time step t . During inference, solving this ODE requires multiple integration steps, which hinders real-time deployment. To address this, MeanFlow replaces the instantaneous velocity field with an average velocity field: u(z_{t},r,t)=\frac{1}{t-r}\int_{r}^{t}v(z_{\tau},\tau)\,d\tau, (2) which summarizes the overall trend of transformation between two time steps (r,t) . This formulation enables direct recovery of the clean latents in one-step: z_{0}=z_{1}-u(z_{1},0,1), (3) where in this paper z_{1} denotes the noisy latents and z_{0} denotes the predicted clean latents. By learning u instead of v , the model bypasses iterative ODE solvers [ 19 ] and achieves efficient one-step generative enhancement. 2.2 Network Architecture As illustrated in Fig. 1 , MeanFlowSE consists of three key modules: an SSL encoder, a VAE encoder–decoder, and a DiT-based MeanFlow backbone. Together, they provide semantic conditioning, establish a structured latent domain, and enable efficient one-step generative enhancement. 2.2.1 SSL Encoder MeanFlowSE leverages a pre-trained SSL model to extract high-level acoustic and semantic latents z_{y} from the noisy input y . These latents are used as conditioning signals for the generative backbone. Compared to noisy latents from a VAE encoder, SSL latents capture long-range dependencies and phonetic content [ 20 ] , which are crucial for preserving intelligibility in low-SNR conditions. During inference, the SSL encoder acts as the sole feature extractor from noisy speech. 2.2.2 VAE Encoder–Decoder The VAE encoder–decoder establishes the latent space in which enhancement is performed. During training, the encoder maps the clean waveform x into its latent representation z_{x}=\text{Enc}(x) , which serves as the supervision target. At inference, the decoder transforms the predicted clean latent z_{0} from the generative backbone into the waveform domain, producing the enhanced waveform \hat{x}=\text{Dec}(z_{0}) . 2.2.3 DiT-based MeanFlow Backbone The DiT backbone serves as the generative core of MeanFlowSE. It takes interpolated latents z_{t} (constructed by combining z_{x} with Gaussian noise \epsilon ) concatenated with noisy latents z_{y} , further enriched with positional embedding PE . Meanwhile, time step embeddings \text{TE}(r) and \text{TE}(t) are summed and injected into the DiT blocks through adaptive layer normalization (AdaLN), explicitly controlling the MeanFlow dynamics. The network is trained to predict the target average velocity field u , derived analytically from z_{x} and \epsilon . At inference, with (r=0,t=1) , the model directly refines Gaussian noise \epsilon into enhanced clean latents z_{0} in one-step using the predicted average velocity field \hat{u} : z_{0}=\epsilon-\hat{u}. (4) 2.3 Training Objective To optimize the MeanFlow module, we minimize the gap between predicted and target average velocities. Instead of a plain \ell_{2} loss, we adopt an adaptive \ell_{2} loss