ZeroSep: Separate Anything in Audio with Zero Training
Chao Huang, Yuesheng Ma, Junxuan Huang, Susan Liang, Yunlong Tang, Jing Bi, Wenqiang Liu, Nima Mesgarani, Chenliang Xu · 2025
Read ZeroSep: Separate Anything in Audio with Zero Training by Chao Huang, Yuesheng Ma, Junxuan Huang, Susan Liang, Yunlong Tang, Jing Bi, Wenqiang Liu, Nima Mesgarani, Chenliang Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23625.txt.
Introduction
Huang and colleagues show in 2025 that a pretrained audio diffusion model can separate sound sources with no task-specific training at all. Their method, ZeroSep, inverts a mixed recording into the model's latent space. Text prompts then steer the denoising process to rebuild one source at a time. Because the model learned rich text-audio links, it handles open-set sounds that supervised separators miss.
Across several benchmarks, ZeroSep matched or beat supervised systems. It works with different pretrained diffusion backbones without fine-tuning. The authors frame it as proof that generative foundation models can be repurposed for classic signal tasks.
Catalog record
| CREATOR | Chao Huang, Yuesheng Ma, Junxuan Huang, Susan Liang, Yunlong Tang, Jing Bi, Wenqiang Liu, Nima Mesgarani, Chenliang Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23625
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ZeroSep: Separate Anything in Audio with Zero Training Chao Huang, Yuesheng Ma, Junxuan Huang, Susan Liang, Yunlong Tang, Jing Bi, Wenqiang Liu, Nima Mesgarani, Chenliang Xu Abstract Audio source separation is fundamental for machines to understand complex acoustic environments and underpins numerous audio applications. Current supervised deep learning approaches, while powerful, are limited by the need for extensive, task-specific labeled data and struggle to generalize to the immense variability and open-set nature of real-world acoustic scenes. Inspired by the success of generative foundation models, we investigate whether pre-trained text-guided audio diffusion models can overcome these limitations. We make a surprising discovery: zero-shot source separation can be achieved purely through a pre-trained text-guided audio diffusion model under the right configuration. Our method, named ZeroSep , works by inverting the mixed audio into the diffusion model’s latent space and then using text conditioning to guide the denoising process to recover individual sources. Without any task-specific training or fine-tuning, ZeroSep repurposes the generative diffusion model for a discriminative separation task and inherently supports open-set scenarios through its rich textual priors. ZeroSep is compatible with a variety of pre-trained text-guided audio diffusion backbones and delivers strong separation performance on multiple separation benchmarks, surpassing even supervised methods. Our project page is here: https://wikichao.github.io/ZeroSep/ . 1 Introduction At the heart of acoustic scene perception lies the fundamental task of source separation, which aims to isolate individual sound sources from a complex audio mixture. Accurate source separation is crucial for a wide range of applications, including media production, surveillance systems, automatic speech recognition in noisy environments, and analysis of complex soundscapes. The dominant approach to audio separation in recent years has relied heavily on supervised learning: deep neural networks are trained on large datasets of paired mixtures and clean sources ( Luo and Mesgarani 2019 ; Subakan et al. 2021 ) . While these methods have achieved impressive performance on specific, well-represented source types and datasets, they often fall short when faced with the open-set variability of real-world acoustic scenes. Consequently, training a foundational separation model becomes exceptionally challenging due to the need for vast amounts of labeled data, the difficulty of defining training objectives and mixing strategies, and the design of effective conditioning mechanisms. Recent efforts, such as LASS-net Liu et al. 2022a , AudioSep Liu et al. 2023a , and FlowSep Yin et al. 2024 , have explored leveraging natural language queries for more flexible separation. Despite these advances, they still contend with the same core challenges: vast data requirements, complex task-specific training regimes, and limited generalization to unseen acoustic scenes. Inspired by the transformative success of large language models in unifying diverse NLP tasks under a generative framework ( Brown et al. 2020 ) , we pose a central question: Can a generative foundation model similarly emerge for audio tasks? In this work, we explore this question by investigating the capabilities of pre-trained text-guided audio diffusion models. We discover that a text-guided audio diffusion model can, out of the box , separate a mixture into its sources – no training or fine-tuning , relying solely on latent inversion and conditioned denoising: (i) Given a mixed audio signal, we can find a corresponding point in the diffusion model’s latent space through an inversion process. This latent representation captures the composite information from all the sound sources present in the mixture. (ii) Subsequently, by guiding the generative denoising process from this latent state using text prompts corresponding to individual sources in the mixture, the model can be steered to reconstruct each source in isolation. Surprisingly, even though this is a generative process, the separated sources are highly faithful to the original sources, especially with classifier-free guidance \leq 1 , which prevents hallucination. This effectively repurposes the generative model for a discriminative task, offering a fundamentally different approach to separation. Based on the above observations, we introduce ZeroSep , a zero-training framework for audio source separation that repurposes pretrained text-guided diffusion models. By casting separation as a two-step generative inference, latent inversion followed by text-conditioned denoising, ZeroSep offers three key advantages: Open-set Separation: As the core of ZeroSep is a pre-trained text-guided audio diffusion model, which has learned to generate realistic audios from diverse, open-domain descriptions and mixing styles, ZeroSep naturally handles open-set queries and is able to separate from diverse mixtures. Model-agnostic Versatility: The inversion plus denoising pipeline is generic to diffusion architectures, allowing ZeroSep to leverage different pre-trained audio diffusion backbones. Interestingly, we observe a trend that the better the audio diffusion model can generate, the better it can separate, which could suggest continuous improvement whenever there is a more advanced audio generation model available. Training-Free Efficacy: Without any fine-tuning or task-specific data, ZeroSep matches or exceeds the performance of existing training-based generative separators, overturning the assumption that high-quality separation requires dedicated training. In summary, our contributions to the community includes 1. We introduce ZeroSep , the first training-free audio source separation framework that repurposes pre-trained text-guided audio diffusion models, representing a fundamental shift away from supervised separation paradigms. 2. We demonstrate that pure generative inference—latent inversion followed by text-conditioned denoising—yields state-of-the-art separation performance, outperforming existing training-based generative methods. 3. We establish ZeroSep ’s versatility and open-set capability: it seamlessly handles diverse mixtures and textual queries and can be applied to many pre-trained audio diffusion backbones, improving separation quality as the underlying model’s generative fidelity increases. 2 Related Works Audio Diffusion Models. Diffusion probabilistic models have rapidly emerged as a leading paradigm for generating high-quality and diverse audio content. Early works like DiffWave ( Kong et al. 2021 ) and WaveGrad ( Chen et al. 2021a ) demonstrated the potential of applying denoising diffusion to synthesize raw audio waveforms, achieving unconditional audio generation. Building on this foundation, diffusion models were successfully extended to conditional audio generation tasks. In text-to-speech (TTS), models such as Diff-TTS ( Jeong et al. 2021 ) and Grad-TTS ( Popov et al. 2021 ) showed that diffusion processes could generate high-fidelity mel-spectrograms conditioned on text input. Researchers also focused on improving the efficiency and controllability of diffusion sampling; for instance, Guided-TTS Kim et al. 2022 introduced classifier guidance for TTS, and PriorGrad ( Lee et al. 2022 ) addressed sampling speed in vocoders through data-dependent priors. Diffusion models have also been applied to other audio synthesis tasks, including singing voice synthesis with DiffSinger ( Liu et al. 2022b ) and waveform super-resolution with NU-Wave ( Lee and Han 2021 ) . More recently, the focus has shifted towards latent-space diffusion models and text-conditioned generation of general audio. AudioLDM ( Liu et al. 2023b ) pioneered combining diffusion with CLAP embeddings to enable text-conditioned generation of diverse sounds and music. AudioLDM2 ( Liu et al. 2024 ) and Tango ( Ghosal et al. 2023 ) further advanced in this direction, providing enhanced control and quality. These text-conditioned latent diffusion models, capable of generating complex audio scenes from natural language, form the technological foundation for our training-free separation method ZeroSep . Audio Separation. The problem of source separation has long been tackled by both classic signal-processing techniques and, more recently, deep learning. Traditional methods such as NMF-MFCC ( Stöter et al. 2021 ) decompose mixtures under assumptions about timbral or spectral structure. While