Diff-VS: Efficient Audio-Aware Diffusion U-Net for Vocals Separation
Yun-Ning (Amy) Hung, Richard Vogl, Filip Korzeniowski, Igor Pereira · 2026
Read Diff-VS: Efficient Audio-Aware Diffusion U-Net for Vocals Separation by Yun-Ning (Amy) Hung, Richard Vogl, Filip Korzeniowski, Igor Pereira (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01120.txt.
Introduction
Hung and colleagues introduce Diff-VS, a vocal separation model built on the EDM framework, tackling the weak scores of current generative music separation. It handles complex short-time Fourier transform spectra with an improved U-Net shaped by music-led design choices. The model matches rival baselines on objective metrics and reaches heard quality on par with state-of-the-art systems by proxy listener scores. The team hopes the results spur wider study of such methods for music source separation.
Catalog record
| CREATOR | Yun-Ning (Amy) Hung, Richard Vogl, Filip Korzeniowski, Igor Pereira |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19689 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.01120
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diff-VS: Efficient Audio-Aware Diffusion U-Net for Vocals Separation Yun-Ning (Amy) Hung, Richard Vogl, Filip Korzeniowski, Igor Pereira Abstract While diffusion models are best known for their performance in generative tasks, they have also been successfully applied to many other tasks, including audio source separation. However, current generative approaches to music source separation often underperform on standard objective metrics. In this paper, we address this issue by introducing a novel generative vocal separation model based on the Elucidated Diffusion Model (EDM) framework. Our model processes complex short-time Fourier transform spectrograms and employs an improved U-Net architecture based on music-informed design choices. Our approach matches discriminative baselines on objective metrics and achieves perceptual quality comparable to state‑of‑the‑art systems, as assessed by proxy subjective metrics. We hope these results encourage broader exploration of generative methods for music source separation. † † address: Moises, USA 1 Introduction Recent advances in music source separation using discriminative approaches, where models focus on directly mapping the input mixture to its separated source, have shown promising results. Leveraging band-splitting and RoFormer-based architectures, BS-RoFormer [ 12 ] and Mel-RoFormer [ 29 ] demonstrate strong performance and good scalability on four-stem separation. With band-splitting and a sparse compression network, SCNet [ 27 ] achieves competitive performance with faster inference, fewer parameters, and a single model for all four stems. Moreover, Hung et al. [ 7 ] show that, with appropriate model design and an efficient band-splitting mechanism, compact U-Net architectures can achieve results comparable to much larger counterparts. Generative approaches, on the other hand, have been less explored for music source separation, despite their successes in related tasks such as audio generation [ 6 ] and speech enhancement [ 30 ] . Mariani et al. [ 15 ] and Karchkhadze et al. [ 8 ] unify separation and generation within a single framework. Although they demonstrate basic separation capabilities, the models require 150 sampling steps during inference and comprise over 400M parameters. Furthermore, since they were trained exclusively on synthesized instrumental music, the generalization capabilities to real-world music are not well studied. More recent work [ 19 ] investigates Denoising Diffusion Probabilistic Models (DDPM) within the UNIVERSE [ 23 ] diffusion framework. This approach processes plain waveforms for music source separation, reducing the model size to 99M parameters and the number of required inference steps to roughly 20. To further accelerate inference, Plaja-Roglans et al. [ 18 ] apply diffusion to latent embeddings from EnCodec [ 4 ] . Other hybrid approaches, such as [ 25 ] and [ 2 ] , combine generative models to post-process the output of discriminative models and enhance the separated signals, yielding improvements over the original models. Despite these advances, most approaches still perform significantly lower than discriminative models in objective metrics (SDR). In speech enhancement, diffusion-based methods have not only achieved comparable performance with discriminative baselines [ 22 ] but also shown greater robustness under train–test mismatches [ 13 ] . Given the similarity between speech enhancement and music source separation, these results motivate further investigation to realize the full potential of generative methods for music source separation. A straightforward approach to improving generative-based source separation is to train a state-of-the-art (SOTA) diffusion-based speech enhancement model, such as SGMSE [ 22 , 21 ] , on music source separation data, as proposed by Bereuter et al. [ 2 ] . Their model obtained degradation mean opinion scores (DMOS) comparable to SOTA Mel-RoFormer, but still performed worse in terms of SDR. Based on the success of the SGMSE model in [ 2 ] , we further investigate diffusion-based source separation in the spectrograms domain. Instead of SGMSE, we apply the EDM diffusion framework proposed by Karras et al. [ 9 ] . This choice was made based on the capabilities of the framework to use different model architectures while achieving better performance on image generation with fewer inference sampling steps. Moreover, inspired by Hung et al. [ 7 ] , we re-design the DDPM++ model [ 24 ] used by EDM [ 9 ] by including band-splitting and dual-path RoFormer blocks. With proper input design, we show that a generative-based approach can achieve comparable SDR scores to discriminative models while outperforming most of the SOTA models on subjective evaluation. In summary, the contributions of this work are threefold: • We are the first to apply the EDM framework on the vocals source separation task using complex spectrograms as audio representation. This reduces the number of sampling steps compared to previous works (to fewer than 10 steps). • We improved the widely-used DDPM++ model with music-informed design choices specifically for music source separation. • We demonstrate that the proposed generative-based method is capable of achieving competitive objective and subjective scores compared to SOTA discriminative methods. 2 Background Elucidated Diffusion Models (EDM) are trained via denoising score matching [ 28 ] , aiming to model the score of a noise-level-dependent marginal distribution of the training data corrupted by Gaussian noise. EDM improves on prior diffusion works in three respects. First, it provides a simplified, unified framework whose training and inference are largely model‑agnostic. In other words, the EDM framework is robust to architectural modifications, allowing us to incorporate music‑specific enhancements on top of the DDPM++ model [ 24 ] . We will discuss these enhancements in Section 3.2 . Second, EDM optimizes the sampling process to achieve higher quality with fewer steps. Is uses Heun’s second‑order method [ 1 ] as ODE solver together with a custom inference noise schedule: \sigma_{i<N}=\left({\sigma_{\max}}^{\frac{1}{\rho}}+\frac{i}{N-1}\left({\sigma_{\min}}^{1/\rho}-{\sigma_{\max}}^{1/\rho}\right)\right)^{\rho} (1) {\sigma_{\max}} is set to 80 and {\sigma_{\min}} is set to 0.002. Since truncation errors near {\sigma_{\min}} are more impactful, {\rho} is adjusted between 5 to 10 to concentrate (i.e. shorten) steps close to {\sigma_{\min}} . These choices allow us to reduce the number of sampling steps and improve inference speed for music source separation, addressing a common limitation of prior generative approaches. Third, EDM improves training by preconditioning the network with a \sigma -dependent skip connection: D_{\theta}(\boldsymbol{x};\sigma)=c_{\text{skip}}(\sigma)\,\boldsymbol{x}+c_{\text{out}}(\sigma)\,F_{\theta}\!\left(c_{\text{in}}(\sigma)\,\boldsymbol{x};\,c_{\text{noise}}(\sigma)\right) (2) where F_{\theta} is the trained model, c_{\text{skip}}(\sigma) modulates the skip connection, c_{\text{in}}(\sigma) and c_{\text{out}}(\sigma) scale the input and output magnitudes, and c_{\text{noise}}(\sigma) maps noise level \sigma into a conditioning input for F_{\theta} . The functions for c_{\text{in}} and c_{\text{out}} are chosen so that network inputs and training targets have unit variance, c_{\text{skip}} so that errors in F_{\theta} are amplified as little as possible, and c_{\text{noise}} is chosen empirically. Finally, because training at intermediate noise levels is most effective, EDM biases training toward the relevant range by sampling \sigma from a log-normal distribution. These modifications speed up training and lead improve output sample quality. For a comprehensive discussion, please refer to the original paper [ 9 ] . 3 Proposed Method In this section, we describe our adaptations of the EDM framework for music source separation, including changes to the input representation and to the model architecture. 3.1 Input Representation We first compute the complex Short-Time Fourier Transform (STFT) of the stereo mixture and split the real and imaginary parts into separate channels. This results in an image‑like representation, a 3D tensor C\times F\times T with C=4 , where F and T denote the frequency and time axes. There are two ways to inject the mixture spectrogram