R2-SVC: TOWARDS REAL-WORLD ROBUST AND EXPRESSIVE ZERO-SHOT SINGING VOICE CONVERSION

Junjie Zheng, Gongyu ChenChaofan Ding, Zihao Chen · 2025

Read R2-SVC: TOWARDS REAL-WORLD ROBUST AND EXPRESSIVE ZERO-SHOT SINGING VOICE CONVERSION by Junjie Zheng, Gongyu ChenChaofan Ding, Zihao Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20677.txt.

Introduction

Junjie Zheng, Gongyu Chen, Chaofan Ding, and Zihao Chen address singing voice conversion, where a system reshapes one singer voice into another while keeping the song intact. Real use brings noise from rooms and rough splits of music into voice and backing tracks, which clean lab training does not prepare a model for.

Their R2-SVC system trains with added pitch wobbles and fake split flaws such as echo, so it stays steady on messy inputs. It learns voice tone from both clean studio takes and filtered real world splits, and it models the harmonic and noise parts of the voice apart. Tests under clean and noisy settings place the system at the top of current methods.

Catalog record

CREATORJunjie Zheng, Gongyu ChenChaofan Ding, Zihao Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20677

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

R2-SVC: TOWARDS REAL-WORLD ROBUST AND EXPRESSIVE ZERO-SHOT SINGING VOICE CONVERSION Junjie Zheng, Gongyu ChenChaofan Ding, Zihao Chen Abstract In real-world singing voice conversion (SVC) applications, environmental noise and the demand for expressive output pose significant challenges. Conventional methods, however, are typically designed without accounting for real deployment scenarios, as both training and inference usually rely on clean data. This mismatch hinders practical use, given the inevitable presence of diverse noise sources and artifacts from music separation. To tackle these issues, we propose R2-SVC, a robust and expressive SVC framework. First, we introduce simulation-based robustness enhancement through random fundamental frequency ( F 0 F_{0} ) perturbations and music separation artifact simulations (e.g., reverberation, echo), substantially improving performance under noisy conditions. Second, we enrich speaker representation using domain-specific singing data: alongside clean vocals, we incorporate DNSMOS-filtered separated vocals and public singing corpora, enabling the model to preserve speaker timbre while capturing singing style nuances. Third, we integrate the Neural Source-Filter (NSF) model to explicitly represent harmonic and noise components, enhancing the naturalness and controllability of converted singing. R2-SVC achieves state-of-the-art results on multiple SVC benchmarks under both clean and noisy conditions. Audio samples are available at: https://c9412600.github.io/svc/index.html † † address: AI Lab, Giant Network 1 Introduction Singing voice conversion (SVC) aims to transform a singer’s voice into that of another without changing the lyrics or overall musical expression. It has broad applications in dubbing, voice chat, and music production [ 1 , 2 , 3 , 4 ] . In real-world scenarios, however, robustness is crucial, as practical systems must handle background noise, reverberation, echoes, and artifacts introduced by singing voice separation. Ensuring such robustness is key to producing high-quality vocals while preserving both semantic content and expressive characteristics. Prior studies explored adversarial learning [ 5 , 6 ] and data augmentation [ 7 , 8 ] , but these approaches often generalize poorly to unseen conditions. ASR-based content modeling, from phonetic posteriorgrams (PPG) [ 9 ] to bottleneck features (BNF) [ 10 ] , improves noise robustness but struggles with F_{0} jitter, reverberation, and echoes, leading to reduced naturalness. To address this trade-off, we propose R2-SVC , a Robust and Real-World zero-shot SVC system. It integrates three modules—Simulation-based Robustness Enhancement (SRE), Singing-Enhanced Timbre and Style Extractor (SETSE), and Neural Source-Filter (NSF)—to preserve both semantic information and expressiveness in noisy conditions. Our contributions are: (1) Simulating realistic conditions during training via random F_{0} perturbation and reverberation/echo artifacts to enhance robustness. (2) Enriching speaker representation with clean vocals, DNSMOS-filtered [ 11 ] separated vocals, and public singing corpora, enabling timbre preservation and capturing singer-specific styles. (3) Incorporating NSF for explicit source–filter decomposition, improving naturalness and controllability in challenging scenarios. (4) Achieving state-of-the-art results on zero-shot SVC, with evaluations on both clean and noisy real-world test sets confirming the effectiveness and applicability of our approach. Figure 1: Architecture overview of R2-SVC. 2 Approach 2.1 Overview Our approach builds upon a pre-trained Seed-VC checkpoint and is further adapted with open-source singing vocal data to enhance robustness and generalization. Specifically, we introduce three improvements: (i) simulation-based robustness enhancement , where random perturbations are applied to the F_{0} and vocal inputs are augmented with reverberation, harmony and echo artifacts to mimic music separation effects; (ii) singing-informed timbre and style extractor , where clean singing vocals, DNSMOS-filtered separated vocals, and public singing corpora are incorporated, enabling the model to not only preserve timbre but also capture speaker-specific singing styles; and (iii) Neural Source-Filter (NSF) integration , which provides explicit harmonic and noise representations, improving naturalness and controllability, particularly in singing scenarios. Formally, the conversion process can be expressed as: \displaystyle\hat{y} \displaystyle=\mathcal{F}_{\theta}\!\left(x^{\mathrm{aug}},\,F_{0}^{\mathrm{pert}},\,\mathbf{s},\,\mathbf{h}_{\mathrm{nsf}}\right) (1) \displaystyle(x^{\mathrm{aug}},\,F_{0}^{\mathrm{pert}}) \displaystyle=\mathcal{R}(x,F_{0}) (2) \displaystyle\mathbf{s} \displaystyle=\mathcal{E}(x;\,\mathcal{D}_{\text{sing}}) (3) \displaystyle\mathbf{h}_{\mathrm{nsf}} \displaystyle=\mathcal{N}(F_{0},t). (4) Here, x is the source waveform, x^{\mathrm{aug}} and F_{0}^{\mathrm{pert}} are produced by the robustness module \mathcal{R}(\cdot) via wet-sound simulation and random F_{0} perturbations; \mathbf{s} is a singer-specific timbre/style embedding from the extractor \mathcal{E}(\cdot) fine-tuned on singing data \mathcal{D}_{\text{sing}} (clean and DNSMOS-filtered separated vocals, plus public corpora); and \mathbf{h}_{\mathrm{nsf}} denotes NSF features. \hat{y} is the converted singing waveform generated by the conversion model \mathcal{F}_{\theta} , which is initialized from the Seed-VC checkpoint and fine-tuned with the augmented singing data. 2.2 Simulation-based Robustness Enhancement In real-world SVC applications, challenges such as inaccurate F_{0} extraction and residual reverberation or echoes from accompaniment separation are common. To enhance robustness, we propose two strategies: random F_{0} perturbation and wet sound simulation. The F_{0} corresponds to the approximate frequency of the quasi-periodic structure in voiced speech, arising from vocal fold oscillations. While incorporating F_{0} features can improve the pitch accuracy and aesthetic quality of converted audio, extracted F_{0} contours are often noisy in practice, degrading perceptual quality. A robust model should therefore tolerate such noise and perform implicit corrections to enhance expressiveness. As shown in Fig. 1 , we introduce a random F_{0} perturbation strategy to guide the model’s attention to F_{0} features. During training, three perturbation types are applied randomly: Jitter, simulating vocal vibrato; Glide, mimicking natural pitch slides; and Jump, emulating abrupt F_{0} transitions due to prediction errors. This approach reduces the model’s dependency on F_{0} and encourages reliance on BNF for reconstruction, thereby improving robustness. Vocals separated from accompaniments often contain residual harmonies, reverberation, or echoes. Existing dry audio extraction methods may fail to remove these artifacts without quality loss. To improve model robustness, we simulate such effects using a configurable chain of harmony, echo, and reverb, each triggered with specific probabilities, to augment training data. This teaches the model to produce dry audio from wet inputs. During training, wet sound simulation is applied to all modules except the F_{0} extractor. Combined with F_{0} perturbations, this strategy mimics challenging acoustic conditions and strengthens noise tolerance. Algorithm 1 Training Step with Simulation-based Robustness Enhancement 1: Clean waveform x ; origin pitch contour F_{0} ; probs (0<p_{\text{jit}}+p_{\text{gld}}+p_{\text{jmp}},p_{h},p_{e},p_{r}<1) 2: Updated model 3: F_{0}^{\mathrm{pert}}\leftarrow Perturb F_{0} by jitter/glide/jump/none chosen by probs p_{\text{jit}},p_{\text{gld}},p_{\text{jmp}} 4: x^{\mathrm{aug}}\leftarrow Apply harmony , echo , reverb independently on x by probs p_{h},p_{e},p_{r} 5: Feed (F_{0}^{\mathrm{pert}},x^{\mathrm{aug}}) into model 6: Compute loss against clean x 7: Update model 2.3 Singing-Enhanced Timbre and Style Extractor To better capture timbre and stylistic nuances in SVC, we extend the CAM++ [ 12 ] framework with a transfer learning strategy using domain-specific singing data. In addition to the original CAM++ datasets, we incorporate public singing voice corpora and high-quality internal vocal data, where tracks are separated and screened with DNSMOS. This augmentation allows the extractor to learn not only the timbre information consistent across speaking and singing, but also singer-specific vocal styles such as vibrato and articulation patterns. By enriching the training data in this way, the extractor adapts more effectively to the singing domain, improving robustness under noisy and reverberant conditions while supporting natural and expressive singing voice conversion. 2.4 Neural Source-Filter-Based Acoustic Enhancement The harmonic-plus-noise Neural Source-Filter (hn-NSF) model [ 13 ] generates waveforms using a source-filter architecture conditioned on acoustic features. Its source module produces a harmonic and noise excitation from F_{0} , which is then spectrally shaped by a filter module. The harmonic component of the excitation, e(t) , is generated as follows: e(t)=\tanh\left(\mathbf{h}(t)\cdot\mathbf{W}_{\text{merge}}^{T}+b_{\text{merge}}\right) (5) where a vector of sinusoids \mathbf{h}(t) , derived from the input frequency f_{0}(t) , undergoes a fixed

Read the complete text