DAFMSVC: One-Shot Singing Voice Conversion with Dual Attention Mechanism and Flow Matching

Wei, Binzhu, Dan, Jing, Zhuo, Fan, Zhiyong · 2025

Read DAFMSVC: One-Shot Singing Voice Conversion with Dual Attention Mechanism and Flow Matching by Wei, Binzhu, Dan, Jing, Zhuo, Fan, Zhiyong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05978.txt.

Introduction

Wei Chen and six co-authors present a 2025 method for changing one singing voice into another from a single sample. The task keeps the tune and the words while swapping the tone of the singer. The hard part is copying a new singer's tone without leaking the source tone or spoiling the sound.

Their method swaps the source audio's learned features for the closest matching features from the target sample, which blocks tone leakage. A dual attention block blends the singer traits, the tune, and the word content. A flow matching block then builds clean audio from the blend. In both human and machine tests, the method beat the best prior systems on how close the tone matched and on how natural the result sounded.

Catalog record

CREATORWei, Binzhu, Dan, Jing, Zhuo, Fan, Zhiyong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05978

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DAFMSVC: One-Shot Singing Voice Conversion with Dual Attention Mechanism and Flow Matching Wei, Binzhu, Dan, Jing, Zhuo, Fan, Zhiyong Abstract Singing Voice Conversion (SVC) transfers a source singer’s timbre to a target while keeping melody and lyrics. The key challenge in any-to-any SVC is adapting unseen speaker timbres to source audio without quality degradation. Existing methods either face timbre leakage or fail to achieve satisfactory timbre similarity and quality in the generated audio. To address these challenges, we propose DAFMSVC, where the self-supervised learning (SSL) features from the source audio are replaced with the most similar SSL features from the target audio to prevent timbre leakage. It also incorporates a dual-cross-attention mechanism for the adaptive fusion of speaker embeddings, melody, and linguistic content. Additionally, we introduce a flow matching module for high-quality audio generation from the fused features. Experimental results show that DAFMSVC significantly enhances timbre similarity and naturalness, outperforming state-of-the-art methods in both subjective and objective evaluations. Chen Sha Luo Yang Wang Fan Wu Shenzhen International Graduate School, Tsinghua UniversityChina Huawei Technologies Co., Ltd.China keywords singing voice conversion, cross-attention, flow matching † † email: [email protected], [email protected] 1 1 footnotetext: Corresponding author. 1 Introduction In recent years, the application of Singing Voice Conversion (SVC) in music creation has been rapidly emerging. The goal of any-to-any SVC is to transfer the timbre of a source song to an unseen target singer while preserving the original content and melody. This technology has a wide range of applications, such as becoming an essential tool for artists and disc jockeys in remixing, sampling, and other creative processes. The core idea behind any-to-any SVC is to model, disentangle, and utilize various speech attributes, including content, timbre, and pitch. Previous SVC methods [ 1 , 2 ] typically rely on pre-trained Automatic Speech Recognition (ASR) [ 3 , 4 ] models or SSL based speech models [ 5 ] and Speaker Verification (SV) [ 6 , 7 ] to extract content and timbre information from the source and target speech, respectively. These models then encode the extracted information, which is later decoded to reconstruct the singing audio. Various deep generative models, such as autoregressive models [ 8 , 9 ] , Generative Adversarial Networks (GANs) [ 10 ] , Variational Autoencoders (VAEs) [ 11 ] , and diffusion models [ 12 ] , are used for decoding. Despite significant advancements in speaker verification technologies, relying solely on the extracted speaker embedding vector to encompass all necessary vocal information remains questionable [ 13 ] . Furthermore, effectively separating speaker characteristics from content presents another challenge, often leading to timbre leakage [ 14 ] , where some of the source speaker’s timbre remains in the converted audio. This issue is particularly prominent when using SSL speech models to extract content features [ 15 ] . To address the issue of timbre leakage, NeuCoSVC [ 16 ] uses an SSL based speech model to extract SSL features from the target speaker’s reference audio and constructs a matching pool. The SSL features from the source audio are then replaced with the most similar features from the matching pool to achieve timbre conversion. Since the SSL features used during the conversion phase come directly from the target audio, NeuCoSVC avoids the timbre leakage problem. However, when replacing the most similar self-supervised features, it overlooks some of the timbre information. This is because the timbre information is scattered across the entire target set, and only a few features are selected for replacement, resulting in incomplete timbre information and, consequently, a decrease in conversion similarity. Additionally, the GAN-based waveform reconstruction used in NeuCoSVC faces issues such as instability, mode collapse, and insufficient audio generation quality [ 17 , 18 , 19 ] . Therefore, to further improve timbre similarity and audio generation quality, this study introduces a novel any-to-any SVC method with Dual Attention mechanism and Flow Matching (DAFMSVC). Following NeuCoSVC, DAFMSVC uses a matching pool strategy to prevent timbre leakage. To enhance the timbre information in the SSL features, we introduce speaker embeddings, which help capture the timbre details scattered across the reference audio. As pointed out in [ 20 ] , speaker characteristics include not only global timbre information but also local pronunciation variations. Furthermore, melody, which contains pitch and loudness, is closely tied to content. Therefore, we introduce a dual cross-attention mechanism module to facilitate the adaptive fusion of speaker embeddings, melody, and linguistic content features. Additionally, as demonstrated in [ 21 , 22 ] , flow matching techniques have been shown to provide more stable training and higher sample quality in both image and speech generation. Building on this, we introduce a conditional flow matching module to improve audio quality, which is trained to predict a vector field and efficiently models the probabilistic distribution of the target audio. Figure 1: DAFMSVC framework. Snowflake represents the parameter that remains unchanged when training the framework. This work makes the following contributions: • We propose DAFMSVC, a novel SVC framework that introduces an innovative dual cross-attention mechanism module with adaptive gate control to effectively capture both timbre and melody information and improve timbre similarity. • We introduce a conditional flow matching (CFM) module that predicts probability density paths conditioned on timbre, melody and content, significantly improving sample quality compared to existing state-of-the-art methods. • Experimental results show that our model achieves higher timbre similarity and naturalness in both subjective and objective evaluations. 2 Proposed Method Figure 1 illustrates the overview of our DAFMSVC model. The source audio is first processed by a pre-trained SSL model to extract fixed-dimensional features that capture both linguistic and timbre information. These SSL features are then matched with those from the reference audio to select the phonetically relevant ones. The selected SSL features retain the content information from the source audio while adopting the timbre of the target speaker. These pre-matched SSL features are encoded and passed through a dual cross-attention mechanism module, enabling the joint utilization of content information, melody, and target timbre representations. Finally, the output of the dual cross-attention mechanism module is concatenated with pitch and loudness, and fed into the conditional flow matching module to reconstruct the converted waveform. The details of each module will be discussed in the following sections. Figure 2: Dual cross-attention mechanism module. 2.1 Feature extraction and matching module The module consists of two stages: 1) extracting compact features from the audio, and 2) replacing the source audio’s SSL features with those extracted from the reference utterances. In the first stage, pitch values are extracted by taking the median of three methods (PYIN 1 1 1 https://github.com/librosa/librosa , REAPER 2 2 2 https://github.com/google/REAPER and Parselmouth 3 3 3 https://github.com/YannickJadoul/Parselmouth ). Loudness features are extracted using the A-weighting mechanism of the power spectrum [ 23 ] . These two features are concatenated to form the melody features. A pre-trained SV model is also used to extract speaker embeddings of the reference waveform. Next, a pre-trained WavLM-large encoder [ 5 ] is used to extract SSL features from the audio. Regarding the matching strategy, following NeuCoSVC [ 16 ] , we adopt the KNN method and use the average of the last five layers of WavLM-large to search for the K

Read the complete text