Everyone-Can-Sing: Zero-Shot Singing Voice Synthesis and Conversion with Speech Reference

Shuqi Dai, Yunyun Wang, Roger B. Dannenberg, Zeyu Jin · 2025

Read Everyone-Can-Sing: Zero-Shot Singing Voice Synthesis and Conversion with Speech Reference by Shuqi Dai, Yunyun Wang, Roger B. Dannenberg, Zeyu Jin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13870.txt.

Introduction

Shuqi Dai and co-authors present a 2025 system that can make a singing voice from a short speech sample. Cloning a singer normally needs singing recordings, which are scarce. Their framework, Everyone-Can-Sing, handles both making a song from lyrics and a score, and converting one voice's song into another's, with the target voice taken from plain speech.

Training mixes speech and song data, and a diffusion generator builds the sound. A style selector steers technique, while the score sets the tune and the lyrics set the words. In tests, the system beat prior methods on how close the voice sounds to the target and on musical quality. The authors see lessons for other low-data music tasks.

Catalog record

CREATORShuqi Dai, Yunyun Wang, Roger B. Dannenberg, Zeyu Jin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-13870

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Everyone-Can-Sing: Zero-Shot Singing Voice Synthesis and Conversion with Speech Reference Shuqi Dai, Yunyun Wang, Roger B. Dannenberg, Zeyu Jin Abstract We propose a unified framework for Singing Voice Synthesis (SVS) and Conversion (SVC), addressing the limitations of existing approaches in cross-domain SVS/SVC, poor output musicality, and scarcity of singing data. Our framework enables control over multiple aspects, including language content based on lyrics, performance attributes based on a musical score, singing style and vocal techniques based on a selector, and voice identity based on a speech sample. The proposed zero-shot learning paradigm consists of one SVS model and two SVC models, utilizing pre-trained content embeddings and a diffusion-based generator. The proposed framework is also trained on mixed datasets comprising both singing and speech audio, allowing singing voice cloning based on speech reference. Experiments show substantial improvements in timbre similarity and musicality over state-of-the-art baselines, providing insights into other low-data music tasks such as instrumental style transfer. Examples can be found at: everyone-can-sing.github.io . I Introduction Singing voice synthesis (SVS), which generates singing voice signals from music scores, is gaining increasing importance in generative AI and benefiting various applications in music production and entertainment. Recent advances in deep-learning-based audio synthesis, such as acoustic models [ 1 ] , neural vocoders [ 2 , 3 ] , and tokenizer-based codec models [ 4 , 5 ] , have greatly improved models’ ability to reproduce singing voices from training data [ 6 , 7 , 8 ] . However, despite these advances, current SVS models struggle with unseen voices in zero-shot settings, especially when the voice reference is very brief (just a few seconds). This challenge is exacerbated by the dearth of relevant data for learning musicality, expressiveness, and other intricacies that define singing. The vast disparity between available speech data and its limitations as a proxy for singing further compounds this issue. Consequently, zero-shot SVS performance remains subpar, especially in musicality, acoustic quality, and voice similarity. While recent progress has been made in zero-shot speech synthesis through voice-content disentanglement [ 9 , 10 , 11 , 12 ] , the field still awaits a comprehensive solution to overcome these limitations. Zero-Shot Singing Voice Conversion (SVC) faces similar challenges. SVC alters a singer’s voice while preserving the song’s content. Unlike SVS, it converts from an existing singing sample rather than using a symbolic (score) as input. In zero-shot settings, current approaches rely on long voice sample references [ 13 ] and are commonly less effective with cross-domain speech targets [ 14 , 15 ] . While zero-shot speech conversion models can function with shorter voice samples [ 10 , 11 ] , they fail to address and capture the essence of singing, particularly expressiveness and musicality. Moreover, speech and singing timbre of the same person can differ significantly; the input singing pitch range may vary widely from the short target speech; singing techniques such as vibrato, head voice, and high notes are often ignored by speech conversion models. This paper addresses the challenges of zero-shot cross-domain SVS and SVC, using only a 5-second speech audio as a reference. To ground our definition of voice similarity, it is necessary to consider what’s being transferred from the voice reference. Past research can be summarized into the following categories: (1) voice timbre transfer [ 10 , 9 , 12 ] , (2) voice timbre and prosody transfer [ 11 ] , (3) further incorporating performance attributes such as rhythmic and pitch habits [ 16 ] which are often seen as music performance style transfer [ 17 ] , or (4) using vague, data-driven definitions [ 14 , 15 , 18 ] . This paper focuses on (1) voice timbre, while the prosody and performance style are controllable by additional model input conditions. We envision that this design is comparatively more beneficial to real-world applications. The key to enabling singing style control and voice transfer without changing the content is the disentanglement of these elements in the signal. Models such as [ 10 , 11 , 19 ] learn such disentanglement through conditional synthesis with large-scale data training. GR0 [ 9 ] uses self-supervised learning to separate voice timbre (global voice embedding) from other components (local content embedding). Our model is built on the assumption that voice, content, and singing styles are naturally disentangled. Our SVC model utilizes pre-trained disentangled representations such as speaker embedding and content embedding in GR0. To achieve good musicality, our SVS model is conditioned on more granular representations called expressive performance attributes including pitch curves, loudness contour, and pronunciation, while adding controls for language, singing style, and technique, following the framework of ExpressiveSinger [ 16 ] . Finally, we address singing data scarcity by utilizing abundant speech data in the scheduled training framework, incorporating pre-training, fine-tuning, and mixed-training strategies. Fig. 1: Proposed zero-shot SVS and SVC framework. Our unified framework, illustrated in Fig. 1 , introduces three models: one for zero-shot SVS and two for zero-shot SVC. First, building on ExpressiveSinger and inspired by GR0’s self-supervised training, we develop a zero-shot SVS with improved timbre embedding and a mixed training approach using both singing and speech data. We then extend this framework to zero-shot SVC with lyrics annotation for language, style, and technique control. Finally, we incorporate GR0’s local content embedding, enabling conversion without additional lyrics input. The key contributions of this paper are: (1) We design a system that simultaneously integrates high-quality SVS, SVC, and zero-shot capabilities for cross-domain scenarios using only a 5-second speech reference. (2) Our framework provides fine-grained control over: linguistic content (via lyrics), performance attributes (via musical score), singing style and vocal techniques (via a selector), and voice identity (via speech samples) in zero-shot scenarios, significantly advancing the flexibility and expressiveness of current zero-shot SVS and SVC. (3) We emphasize the crucial role of performance attributes, and demonstrate that better disentanglement and control of singing components—especially expressive performance attributes—improve singing generation quality and mitigate data scarcity. (4) We introduce a method for integrating speech data into singing model training without compromising output quality, addressing the scarcity of singing data, and offering insights for similar low-data scenarios. (5) Experiments including extensive ablation studies and large-scale subjective evaluations, show improvements in singing performance and also offer insights for speech. II Method II-A Unified Framework Our method is built on a unified framework, as shown in Fig. 1 , from which we derive three models: one zero-shot SVS model and two zero-shot SVC models. Inspired by ExpressiveSinger [ 16 ] , the core principle here is to break down singing into three components: musical perspective attributes ( expressive performance control [ 17 ] ), pronunciation related to lyrics, and voice timbre. Each component is modeled separately, and the synthesis process learns to disentangle them in training. During inference, the main synthesizer takes all three components and generates the output singing. It consists of two parts: an acoustic model that generates a mel-spectrogram from the component outputs and a neural vocoder that converts the mel-spectrogram into a waveform. The three components are defined as follows. First, expressive performance control attributes capture how a singer interprets a music score with personal style and emotion. Building on ExpressiveSinger, we utilize two key attributes:

Read the complete text