Controllable Singing Voice Synthesis using Phoneme-Level Energy Sequence
Yerin Ryu, Inseop Shin, Chanwoo Kim · 2025
Read Controllable Singing Voice Synthesis using Phoneme-Level Energy Sequence by Yerin Ryu, Inseop Shin, Chanwoo Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07038.txt.
Introduction
Singing voices whose loudness a user can sculpt over time are synthesized by Yerin Ryu and two co-authors in a 2025 audio technology paper. Singing voice synthesis has reached high audio quality, but most systems model sound probabilistically, leaving attributes like dynamics imprecise. Dynamics, the moment-to-moment loudness variation, carry much of a performance's expression.
The model is conditioned explicitly on energy sequences taken from ground-truth spectrograms, which cuts annotation costs while improving control. A phoneme-level version of the energy sequence makes the control user-friendly: loudness can be set sound by sound. The authors describe this as the first attempt at user-driven dynamics control in singing synthesis. Experiments cut the error of the produced energy sequences by over half compared with baselines and energy-predictor models, with no loss in synthesis quality.
Catalog record
| CREATOR | Yerin Ryu, Inseop Shin, Chanwoo Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07038
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Controllable Singing Voice Synthesis using Phoneme-Level Energy Sequence Yerin Ryu, Inseop Shin, Chanwoo Kim Abstract Controllable Singing Voice Synthesis (SVS) aims to generate expressive singing voices reflecting user intent. While recent SVS systems achieve high audio quality, most rely on probabilistic modeling, limiting precise control over attributes such as dynamics. We address this by focusing on dynamic control—temporal loudness variation essential for musical expressiveness—and explicitly condition the SVS model on energy sequences extracted from ground-truth spectrograms, reducing annotation costs and improving controllability. We also propose a phoneme-level energy sequence for user-friendly control. To the best of our knowledge, this is the first attempt enabling user-driven dynamics control in SVS. Experiments show our method achieves over 50% reduction in mean absolute error of energy sequences for phoneme-level inputs compared to baseline and energy-predictor models, without compromising synthesis quality. I Introduction Singing Voice Synthesis (SVS) is the task of generating singing voice waveforms from symbolic musical representations such as scores. While recent SVS systems have achieved substantial improvements in synthesis quality, most existing approaches generate expressive singing voices in a probabilistic manner conditioned only on the provided score. These existing systems lack the capability to provide users to precisely control or convey user intent and musical expression. Consequently, there has been increasing interest in developing controllable SVS frameworks that can explicitly reflect user-specified expressive attributes. For example, SinTechSVS [ 1 ] demonstrated the synthesis of singing voices with eight distinct singing techniques. More recently, Prompt-Singer [ 2 ] proposed a system for controlling singing audio via natural language prompts, and TechSinger [ 3 ] further extended controllability by enabling user manipulation of various singing techniques through natural language instructions. In this work, we focus on the controllable modeling of dynamics, a key expressive attribute in SVS. In the context of singing, As illustrated in Fig. 1, dynamics refers to the temporal variation of loudness, which is essential for conveying musical expressiveness [ 23 ] . The significance of dynamics has been widely acknowledged in the fields of music generation [ 5 ] and audio synthesis [ 6 ] . To the best of our knowledge, most existing SVS approaches [ 10 , 11 , 12 , 13 , 14 , 15 ] have not explicitly incorporated dynamics as a controllable condition. (a) Energy Plot (b) Mel-Spectrogram Fig. 1: The energy plot in (a) represents the temporal variation of energy across time, calculated as the sum of mel-spectrogram amplitudes for each frame. When the energy is low, the corresponding regions in the mel-spectrogram (b) exhibit lower amplitude across frequency bins, appearing as darker. This indicates reduced loudness or silence in the audio signal during those time intervals. According to [ 4 ] , dynamics is typically defined as a measure of the energy of an acoustic signal, computed based on the amplitude of the waveform. Since this energy can be extracted from the ground-truth waveform or spectrogram in most singing voice datasets, the need for manual annotation is significantly reduced. In this work, we present experimental results demonstrating that conditioning the model on an energy sequence enables effective control over the dynamics of the synthesized singing voice. Furthermore, we propose a method for caculating a phoneme-level energy sequence to facilitate user-friendly control, and validate its effectiveness for dynamic manipulation through experimental result. The main contributions of this paper are summarized as follows: • We show that dynamic control can be effectively achieved in a controllable SVS system by simply summing the energy sequence embedding with other input embeddings. • We introduce a phoneme-level energy sequence representation to enable intuitive and user-friendly control. • We empirically analyze the causal relationship between the input energy sequence and the dynamics of the synthesized singing voice. Audio examples are available at https://kongtory.github.io/DCSVS/ , including the baseline system, ground-truth waveforms reconstructed with a vocoder, and our proposed SVS systems conditioned on energy at the frame and phoneme levels. II Related Works II-A Singing Voice Synthesis SVS has seen significant improvements in quality enabled by advances in deep learning methodologies and large-scale singing datasets. Among the most influential studies in this field are DiffSinger [ 7 ] and VISinger [ 8 ] . DiffSinger employs a two-stage architecture featuring an encoder-decoder framework paired with a vocoder. The encoder generates a musical score embedding from input lyric, note, and duration sequences, while a diffusion-based decoder synthesizes mel-spectrograms conditioned on this embedding. A pre-trained vocoder converts the generated mel-spectrogram into waveform but is excluded from evaluation to focus solely on mel-spectrogram quality assessment. VISinger, in contrast, is an end-to-end SVS system built upon Variational Inference with adversarial learning for end-to-end text-to-speech (VITS) [ 9 ] . In this architecture, the input sequences are encoded into a latent representation, from which the decoder directly generates the waveform. Additionally, an adversarial discriminator is applied to the latent space to enhance output quality. Building upon these two foundational studies as baselines, diffusion-based [ 10 , 11 , 12 , 13 ] and VITS-based SVS models [ 14 , 15 ] have continued to evolve, further advancing the quality of synthesized singing voices. Inspired by DiffSinger, our work adopts a diffusion-based framework for SVS, utilizing a Denoising Diffusion Probabilistic Model (DDPM) [ 16 ] as the mel-spectrogram decoder. II-B Controllable Singing Voice Synthesis The audio quality of SVS has advanced significantly, and recent research increasingly emphasizes not only sound quality but also expressiveness and user-centric control. Prompt-Singer [ 2 ] pioneered the first controllable SVS model, enabling users to adjust attributes such as singer gender, vocal range, and volume through natural language prompts. However, the range of controllable attributes remains limited, and challenges persist in controlling singing style. SinTechSVS [ 1 ] and TechSinger [ 3 ] address these challenges by developing systems that manipulate singing style through vocal techniques. SinTechSVS introduced four timbre-related and four pitch-related technique sequences, successfully synthesizing singing voices with expert-annotated technique data. TechSinger further advanced this approach by enabling technique control via natural language input, utilizing a dataset annotated with singing techniques. As shown by theses studies, controlling singing style through vocal techniques represents a highly promising research direction. While prior work has relied on annotated datasets, we propose a method for controlling singing style using the energy sequence—a technique attribute that does not require manual annotation. II-C Energy Predictor The energy predictor is a module that implicitly incorporates energy information to achieve more natural speech synthesis. During training, this module is optimized by minimizing the Mean Squared Error (MSE) between the predicted energy and the ground-truth energy, thereby encouraging the model to reflect energy in the generated output. The first study to introduce an energy predictor was FastSpeech 2 [ 17 ] , and since then, speech synthesis and SVS studies [ 18 , 19 , 20 ] have adopted this approach to achieve natural singing synthesis. However, implicitly modeling musical dynamics using an energy predictor limits both the diversity and controllability of dynamics. Therefore, we propose a system that removes the energy predictor and instead uses energy sequence as an explicit input. III Method III-A Model Architecture Fig. 2: Example of input sequences used in the SVS system. The first row shows the phoneme-level musical note sequence, the