LAPS-Diff: A Diffusion-Based Framework for Singing Voice Synthesis With Language Aware Prosody-Style Guided Learning
Sandipan Dhar, Mayank Gupta, Preeti Rao · 2025
Read LAPS-Diff: A Diffusion-Based Framework for Singing Voice Synthesis With Language Aware Prosody-Style Guided Learning by Sandipan Dhar, Mayank Gupta, Preeti Rao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04966.txt.
Introduction
Sandipan Dhar, Mayank Gupta, and Preeti Rao build a singing voice synthesizer tuned to Hindi film songs, in this 2025 paper. LAPS-Diff is a diffusion model fed with rich word and sound-unit features from pre-trained language tools. A style encoder and a pitch tracker teach it the vocal manner and pitch bends that mark the Bollywood style.
Music and speech models add further guidance on tone and context. On a small Hindi dataset, the sort of scarce setting the authors target, the system beat a leading rival in both lab measures and listener ratings. They present it as a path for singing synthesis in low-resource languages.
Catalog record
| CREATOR | Sandipan Dhar, Mayank Gupta, Preeti Rao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-04966
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LAPS-Diff: A Diffusion-Based Framework for Singing Voice Synthesis With Language Aware Prosody-Style Guided Learning Sandipan Dhar, Mayank Gupta, Preeti Rao Abstract The field of Singing Voice Synthesis (SVS) has seen significant advancements in recent years due to the rapid progress of diffusion-based approaches. However, capturing vocal style, genre-specific pitch inflections, and language-dependent characteristics remains challenging, particularly in low-resource scenarios. To address this, we propose LAPS-Diff, a diffusion model integrated with language-aware embeddings and a vocal-style guided learning mechanism, specifically designed for Bollywood Hindi singing style. We curate a Hindi SVS dataset and leverage pre-trained language models to extract word and phone-level embeddings for an enriched lyrics representation. Additionally, we incorporated a style encoder and a pitch extraction model to compute style and pitch losses, capturing features essential to the naturalness and expressiveness of the synthesized singing, particularly in terms of vocal style and pitch variations. Furthermore, we utilize MERT and IndicWav2Vec models to extract musical and contextual embeddings, serving as conditional priors to refine the acoustic feature generation process further. Based on objective and subjective evaluations, we demonstrate that LAPS-Diff significantly improves the quality of the generated samples compared to the considered state-of-the-art (SOTA) model for our constrained dataset that is typical of the low resource scenario. I Introduction Singing Voice Synthesis (SVS) is the process of generating natural-sounding singing voices using statistical or Artificial Intelligence (AI)-based algorithms, guided by musical scores [ 1 ] . Unlike Text-to-Speech (TTS) synthesis systems that convert text into audible speech [ 2 ] , the primary objective of SVS models is to produce a singing voice that is accurately synchronized with the musical composition as given by a music score. Like a TTS model, an SVS model also follows a three-stage pipeline: converting the input music score into bottleneck features, generating acoustic features from bottleneck features, and reconstructing the audible singing voice [ 3 ] . In recent years, the growing use of SVS applications in the media and entertainment industry has highlighted the significant potential of this evolving field [ 2 , 4 ] . In the recent past, improved variants of diffusion models have exhibited significant advancements in SVS [ 5 , 6 , 7 , 8 ] . In particular, the DiffSinger model [ 1 ] is widely recognized as a pioneering approach in diffusion-based SVS research and has demonstrated promising results. It includes an auxiliary decoder that generates the target mel-spectrogram from music score embeddings. Specifically, the shallow diffusion mechanism introduced in DiffSinger accelerates the inference by providing the denoiser a boundary predicted intermediate mel-spectrogram rather than the noise input. Learn2Sing 2.0 [ 2 ] is also a well-known diffusion-based approach in which the authors adapted the GradTTS framework [ 9 ] to transform speech data into singing audio. In Learn2Sing 2.0, the authors effectively demonstrated how TTS models can be adapted for SVS with significant adjustments. Specifically, to improve the denoising process in the reverse diffusion for accurate estimation of the denoised mel-spectrogram having precise detail of the speaker’s vocal characteristics, Learn2Sing 2.0 [ 2 ] introduced style embedding as an additional conditioning feature within the Grad-TTS framework. As a newer variant of diffusion-based approach, the authors of MakeSinger [ 5 ] introduced an efficient semi-supervised training mechanism, enabling the model to learn from both labeled and unlabeled singing data. They conditioned the denoiser on text, pitch, and speaker embeddings, ensuring an improved reverse denoising process. While diffusion-based approaches have shown impressive results, models like VISinger2 [ 10 ] have also exhibited notable performance in SVS by utilizing a conditional variational autoencoder (CVAE)-based framework. Although the discussed models have shown significant advancements, most of these existing SVS models, including DiffSinger, have been trained and evaluated on singing audio datasets [ 11 , 12 , 13 , 14 ] that typically contain more than 5 hours of audio data (single voice) in languages such as Mandarin, Korean, Japanese, and English. Moreover, each of these language-specific singing datasets belongs to a distinct musical genre with its own peculiar characteristics for melody (pitch variation), rhythm, and pronunciation. These aspects are also strongly influenced by the individual singer’s unique vocal characteristics. However, due to the scarcity of labeled singing data in most languages and genres, it is highly challenging for an SVS model to effectively capture linguistic content, style, and pitch related information from low-resource singing data, highlighting a significant research gap. To address this challenge of capturing content information, vocal style, and pitch variations from low-resource SVS data, we introduce LAPS-Diff, a diffusion model that incorporates language-aware embeddings and prosody-style guided learning mechanism. The proposed model is built on the DiffSinger framework [ 1 ] . As part of this work, we curate a dataset of about one hour duration of Hindi Bollywood-style songs by a single male singer and process the audio to obtain the music scores. To effectively capture the lyrical content information, we leverage additionally two pre-trained language models, IndicBERT [ 15 ] and XPhoneBERT [ 16 ] , for Hindi word and phone-level embeddings. We combine these embeddings with music score embedding [ 1 ] to form an enriched content representation. Further, we integrate a style encoder [ 17 ] and pre-trained JDCNet pitch extraction model [ 18 , 19 ] to obtain style and pitch information (melody) to compute the corresponding losses for the training of the auxilary decoder of our proposed LAPS-Diff model. Given the critical importance of preserving expressive pitch inflections in the songs in the genre of interest, we go beyond L1 loss of F0 (or log F0), and investigate the role of linear correlation in the loss function. The resulting style and pitch losses are found to improve the model’s ability to effectively capture vocal style and pitch-related information, enabling a closer resemblance to the underlying singing dynamics. Moreover, we utilize pre-trained MERT [ 20 ] and IndicWav2Vec [ 21 ] models for extracting musical feature embeddings and contextual embeddings, respectively, to use as conditional priors to the denoiser for improving the reverse diffusion process [ 22 ] through explicit feature guidance. Finally, we evaluate the performances of the proposed LAPS-Diff model and the considered state-of-the-art (SOTA) SVS model using our Bollywood Hindi dataset with objective and subjective measures and discuss the obtained improvements. II Dataset Given that music generation must satisfy strong musical and cultural norms, it is critical to test with a dataset of the music in the genre of interest. We select the vastly popular genre of Indian music called Bollywood where iconic singers have large repertoires and significant following. While considered a pop genre in terms of song structure, Bollywood is known to be influenced by Indian classical traditions in terms of drawing on melodic material from ragas and folk songs, with Hindi as its primary language 1 1 1 Dataset details, and samples of reference and generated audio, can be accessed at anonymised site: https://shorturl.at/2uARh . We choose songs by popular singer Arijit Singh over the period 2012-2020. The selected 37 songs are in a single male voice. The original song audio (as obtained from publicly accessible sites) is processed for vocals separation using a commercial tool [ 23 ] . Automatically detected silences greater than 500