DiffRhythm+: Controllable and Flexible Full-Length Song Generation with Preference Optimization * Corresponding author.

Huakang Chen 2 , Yuepeng Jiang 2 , Guobin Ma 2 , Chunbo Hao 2 Shuai Wang 3 , Jixun Yao 2 , Ziqian Ning 2 , Meng Meng 4 , Jian Luan 4 , Lei Xie * 2 · 2025

Read DiffRhythm+: Controllable and Flexible Full-Length Song Generation with Preference Optimization * Corresponding author. by Huakang Chen 2 , Yuepeng Jiang 2 , Guobin Ma 2 , Chunbo Hao 2 Shuai Wang 3 , Jixun Yao 2 , Ziqian Ning 2 , Meng Meng 4 , Jian Luan 4 , Lei Xie * 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12890.txt.

Introduction

Chen and nine co-authors upgrade a diffusion-based system that generates full-length songs with vocals and accompaniment. The earlier version suffered from lopsided training data and weak control over style. Songs came out uneven, with repeated or dropped lyrics.

The new system trains on a larger, balanced dataset. Users can steer style through written descriptions or a reference clip. A preference-tuning step nudges the model toward outputs listeners rate higher. Tests showed gains in naturalness, arrangement richness, and listener satisfaction over the earlier system. The authors present the work as a step toward songs a creator can shape in detail.

Catalog record

CREATORHuakang Chen 2 , Yuepeng Jiang 2 , Guobin Ma 2 , Chunbo Hao 2 Shuai Wang 3 , Jixun Yao 2 , Ziqian Ning 2 , Meng Meng 4 , Jian Luan 4 , Lei Xie * 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12890

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiffRhythm+: Controllable and Flexible Full-Length Song Generation with Preference Optimization * Corresponding author. Huakang Chen 2 , Yuepeng Jiang 2 , Guobin Ma 2 , Chunbo Hao 2 Shuai Wang 3 , Jixun Yao 2 , Ziqian Ning 2 , Meng Meng 4 , Jian Luan 4 , Lei Xie * 2 Abstract Songs, as a central form of musical art, exemplify the richness of human intelligence and creativity. While recent advances in generative modeling have enabled notable progress in long-form song generation, current systems for full-length song synthesis still face major challenges, including data imbalance, insufficient controllability, and inconsistent musical quality. DiffRhythm, a pioneering diffusion-based model, advanced the field by generating full-length songs with expressive vocals and accompaniment. However, its performance was constrained by an unbalanced model training dataset and limited controllability over musical style, resulting in noticeable quality disparities and restricted creative flexibility. To address these limitations, we propose DiffRhythm+, an enhanced diffusion-based framework for controllable and flexible full-length song generation. DiffRhythm+ leverages a substantially expanded and balanced training dataset to mitigate issues such as repetition and omission of lyrics, while also fostering the emergence of richer musical skills and expressiveness. The framework introduces a multi-modal style conditioning strategy, enabling users to precisely specify musical styles through both descriptive text and reference audio, thereby significantly enhancing creative control and diversity. We further introduce direct performance optimization aligned with user preferences, guiding the model toward consistently preferred outputs across evaluation metrics. Extensive experiments demonstrate that DiffRhythm+ achieves significant improvements in naturalness, arrangement complexity, and listener satisfaction over previous systems. Audio samples and code are available at https://longwaytog0.github.io/DiffRhythmPlus/ and https://github.com/ASLP-lab/DiffRhythm . I Introduction Music plays an essential role in human culture, serving as a profound medium for emotional expression, cultural transmission, and social interaction throughout history [ 1 , 2 ] . Despite its ubiquity, traditional music composition remains technically demanding, typically requiring extensive expertise and substantial resources, thus limiting accessibility for broader audiences [ 3 ] . Recent advancements in neural generative models have significantly lowered barriers, enabling more accessible, efficient, and interactive music creation. These developments facilitate various applications, such as personalized music generation for short videos, film scoring, educational tools, and therapeutic practices [ 4 , 5 ] . Research within the domain of music generation generally encompasses three primary areas: Singing Voice Synthesis (SVS), text-to-music generation, and lyric-to-song generation. SVS generates expressive, human-like singing voices from provided lyrics and musical scores [ 6 , 7 , 8 , 9 , 10 ] , supporting applications such as virtual singers, artist voice cloning, and assistive tools. In contrast, text-to-music generation creates instrumental music conditioned on textual descriptions, mood cues, or prompts [ 11 , 12 , 13 , 14 , 15 , 16 ] . However, both approaches, when used independently, present inherent limitations. SVS models typically produce only vocal tracks without instrumental accompaniment, whereas text-to-music systems generate instrumental tracks lacking vocal melodies and lyrics. In real-world compositions, vocals and accompaniment intertwine to create rich semantic and acoustic coherence, posing substantial challenges for comprehensive song generation. Song generation specifically addresses this challenge by synthesizing complete songs, including both vocals and instrumentals, directly from raw lyrics and style prompts. Current methods generally adopt either autoregressive language model (LM)-based approaches or diffusion-based techniques. LM-based approaches [ 17 , 18 , 19 , 20 , 21 ] formulate song generation as sequential modeling tasks, employing large-scale language models to produce musical tokens from lyrics in an autoregressive manner. While effective at capturing complex musical structures and multi-modal conditioning, these models suffer from slow inference, hindering real-time and interactive applications. In contrast, diffusion-based approaches [ 12 , 22 , 23 , 24 ] generate song by iteratively denoising latent representations, providing superior controllability, high audio fidelity, and robust long-term coherence. Diffusion-based approaches excel at flexible conditioning on various musical attributes, enabling nuanced control over melody, rhythm, and timbre. Furthermore, diffusion-based methods naturally support detailed editing, inpainting, and style transfer, making them highly versatile and expressive for song generation tasks. DiffRhythm [ 25 ] is the first open-source diffusion-based model designed for full-length song generation, capable of synthesizing both vocal and accompaniment tracks for songs up to more than 4 minutes. DiffRhythm stands out from previous methods that rely on complex pipelines or can only generate short or isolated musical elements, by enabling fast, end-to-end generation of complete songs. With only lyrics and a style prompt as input, it can deliver a full-length song in as little as ten seconds, making it ideal for real-time and interactive music applications. Despite its strengths, DiffRhythm exhibits several notable limitations. First, the training dataset used in DiffRhythm is imbalanced, with an approximate ratio of 3:6:1 among Chinese songs, English songs, and instrumental music. This imbalance results in significantly poorer performance for Chinese songs compared to English ones, often causing repetitive or missing lyrics. Second, recent studies such as YuE [ 21 ] have shown that large-scale training data is crucial for improving musical complexity and expressive quality. Consequently, the relatively limited scale of DiffRhythm’s dataset constrains its ability to produce outputs with rich structural complexity, detailed arrangements, and nuanced instrumental expression. Moreover, DiffRhythm’s style control is restricted to audio-based prompts, limiting the flexibility and precision of stylistic variations. Finally, the model occasionally demonstrates instability, generating outputs with inconsistent clarity and coherence. To address these shortcomings, we propose DiffRhythm+, an enhanced song generation framework that significantly improves style control flexibility and aligns generated outputs closely with listener preferences. DiffRhythm+ substantially elevates the quality, diversity, and controllability of generated music. The key contributions of DiffRhythm+ are summarized as follows: • We propose DiffRhythm+, an advanced framework for full-length song generation that integrates multi-modal style control, large-scale balanced training data, and listener preference optimization, significantly enhancing the quality and creative versatility of AI-generated music. • We incorporate preference optimization guided by music aesthetic evaluation models, aligning generated outputs more closely with human preferences and enhancing musical expressiveness, arrangement complexity, and listener satisfaction. • We employ MuLan [ 26 , 27 ] , a multi-modal style extractor that enables fine-grained style conditioning through both textual descriptions and audio prompts, greatly increasing flexibility and creative control. Fig. 1 : (a) Overview of the DiffRhythm+ architecture. Style descriptions are encoded by a style extractor, and lyrics are tokenized by a text tokenizer; both resulting embeddings serve as external control signals to the DiT-based conditional flow matching module. A VAE module decodes the resulting latent variables to produce the final audio output. (b) Detailed structure of the conditional flow matching module based on DiT. II Related Work II-A Language Model-Based Song Generation Early work in language model-based song generation, such as Jukebox [ 17 ] , leveraged hierarchical VQ-VAE architectures combined with transformer decoder architecture to generate complete songs from textual inputs. More recent models, such as MelodyLM [ 18 ] , SongCreator [ 19 ] , and SongGen [ 20 ] , introduce multi-stage tokenization schemes and incorporate multi-modal conditioning to enhance the alignment between generated audio components and input lyrics. These approaches represent meaningful progress in modeling musical structure, lyric expressiveness, and stylistic control. However, they still face significant challenges in generating long-form, high-fidelity audio with consistent coherence across sections.

Read the complete text