DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching

Yuepeng Jiang, Huakang Chen, Ziqian Ning, Jixun Yao, Zerui Han, Di Wu, Meng Meng, Jian Luan, Zhonghua Fu, Lei Xie 1. ASLP Lab2. MiLM Plus, Xiaomi Inc., China · 2025

Read DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching by Yuepeng Jiang, Huakang Chen, Ziqian Ning, Jixun Yao, Zerui Han, Di Wu, Meng Meng, Jian Luan, Zhonghua Fu, Lei Xie 1. ASLP Lab2. MiLM Plus, Xiaomi Inc., China (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22950.txt.

Introduction

Yuepeng Jiang and nine co-authors introduce DiffRhythm 2, a system that creates full songs with vocals matched to lyrics. Long songs are hard because words and music must stay in step over minutes, not seconds. Older fast methods often let vocals drift from the written lyrics.

The new design works block by block with flow matching, which keeps lyric alignment tighter across a song. It also learns from human feedback to suit varied tastes without merging many models, a step that hurt quality before. The authors present it as an end-to-end route to high-quality, controllable song creation.

Catalog record

CREATORYuepeng Jiang, Huakang Chen, Ziqian Ning, Jixun Yao, Zerui Han, Di Wu, Meng Meng, Jian Luan, Zhonghua Fu, Lei Xie 1. ASLP Lab2. MiLM Plus, Xiaomi Inc., China
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22950

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching Yuepeng Jiang, Huakang Chen, Ziqian Ning, Jixun Yao, Zerui Han, Di Wu, Meng Meng, Jian Luan, Zhonghua Fu, Lei Xie 1. ASLP Lab2. MiLM Plus, Xiaomi Inc., China Abstract Generating full-length, high-quality songs is challenging, as it requires maintaining long-term coherence both across text and music modalities and within the music modality itself. Existing non-autoregressive (NAR) frameworks, while capable of producing high-quality songs, often struggle with the alignment between lyrics and vocal. Concurrently, catering to diverse musical preferences necessitates reinforcement learning from human feedback (RLHF). However, existing methods often rely on merging multiple models during multi-preference optimization, which results in significant performance degradation. To address these challenges, we introduce DiffRhythm 2, an end-to-end framework designed for high-fidelity, controllable song generation. To tackle the lyric alignment problem, DiffRhythm 2 employs a semi-autoregressive architecture based on block flow matching. This design enables faithful alignment of lyrics to singing vocals without relying on external labels and constraints, all while preserving the high generation quality and efficiency of NAR models. To make this framework computationally tractable for long sequences, we implement a music variational autoencoder (VAE) that achieves a low frame rate of 5 Hz while still enabling high-fidelity audio reconstruction. In addition, to overcome the limitations of multi-preference optimization in RLHF, we propose cross-pair preference optimization. This method effectively mitigates the performance drop typically associated with model merging, allowing for more robust optimization across diverse human preferences. We further enhance musicality and structural coherence by introducing stochastic block representation alignment loss. Experimental results demonstrate that DiffRhythm 2 can generate complete songs up to 210 seconds in length, consistently outperforming existing open-source models in both subjective and objective evaluations, while maintaining efficient generation speed. To encourage reproducibility and further exploration, we release the inference code and model checkpoints. Code and weights are availble at: https://github.com/xiaomi-research/diffrhythm2 . 1 Introduction Music serves as an abstract expression of human emotion and culture. Songs represent a unique art form, combining music and language to convey emotions and stories through melody and lyrics. With the rapid development of deep learning, music generation has progressed beyond symbolic music generation ( Wang et al. 2025 ; Qu et al. 2024 ) and singing voice synthesis ( Zhang et al. 2022 ; Chen et al. 2020 ; Ning et al. 2025b ) , moving towards more challenging tasks such as instrumental music generation ( Copet et al. 2023 ; Schneider et al. 2023 ; Zhang et al. 2025 ) and song generation. Unlike singing voice synthesis, which produces vocals with predefined melodies, or instrumental music generation, which only models melody and instrumentation, song generation requires joint modeling of lyrics, structure, singing vocal and accompaniment. This is further complicated by the typical length of songs, which often exceeds three minutes, making lyric alignment and stable style modeling particularly challenging. Early approaches, such as Jukebox ( Dhariwal et al. 2020 ) and SongCreator ( Lei et al. 2024 ) , generate songs by predicting the acoustic tokens of vocals and accompaniment mixed into a single track. However, the limited information density of these tokens hampers high-quality song generation. Yue ( Yuan et al. 2025 ) improves generation quality by modeling the vocals and accompaniment on separate tracks, while SongGen ( Liu et al. 2025 ) further enhances quality by introducing interleaved dual-track prediction. Despite these improvements, Yue and SongGen predict the tracks independently, which often leads to inconsistencies between the vocals and accompaniment. LeVo ( Lei et al. 2025 ) addresses this issue by first generating a mixed track and using it to guide the separate prediction of vocals and accompaniment, thereby reducing mismatches. However, the overall quality is still limited by the information compression in the mixed track tokenization, and it is difficult to generate complex and rich accompaniment. Most of these methods are based on autoregressive frameworks, which result in slow generation speeds. This hinders real-time applications and user interactivity. Unlike the aforementioned approaches, DiffRhythm ( Ning et al. 2025a ) takes a bold approach within a non-autoregressive (NAR) framework. By leveraging the generation ability of diffusion models and continuous representations, it achieves high-quality mixed-track generation of vocals and accompaniment at over 50 times faster than autoregressive methods. However, NAR models struggle with lyric alignment in long sequences. DiffRhythm addresses this issue by conditioning on sentence-level timestamps. However, this solution significantly reduces creativity and diversity, and also imposes higher requirements on the training data. ACE-Step ( Gong et al. 2025 ) solved the alignment problem without timestamps by introducing representation alignment (REPA) ( Yu et al. 2024 ) loss with mHuBERT ( Boito et al. 2024 ) as semantic constraints. However, the addition of these constraints significantly reduced musicality, creating a delicate trade-off between lyric alignment and generation quality. In addition, pretrained models often fail to align with human preferences due to the gap between training objectives and desired song quality. Therefore, a typical method is to introduce reinforcement learning from human feedback (RLHF) in the post-training process to refine generation preferences across various musical dimensions. DiffRhythm+ ( Chen et al. 2025 ) demonstrated the feasibility of improving overall musicality with Direct Preference Optimization (DPO) ( Rafailov et al. 2023 ) . LeVo extended this approach by applying DPO to different dimensions separately and interpolating model weights to enhance overall performance. While this strategy improved balance across dimensions, it inevitably limited the upper bound of each individual capability due to the averaging effect. To address these challenges, we present DiffRhythm 2, a semi-autoregressive end-to-end framework for song generation based on block flow matching. This framework partitions latent representations into fixed-length blocks and generates each block non-autoregressively using flow matching, while maintaining autoregressive dependencies between blocks. This design enables faithful lyric alignment without requiring additional labels or constraints. The semi-autoregressive structure provides rich bidirectional context within blocks, ensuring long-sequence consistency, and supports fast inference for songs up to 210 seconds long. To accommodate block flow matching during training, we introduce stochastic block REPA loss that ensures efficient training and provides accurate representation guidance for the current block. We group similar preferences and perform pairwise optimization across groups to improve the performance of multi-preference alignment. Finally, as the block flow matching training strategy significantly increases the sequence length, we implement a high-compression music variational autoencoder (VAE), achieving high compression while still enabling high-quality audio reconstruction. To encourage reproducibility and further exploration, we will release the inference code and model checkpoints. The main contributions of this paper are summarized below: • We propose DiffRhythm 2, a novel semi-autoregressive song generation framework based on block flow matching, which is capable of producing high-quality songs with faithful lyric alignment and excellent accompaniment performance. • We introduce stochastic block REPA loss to enable representation alignment in block flow matching, thereby enhancing structural modeling and achieving remarkable coherence and hierarchical structure in long-form song generation. • We propose a cross-pair preference optimization strategy to efficiently handle multi-preference alignment, which enhances final performance while reducing the number of optimized models through a group-based optimization approach. • We design a music VAE with low-frame-rate compression at 5 Hz that preserves reconstruction quality while reducing sequence length, which not only

Read the complete text