JAM : A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment

Renhang Liu, Chia-Yu Hung, Navonil Majumder, Taylor Gautreaux, Amir Ali Bagherzadeh, Chuan Li, Dorien Herremans, Soujanya Poria · 2025

Read JAM : A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment by Renhang Liu, Chia-Yu Hung, Navonil Majumder, Taylor Gautreaux, Amir Ali Bagherzadeh, Chuan Li, Dorien Herremans, Soujanya Poria (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20880.txt.

Introduction

Renhang Liu and co-authors present a 2025 arXiv paper on JAM, a compact flow-based model that turns lyrics into songs. Recent song generators sound decent but give musicians little control at the word level. JAM claims the first word-level timing and duration control in this class, letting users place each sung word in time.

A preference tuning method polishes quality using a synthetic feedback set, with no hand labels needed. The team also released a public test set, JAME, to standardize how lyrics-to-song models are judged. On the authors' measures, JAM beat current rivals on music-specific traits while staying small.

Catalog record

CREATORRenhang Liu, Chia-Yu Hung, Navonil Majumder, Taylor Gautreaux, Amir Ali Bagherzadeh, Chuan Li, Dorien Herremans, Soujanya Poria
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-20880

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

JAM : A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment Renhang Liu, Chia-Yu Hung, Navonil Majumder, Taylor Gautreaux, Amir Ali Bagherzadeh, Chuan Li, Dorien Herremans, Soujanya Poria Abstract Diffusion and flow-matching models have revolutionized automatic text-to-audio generation in recent times. These models are increasingly capable of generating high quality and faithful audio outputs capturing to speech and acoustic events. However, there is still much room for improvement in creative audio generation that primarily involves music and songs. Recent open lyrics-to-song models, such as, DiffRhythm, ACE-Step, and LeVo, have set an acceptable standard in automatic song generation for recreational use. However, these models lack fine-grained word-level controllability often desired by musicians in their workflows. To the best of our knowledge, our flow-matching-based JAM is the first effort toward endowing word-level timing and duration control in song generation, allowing fine-grained vocal control. To enhance the quality of generated songs to better align with human preferences, we implement aesthetic alignment through Direct Preference Optimization, which iteratively refines the model using a synthetic dataset, eliminating the need for manual data annotations. Furthermore, we aim to standardize the evaluation of such lyrics-to-song models through our public evaluation dataset JAME . We show that JAM outperforms the existing models in terms of the music-specific attributes. 1 Introduction Music plays an essential role in human culture: it brings people together, expresses emotions, embodies cultural elements, and through them enriches our daily lives via shared experiences. From ancient rituals to modern celebrations, music has shaped our social bonds and personal well–being ( Freeman III 1998 ; Agres et al. 2021 ) . Creating music, however, is a complex and time–consuming process that often demands extensive effort from skilled musicians. Recent advances in neural generative models have made AI–generated music a reality, providing composers with reliable first drafts that accelerate the creative workflow. The recent audio-based generative music AI models can be grouped into three main trends: • Singing voice generation focuses on producing expressive vocal performances from given lyrics and musical notes—often using techniques like voice cloning—without generating instrumental accompaniment. • Textual description to music generation learns to translate prompts that may include instructions about mood, desired musical features, or instrumentation into fully synthesized music, but typically does not handle vocals. These works are very similar to text-to-audio generative models, such as, Tango series models ( Ghosal et al. 2023 ; Majumder et al. 2024a ; Hung et al. 2025a ) , AudioLDM ( Liu et al. 2024 ) , and Stable Audio Open ( Evans et al. 2024 ) . These models are capable of generating both sound effects and instrumental music from textual prompts. • Lyrics-to-song generation combines both singing voice and accompaniment to produce full songs, ensuring semantic coherence between lyrics and music as well as acoustic harmony across vocals and instruments. Among these, lyrics-to-song generation presents unique challenges in aligning linguistic content with musical structure, preserving prosody, and generating high–quality audio across longer durations. Existing approaches to lyrics-to-song generation fall into two broad categories. Autoregressive models generate audio tokens sequentially, which allows them to maintain strong long–range coherence and to incorporate explicit musical style controls. However, their step–by–step decoding can be prohibitively slow for practical use. In contrast, diffusion–based methods iteratively denoise latent representations, offering high audio fidelity, flexible conditioning on melody, rhythm, and timbre, and natural support for editing and style transfer. Despite these advantages, diffusion models can still struggle with generation speed and fine–grained alignment between lyrics and audio. Recent systems such as DiffRhythm ( Ning et al. 2025 ) , YuE ( Yuan et al. 2025 ) , LeVo ( Lei et al. 2025 ) , and ACE-Step ( Gong et al. 2025 ) have demonstrated impressive results, but they share several limitations: 1. Large model size: All are based on hundreds of millions to billions of parameters, leading to slow inference regardless of the generation paradigm. This also makes the model more resource hungry. 2. Coarse timing control: While DiffRhythm allows specifying sentence–level start times, none support word– or phoneme–level alignment, limiting the user’s ability to shape prosody and rhythm precisely. 3. Weak lyric fidelity: High word error rates (WER) and phoneme error rates (PER) indicate that the models often misalign or omit lyric content. 4. Lack of duration control: Without explicit control over the overall song duration and inter–word pauses, users cannot easily shape the global structure or pacing of the generated song. To address these challenges, we introduce JAM , a rectified–flow based model for lyrics to song generation. JAM is a 530M–parameter conditional flow–matching model ( Lipman et al. 2023 ) built on 16 LLaMA–style Transformer layers as the Diffusion Transformer (DiT) backbone ( Peebles and Xie 2023 ) . It is jointly conditioned on: • Lyrics , with fine–grained word– and phoneme–level start/end times for precise prosody control. • Target duration , guiding the model on the overall song length and the spacing between vocal phrases. • Style prompt , which can be either a reference audio clip or a text description, to capture desired timbral and structural characteristics. JAM generates full songs at 44.1kHz for up to 3 minutes and 50 seconds by learning a rectified flow trajectory through the latent space of a variational autoencoder (VAE). Our contributions are: 1. Compact architecture: At 530M parameters, JAM is less than half the size of the next smallest system (Diffrhythm-1.1B), enabling faster inference and is less resource demanding. 2. Fine–grained alignment: By accepting word– and phoneme–level timing inputs, JAM lets users control the exact placement of each vocal sound, improving rhythmic flexibility and expressive timing. 3. Enhanced lyric fidelity: This precise alignment reduces WER and PER by over 3× compared to prior work, as the model can directly attend to phoneme boundaries and correct misalignments. 4. Global duration control: Our novel duration mechanism not only sets the inter–word pacing implicitly (it is controlled explicitly through lyric timing) but also specifies how much instrumental introduction and coda to generate, giving composers full control over song structure. 5. Rigorous evaluation: Assessing prior methods is difficult when their training data is undisclosed. To avoid data contamination, we compiled lyrics for 250 tracks released after the models’ training cut‑offs—ensuring neither JAM nor any baseline had access to them. Additionally, these tracks span a variety of genres, allowing us to evaluate performance across different musical styles. 6. Aesthetic alignment: Most prior systems (with the exception of LeVo) lack any mechanism for aligning model outputs to human aesthetic preferences. In text–to–audio work such as Tango2 and Tangoflux, preference alignment has proven effective, and LeVo recently adapted this idea for song generation. However, LeVo relies on a manually annotated preference dataset, which incurs significant human effort. Inspired by Tango2 and Tangoflux, we instead use automated song–quality models like SongEval to generate synthetic preference labels. We further apply this alignment in multiple rounds, yielding additional performance gains. With these advances, JAM offers an efficient, controllable, and high–fidelity solution for turning lyrics into complete songs, paving the way for AI–assisted composition in both professional and amateur settings. 2 Related Works 2.1 Music Generation Music generation aims to generate coherent and aesthetically pleasing musical content, either as raw audio, conditioned on various inputs

Read the complete text