Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning

Jisi Zhang, Pablo Peso Parada, Md Asif Jalal, Karthikeyan Saravanan · 2025

Read Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning by Jisi Zhang, Pablo Peso Parada, Md Asif Jalal, Karthikeyan Saravanan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14680.txt.

Description

Jisi Zhang and three co-authors present a 2025 paper on generating music from text with diffusion models. Most such models guide generation using text features from one source only. The team conditions their model on two views at once: detailed local features from a language model and a global summary from an audio-language model. The global view steers the whole piece while the local view tracks detail.

They also show the global view can be squeezed out of the language model itself by pooling tricks, dropping the second encoder and saving parameters. Adding the audio-language summary improved how well the music matched its prompt. The pooled version gave the best raw audio quality with slightly weaker prompt match. The authors present both routes as efficient options. The trade-off between quality and text fit is mapped for builders of music generators.

Catalog record

CREATORJisi Zhang, Pablo Peso Parada, Md Asif Jalal, Karthikeyan Saravanan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14680

Browse the full library · About the Tower of Babel