MusicGen-Chord: Advancing Music Generation through Chord Progressions and Interactive Web-UI

2024

Read MusicGen-Chord: Advancing Music Generation through Chord Progressions and Interactive Web-UI (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00325.txt.

Introduction

Jongmin Jung, Andreas Jansson, and Dasaem Jeong extended a music generation model in 2024 so it can follow chord progressions, not just text and melody. The base model makes music from written prompts and a tune, but harmony was out of the user's control.

Their version turns single note melody codes into chord codes that can hold several notes at once. The result is music that reflects both a chord path and a text prompt. They also built a remixer app that reshapes uploaded music under text guidance. Both tools were put on a public web platform with a friendly interface, so anyone can try making and hearing AI music with chord control.

Catalog record

YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE8 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00325

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MusicGen-Chord: Advancing Music Generation through Chord Progressions and Interactive Web-UI


Abstract
MusicGen is a music generation language model (LM) that can be conditioned on textual descriptions and melodic features. We introduce MusicGen-Chord 1 1 1 https://replicate.com/sakemin/musicgen-chord , which extends this capability by incorporating chord progression features. This model modifies one-hot encoded melody chroma vectors into multi-hot encoded chord chroma vectors, enabling the generation of music that reflects both chord progressions and textual descriptions. Furthermore, we developed MusicGen-Remixer 2 2 2 https://replicate.com/sakemin/musicgen-remixer , an application utilizing MusicGen-Chord to generate remixes of input music conditioned on textual descriptions. Both models are integrated into Replicate’s web-UI using cog , facilitating broad accessibility and user-friendly controllable interaction for creating and experiencing AI-generated music.



Figure 1
:
(a) MusicGen’s melodic features in a matrix of one-hot encoded chroma vectors. (b) MusicGen-Chord’s chord progression features in a matrix of multi hot encoded chroma vectors. For example, in the chromagram above right there is an E
\flat
chord (E
\flat
, G, B
\flat
) followed by a G chord (D, G, B), followed by a C minor chord (C, E
\flat
, G), etc.

1
Introduction

The trend in generative AI emphasizes the controllability of models, allowing users to direct and refine outputs according to their preferences. Notable examples include Stable Diffusion
[
1
,
2
]
, supported by interfaces like
AUTOMATIC1111’s web-UI

[
3
]
and
ComfyUI

[
4
]
, which offer extensive user control over image generation processes. In the realm of music generation, models with enhanced controllability are emerging, offering conditions such as chord
[
5
,
6
,
7
]
, rhythm
[
5
,
6
,
7
]
, melody
[
8
,
6
]
, and style based on reference audio
[
9
]
. This paper explores the integration of controllability in music generation through the example of MusicGen-Chord.

MusicGen
[
8
]
is an auto-regressive, Transformer-based music generation model that enables user control through textual descriptions and melodic features. It processes multiple streams of compressed discrete audio representations
[
10
]
to generate high-quality, coherent, and stylistically diverse music. MusicGen-Chord extends this model by conditioning on chord progressions instead of melodies. This modification uses a matrix of multi-hot encoded chroma vectors to represent chord progression features. MusicGen-Chord was released in October 2023, and since then, several similar but more advanced studies have been introduced, such as MusiConGen
[
7
]
.

To demonstrate the practical benefits of this approach, we developed MusicGen-Remixer, an application based on MusicGen-Chord. This application allows users to upload a music track, provide a textual description prompt, and generate a new background track that is remixed with the input audio. By leveraging Replicate’s web-UI and the
cog

[
11
]
package, MusicGen-Remixer and MusicGen-Chord are made widely accessible on the cloud, promoting user-friendly interaction and broad accessibility for creating and experiencing AI-generated music.

2
MusicGen-Chord

MusicGen-Chord extends the original MusicGen model by shifting the conditioning target from melodies to chord progressions. The original MusicGen model uses one-hot encoded chroma vectors as input condition to represent melodies (
Figure
1
.(a)). In this approach, each vector indicates the presence of a single pitch class at a given time, which is effective for simple melodies but limited in capturing complex harmonic content.

We found that we can tweak this input format to a multi-hot format to represent chord condition (
Figure
1
.(b)). These multi-hot chroma vectors can encode multiple active pitch classes for each time frame, providing a more comprehensive representation of harmonic structures. This “trick” works surprisingly well, enabling MusicGen-Chord to generate chord progressions that align with the style indicated by the prompt using the pretrained MusicGen model weights,
without
requiring any fine-tuning.

The interface for MusicGen-Chord accepts chord progression inputs both as audio and text formats, automatically converting them into multi-hot chroma vectors for the model. This flexibility allows users detailed control over the harmonic structure, enabling a more interactive and customizable music creation experience. For text-based inputs, users can specify chords using a simple format that includes the root and type of each chord, defined by
ROOT:TYPE

[
12
]
. Each chord lasts for a single bar, with the option to add multiple chords in a bar by separating them with commas. For example:
"G:maj7 D:min7,G:7 C:maj7 F:7 B:min7,Bb:7 A:min7,D:7"
. These text inputs are converted into chroma representations based on the input BPM value. For audio-based inputs, a chord extraction model, BTC
[
13
]
is employed to predict symbolic chords along with their timestamps, which are then represented in the chord chroma feature, ensuring effective incorporation of the harmonic content into the music generation process.

3
MusicGen-Remixer

MusicGen-Remixer utilizes the features of MusicGen-Chord to enable the creation of remixed music tracks. This application allows users to upload a music track, provide a textual description prompt, and generate a new background track that is remixed with the input audio.

The process involves several sophisticated steps to ensure the generation of coherent and contextually relevant remixes:
3
3

3

https://github.com/sakemin/musicgen-remixer

1.

Input Music Structure Analysis:
Utilizing the
All-in-One
[
14
]
framework, the input music’s BPM and downbeats are detected to maintain temporal integrity.

2.

Source Separation:
A neural source separation model, Demucs
[
15
]
is employed to separate vocal tracks from instrumental components, ensuring the original vocal performance is preserved.

3.

Chord Progression Feature Extraction:
BTC is used to extract chord progression features from the input audio, guiding the generation of the new background track.

4.

Dynamic Time Warping:
Using
Py-TSMod
[
16
]
, the timing of the generated track is adjusted to match the downbeats of the input audio, ensuring rhythmic consistency.

5.

Mixing:
The aligned background track is mixed with the separated vocal track to produce a cohesive remixed output.

Figure 2
:
Replicate’s webUI demo of MusicGen-Chord.

4
REPLICATE INTEGRATION

Replicate’s web-UI, combined with the
cog
package, provides a seamless and convenient platform for deploying AI models. The
cog
package can encapsulate AI models with all their dependencies, including Python packages, operating system components, and CUDA versions. This integration ensures that models are portable and easily deployable, facilitating a user-friendly interface for model interaction and management .

MusicGen-Chord and MusicGen-Remixer are integrated with Replicate through the
cog
package, making them widely accessible on the cloud. Users can easily interact with these models via the web interface, API, or directly using the
cog
-wrapped repository
4
4

4

https://github.com/sakemin/cog-musicgen-chord
on local machines. This setup ensures a straightforward and accessible experience for generating and remixing AI-driven music, enhancing both usability and adaptability.

Replicate offers a practical solution for sharing AI model demonstrations within the MIR community. Researchers and developers can utilize Replicate as an effective tool for presenting and disseminating their work, as demonstrated by the Music Technology Group (MTG)
5
5

5

https://replicate.com/mtg
and others who have successfully used this platform. For example, we implemented MusiConGen
[
7
]
, a recent MusicGen variant with controllable chord and rhythm features, into a
cog
wrapped demo
6
6

6

https://replicate.com/sakemin/musicongen

Read the complete text