Versatile Symbolic Music-for-Music Modeling via Function Alignment
2025
Read Versatile Symbolic Music-for-Music Modeling via Function Alignment (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15548.txt.
Introduction
Junyan Jiang and co-authors unify music tasks by treating labels as music itself. A chord chart, after all, can be written as notes. Their 2025 paper frames both understanding and generation as music-for-music sequence modeling: chord reading in, melody out, and so on.
The design runs two pretrained language models, one for the source music and one for the target. A light adapter links them, so the heavy models stay frozen. Across chord recognition, melody generation, and drum track generation, the method beat task-built rivals. Code, demos, and weights are public. Music AI readers get a cheap way to turn one strong model into many music skills.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15548
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Versatile Symbolic Music-for-Music Modeling via Function Alignment Abstract Many music AI models learn a map between music content and human-defined labels. However, many annotations, such as chords, can be naturally expressed within the music modality itself, e.g., as sequences of symbolic notes. This observation enables both understanding tasks (e.g., chord recognition) and conditional generation tasks (e.g., chord-conditioned melody generation) to be unified under a music-for-music sequence modeling paradigm. In this work, we propose parameter-efficient solutions for a variety of symbolic music-for-music tasks. The high-level idea is that (1) we utilize a pretrained Language Model (LM) for both the reference and the target sequence and (2) we link these two LMs via a lightweight adapter. Experiments show that our method achieves superior performance among different tasks such as chord recognition, melody generation, and drum track generation. All demos, code and model weights are publicly available 1 1 1 https://ismir2025submission25.github.io/function-alignment/ . 1 Introduction Many foundational tasks in music AI, such as music information retrieval (MIR) and conditional music generation, have traditionally been formulated as mappings between music and labels: either from music to task-specific annotations (e.g., chord recognition), or from descriptive conditions to music (e.g., chord-conditioned melody generation). While these tasks have long been treated separately, a key observation is that in many cases, the “labels” themselves can also be represented in the same music modality —for example, as note sequences. This suggests a unifying perspective: a wide range of MIR and generation tasks can be reformulated as sequence-to-sequence problems within the music domain. We refer to this formulation as music-for-music modeling. To achieve versatile music-for-music modeling in a sample-efficient way, we apply knowledge transfer to pretrained foundational Language Models (LMs) using a light-parameterized adaptor. As illustrated in Fig. 1 (a)-(b), many existing methods such as probing [ 1 , 2 , 3 ] and prefix tuning [ 4 , 5 , 6 ] transfer knowledge of foundation models to downstream tasks by adapting them to new input or output, but the knowledge resides in only one language—either the LM of source \mathbf{x} or the LM of target \mathbf{y} . In contrast, our method distills knowledge from both LMs via aligning them in a layer-wise manner, as shown in Fig. 1 (c). Figure 1 : Three types of sequence-to-sequence models by knowledge transfer from pretrained LMs. \mathbf{x} and \mathbf{y} are input sequences and \hat{\mathbf{x}} and \hat{\mathbf{y}} are predictions, possibly right-shifted due to the autoregressive targets. (a) Probing; (b) Prefix tuning; (c) Function alignment ( \mathbf{x}\rightarrow\mathbf{y} ). At the methodology level, our approach is inspired by function alignment [ 7 ] , a recently proposed theory of mind that attributes the emergence of intelligence to the dynamic synergy among interacting agents. In our work, we contribute two concrete implementations of this idea—treating two language models (LMs) as agents and creating synergy through Parameter-Efficient Fine-Tuning (PEFT). The first approach introduces a trainable cross-attention layer between two separately pretrained LMs. The second, more concise solution, uses a lightweight self-attentive adapter applied to concatenated input-output sequences within a single shared LM —a strategy applicable when both input and output share the same vocabulary. We show the effectiveness of both implementations using experiments on both generative and analysis tasks, including: (1) chord-conditioned melody generation, (2) melody-conditioned chord generation, (3) drum-conditioned song generation, (4) song-conditioned drum generation and (5) few-shot symbolic music analysis. The main contribution of this paper is as follows: 1. We achieve versatile music-for-music modeling, unifying a broad range of music understanding and controllable generation tasks under a shared framework. 2. At the methodological level, we are the first to introduce function alignment—a recently proposed theory of mind emphasizing the synergy among agents—into the domain of music AI, offering a novel perspective on modeling music sequence-to-sequence tasks. 3. While function alignment remains at a theoretical level, we present two concrete, parameter-efficient implementations in the context of modern language models: one via cross-attentive adapters across two LMs, and another via a self-attentive adapter within a shared LM. We demonstrate the effectiveness of both approaches through theoretical analysis and empirical validation. 2 Related Works 2.1 Music Foundation Models Since the invention of the Transformer architecture [ 8 ] , transformer-based language models have become the mainstream of music foundation models on multiple modalities, including audio [ 9 , 10 , 11 , 12 ] , symbolic [ 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ] and text-based music representation [ 22 ] . In addition to autoregressive models, masked language models [ 2 , 23 ] and diffusion models [ 24 , 25 , 26 , 27 , 28 , 29 ] and flow-based models [ 30 ] can also be used as foundation models, but we focus on autoregressive models in the literature review. For symbolic music, music transformer [ 13 ] is an early work to adopt the transformer architecture to music. Some follow-up works try to design a better representation of the music content. For example, pop music transformer imposes a metrical structure in the data representation [ 15 ] . MuPT trains transformers on their proposed synchronized multi-track ABC notation [ 20 ] . Other works aim to introduce controllability to the generative model. Musecoco generates the music score from text [ 14 ] . METEOR performs melody-aware orchestral music generation with texture control [ 16 ] . SymPAC trains symbolic generation models from transcribed audio data with chord, section, and instrument controls [ 17 ] . Zhang et al. improve generation discriminators to better follow rhythm and melody conditions [ 18 ] . The Theme Transformer [ 19 ] uses a short theme condition for generation. MuseBarControl generates music with fine-grained control to the bar level [ 21 ] . 2.2 Parameter-Efficient Fine-Tuning Parameter-Efficient Fine-Tuning (PEFT) methods add lightly parameterized adapters to large pretrained models. Compared to full-parameter fine-tuning, PEFT requires significantly less computation and training data. Existing methods include appending task-specific prefixes to input sequences [ 4 , 31 ] , injecting low-rank adaptation (LoRA) to linear layers [ 32 ] , and adding learnable hidden states to the self-attention blocks [ 5 , 33 ] . PEFT has been applied to music foundation models to support new tasks. Coco-Mulla [ 6 ] and MusiConGen [ 34 ] both adapt MusicGen to follow content controls such as chord and rhythm. Additionally, AirGen enables MusicGen to infill segments based on content controls [ 35 ] . Instruct-MusicGen extends MusicGen for music editing by text instructions [ 36 ] . Audio Prompt Adapter extends AudioLDM2 for music editing following controls such as genre, timbre, and melody [ 37 ] . Ou et al. finetunes a symbolic language model for band arrangement, piano reduction, drum arrangement, voice separation, and more [ 38 ] . 3 Methodology 3.1 Base Model For this study, we choose the base model (the pretrained symbolic LM) with two main considerations. First, we do not wish to introduce any control in the pretraining stage, since we want to demonstrate the controllability using PEFT. We refain from using any annotation or metadata (i.e., chord, bar or text annotations) to pretrain the base model. Second, we want to adopt a data representation