MERIT: Learning Disentangled Music Representations for Audio Similarity

2026

Read MERIT: Learning Disentangled Music Representations for Audio Similarity (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27346.txt.

Introduction

Abhinaba Roy, Junyi Liang, and Dorien Herremans build MERIT, a music similarity framework that separates melody, rhythm, and timbre. Training pairs conditional audio generation with source-separated stems so each factor varies on its own. Each head then tracks its own dimension strongly while staying near chance on the others, on synthetic and real audio alike.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33028
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27346

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MERIT: Learning Disentangled Music Representations for Audio Similarity Abstract Current music similarity models typically compute a single, monolithic score, entangling distinct musical dimensions like melody, rhythm, and timbre. This limits user control and interpretability, making it impossible to execute nuanced queries. We introduce MERIT, a framework for learning disentangled, factor-specific music representations tailored to these three core dimensions. To overcome the lack of isolated musical variations in real-world audio, we use a novel training strategy that uses conditional audio generation and source-separated stems to strongly encourage single-factor variation in training data. Our evaluations demonstrate strong factor-wise disentanglement. Each head responds strongly to its intended perceptual dimension while remaining near chance on the others, a representational property that holds across both the synthetic training domain and independent real-world audio. In a Nutshell MERIT learns three lightweight projection heads (melody, rhythm, timbre) on top of a pre-trained frozen MERT backbone. A generative triplet pipeline provides factor-isolated training data with no manual labelling. Each head achieves \geq\!99.6\% triplet accuracy on held-out factor-controlled tests and the correct head dominates on all three independent zero-shot audio probes, confirming that disentanglement transfers to real-world audio. 1. Introduction Music similarity is inherently multi-dimensional. A solo piano cover of a rock anthem preserves the melody and harmonic identity of the original while replacing every instrument and reshaping the groove. Two recordings by the same artist often share a timbral signature with no constraint at all on melody. Within a dance genre, different tracks might share the same syncopation pattern despite covering different songs and using different instrumentation. Listeners can attend to any one of these dimensions independently, use it as the basis of a similarity judgement, and articulate why two recordings are alike along that axis without committing to a single overall verdict. Most audio similarity systems collapse this structure into singular similarity number. Learned audio embeddings such as CLAP [ 18 ] and MuLan [ 6 ] , and self-supervised audio backbones such as MERT [ 8 ] , expose a single vector per clip; cosine similarity in that vector is an aggregate score whose dependence on melody, rhythm, and timbre is fixed by pre-training and not exposed to the user. Factor-specific specialists exist for cover song detection [ 19 ] , melody similarity [ 10 ] , and instrument classification, caption generation [ 2 ] but each addresses one dimension in isolation and cannot expose the others as comparable scores from the same query. To the authors’ knowledge, no existing framework learns three similarity spaces from a unified pipeline such that each space is shaped by a different perceptual factor and all three can be queried together. We present MERIT , a representational framework that learns three independent projection heads (one each for melody , rhythm , and timbre ) from a frozen encoder backbone (Fig. 2 ). Rather than optimizing for an aggregated similarity score, our objective is the structural decoupling of these musical dimensions. The primary aim of MERIT is to achieve high functional selectivity, where each projection head responds exclusively to its target factor while remaining invariant to others. A key challenge in learning such granular representations is the lack of training data where musical factors are strictly isolated. Real-world recordings naturally entangle melody, rhythm, and timbre, making it difficult for a model to learn their independent boundaries. To address this bottleneck, we propose a triplet construction strategy that leverages both conditional audio generation and source separation. This allows us to curate a large-scale training set where only a single factor varies at a time, providing the clear supervision signal necessary for true disentanglement. Figure 1 : Monolithic vs. factor-wise music similarity. ( Left ) A conventional encoder collapses melody, rhythm, and timbre into a single indistinguishable score. ( Right ) MERIT exposes three independent, interpretable scores from the same query, enabling targeted retrieval and factor-level explanations. Scores are schematic; the scenario illustrates the piano-cover example discussed in § 1 . Each factor head leverages a MERT backbone to capture a multi-scale hierarchy of features. These features are projected through a shallow MLP architecture and trained using Circle Loss to ensure stable and efficient representation learning. Because the backbone is frozen, training is fast while maintaining the robust musical knowledge of the underlying foundation model. We assess the selectivity of MERIT representations using a disentanglement table: each of our three trained heads is evaluated on all three factor test sets. We also evaluate standard, monolithic audio representations—including CLAP and the raw MERT backbone on these same factor-controlled tests. Beyond this internal evaluation, we subject the model to zero-shot diagnostic probes using real-world collections with known perceptual structures, such as rhythmic style datasets for groove. The goal is to determine if the cross-factor selectivity pattern observed in training holds true across diverse, non-synthetic audio environments. In summary, the main contributions of this paper are: 1. A scalable data pipeline for constructing factor-controlled music triplets via generative conditioning and source separation, along with our constructed dataset. 2. MERIT, a representational architecture that demonstrates high functional selectivity by decoupling entangled musical dimensions into independent, addressable scoring channels. 3. An evaluation protocol that quantifies factor-wise selectivity, alongside zero-shot probes confirming that this selectivity generalizes to independent, real-world audio collections. Code and pre-trained models are available at https://github.com/AMAAI-Lab/MERIT . Figure 2 : MERIT architecture and training pipeline. Backbone: Audio input passes through frozen MERT encoder, layers 3-6, and 23 are concatenated and pooled. Projection: Three distinct & separate projection head maps the pooled features into Melody, Timbre, and Rhythm. Optimization & Inference: A circular loss maximizes the pair similarity margin across all triplets. During inference, retrieval is performed by cosine similarity. 2. Related Work General audio and music embeddings. Large-scale contrastive audio–language pre-training, as in CLAP [ 18 ] and MuLan [ 6 ] , produces rich audio representations by aligning audio with free-form text descriptions. Self-supervised music encoders such as MERT [ 8 ] extend masked language modelling to audio with auxiliary pitch, chroma, and beat objectives, yielding representations that probe well on a wide range of music understanding tasks. Such systems produce a single embedding per clip whose dependence on melody, rhythm, and timbre is entangled and fixed by pre-training. Contrastive and metric learning for audio. Contrastive learning with triplet [ 5 ] and Circle Loss [ 16 ] objectives has been applied to audio fingerprinting [ 1 ] , speaker verification [ 20 ] , and music tagging [ 15 , 12 ] . Most of these applications define a single similarity space; our work extends the paradigm to three simultaneous factor-specific spaces trained on independent, factor-controlled triplet datasets. Luo et al. [ 11 ] learn disentangled timbre and pitch representations for musical instrument sounds using Gaussian mixture VAEs, motivating the general principle that musical factors can be separated in the latent space; MERIT scales this idea to full musical recordings via a discriminative, shared backbone approach. Factor-specific retrieval. Cover song detection [ 19 ] focuses narrowly on melodic and harmonic identity, using features such as chroma, key, and harmonic pitch-class profiles. MelodySim [ 10 ] specifically targets melodic similarity for short audio clips. Beat and rhythm characterisation and instrument classification address complementary factors in

Read the complete text