SLAP: Siamese Language-Audio Pretraining without negative samples for Music Understanding

2025

Read SLAP: Siamese Language-Audio Pretraining without negative samples for Music Understanding (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17815.txt.

Introduction

Julien Guinot, Alain Riou, Elio Quinton, and György Fazekas present a 2025 paper on teaching machines to link music with words without the usual training crutch. Standard multimodal training contrasts matching pairs against many negative examples, which demands huge batches and memory, and still leaves the two modalities sitting apart in the learned space. Their method, SLAP, borrows a self-distillation trick: the model learns by predicting its own representations across the two views, no negatives needed.

The model beat the leading contrastive system on text-music retrieval and zero-shot classification. It also stayed competitive on genre, instrument, and tagging tasks against bigger and even supervised rivals. Its embedding space showed a measurably smaller gap between sound and text, and retrieval held steady as batch sizes shrank. Training can run on a single graphics card. The authors pitch it as a scalable route to music understanding.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-17815

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SLAP: Siamese Language-Audio Pretraining without negative samples for Music Understanding Abstract Joint embedding spaces have significantly advanced music understanding and generation by linking text and audio through multimodal contrastive learning. However, these approaches face large memory requirement limitations due to relying on large batch sizes to effectively utilize negative samples. Further, multimodal joint embedding spaces suffer from a modality gap wherein embeddings from different modalities lie in different manifolds of the embedding space. To address these challenges, we propose Siamese Language-Audio Pretraining (SLAP), a novel multimodal pretraining framework that allows learning powerful representations without negative samples. SLAP adapts the Bootstrap Your Own Latent (BYOL) paradigm for multimodal audio-text training, promoting scalability in training multimodal embedding spaces. We illustrate the ability of our model to learn meaningful relationships between music and text — specifically, we show that SLAP outperforms CLAP on tasks such as text-music retrieval and zero-shot classification. We also observe competitive downstream performance on several MIR tasks, including with larger or supervised models (genre and instrument classification, auto-tagging). Additionally, our approach has attractive properties, such as a quantifiably reduced modality gap and improved robustness to batch size variations on retrieval performance. Finally, its novel formulation unlocks large-scale training on a single GPU through gradient accumulation. 1 Introduction Joint embedding spaces for text and audio have been foundational in recent developments in music understanding and generation. Such spaces are typically learned via Multimodal Contrastive Learning (MCL), which optimizes a pair of encoders for maximal similarity between positive pairs, while minimizing similarity for negative pairs [ 1 , 2 ] . Though widely successful, some drawbacks have been identified with this method. Recent trends for joint multimodal embedding spaces have emphasized fine-grained understanding between individual text tokens and individual timesteps of music [ 3 , 4 ] and need for text augmentation strategies to alleviate the lack of large scale datasets [ 5 , 6 ] . A modality gap emerging from the contrastive approach has been observed, where modalities lie in separate manifolds of the embedding space [ 7 , 8 ] . This leads to “joint” representations not being truly joint, potentially harming performance. Importantly, contrastive approaches face an inherent scalability issue. Their formulation makes them 1) dependent on batch size for representation quality and 2) more difficult to scale than masked modelling approaches due to the formulation of the contrastive loss [ 9 ] . These drawbacks pose issues for foundation models, which by usage should be scalable for adaptation on large-scale datasets. Further, text-music spaces are a many-to-many space. I.e., there is often no one single corresponding appropriate caption for a piece of music, and vice-versa . Inspired by recent advances in Self-Supervised Learning for images and general audio [ 10 , 11 ] , we propose SLAP , S iamese L anguage A udio P retraining. We adapt Bootstrap Your Own Latent (BYOL) [ 10 ] as a joint-embedding pretraining approach without negative samples to multimodal text-audio pretraining. We show that SLAP alleviates both the scalability issues and the modality gap inherent to MCL. Our contributions are: 1. We introduce a scalable, hyperparameter-robust approach to language-audio pretraining which does not require negative pairs to learn strong representations. 2. We outperform comparable contrastive models on retrieval and downstream probing. 3. We show that our approach significantly decreases the modality gap between audio and text embeddings compared to contrastive approaches. 4. Our approach enables larger batch sizes via gradient accumulation, which was previously inaccessible due to the formulation of the contrastive loss. To facilitate further research in this direction, we make our code available. 1 1 1 https://github.com/Pliploop/SLAP 2 Background 2.1 Multimodal Contrastive Learning In contrastive learning, models learn representations by maximizing the similarity between two views of an input while minimizing its similarity with negative samples , typically by optimizing an InfoNCE loss [ 12 ] . While these views are typically crafted by randomly applying transforms to the input data points [ 12 , 13 ] , Multimodal Contrastive Learning (MCL) instead relies on paired data from different modalities, such as text-image [ 1 ] , text-video [ 14 ] , audio-image [ 15 ] , text-music [ 16 , 17 ] , and text-audio [ 2 , 18 ] . Contrastive learning spanning more than two modalities has also been investigated [ 19 , 20 , 21 ] . MCL has been leveraged to great extent in multiple modalities for tasks such as understanding [ 1 ] , retrieval [ 16 ] , and generation [ 22 ] , with many multimodal foundation models emerging from the approach [ 23 , 24 ] . MCL, specifically text-audio and text-music contrastive learning, has also been widely adopted in the field of audio and music. Notably, CLAP [ 2 ] and subsequent developments building upon it have demonstrated high audio understanding performance [ 5 , 3 , 18 , 25 ] . Learned audio-text representations enable effective text-audio and audio-text retrieval by leveraging similarity metrics between modalities. Beyond catalogue navigation, these capabilities support retrieval-augmented audio captioning [ 26 , 27 ] . In music-text representation learning, MusCALL [ 16 ] and MuLan [ 17 ] have both achieved strong performance on downstream tasks such as text-music retrieval, music classification, and zero-shot classification. Finally, joint embedding text-audio spaces have also been leveraged to condition audio and music generative models [ 28 , 29 , 30 , 31 , 32 ] . 2.2 Limitations of contrastive learning Despite its widespread success, contrastive learning faces notable limitations, typically related to its reliance on negative samples. First, ensuring a diverse and representative set of negative samples is crucial for stable training, which usually necessitates a large batch size B in practice. However, computing the InfoNCE loss requires storing a B\times B matrix of pairwise similarities on a single GPU, which limits scalability. SigLIP [ 33 ] addresses this by replacing the softmax in the criterion with a sigmoid, making distributed training more tractable, though it still demands costly inter-device communication. In addition, contrastive learning implicitly assumes a uniform prior for the data distribution, which is detrimental in long-tailed scenarios [ 34 ] . MCL, more specifically, also suffers from an issue often referred to as the modality gap , where embeddings from different modalities occupy non-overlapping cone manifolds in the joint latent space [ 8 ] . [ 35 ] observe a relationship between this gap and the initial model weights and loss temperature, while recent findings suggest this problem could be intrinsic to the loss formulation itself [ 7 ] . Figure 1 : (a) CLAP: The multimodal contrastive loss \mathcal{L}_{\text{CLAP}} is optimized between batches of audio/text pairs. (b) SLAP: For each audio/text pair, we compute a query q and a target \bar{z} for each modality, then optimize both the intermodal losses \mathcal{L}_{A\to T} and \mathcal{L}_{T\to A} and the intramodal losses \mathcal{L}_{A} and \mathcal{L}_{T} between the queries and the targets. As indicated by the stop-gradient operator sg , the gradient flows only through the context branches, while the target encoders are updated via EMA. 2.3 Self-supervised learning without negative samples Recent advances in Self-Supervised Learning (SSL) have demonstrated that contrastive learning can be outperformed by alternative approaches that do not

Read the complete text