Leave-One-EquiVariant: Alleviating invariance-related information loss in contrastive music representations

Julien Guinot*, Elio Quinton, György Fazekas · 2024

Read Leave-One-EquiVariant: Alleviating invariance-related information loss in contrastive music representations by Julien Guinot*, Elio Quinton, György Fazekas (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18955.txt.

Introduction

Julien Guinot, Elio Quinton, and Gyorgy Fazekas propose a learning framework in 2024 for music data. Contrastive learning trains models by comparing altered versions of the same track. The alterations teach the model to ignore certain traits.

That ignoring can hurt tasks that need those very traits. Their framework, called LOEV, keeps information about all but one alteration type. A variant builds a split latent space by design. Tests show better results on alteration-based tasks and search, with no loss in general quality. The approach lets one model serve tasks with different needs.

Catalog record

CREATORJulien Guinot*, Elio Quinton, György Fazekas
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18955

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leave-One-EquiVariant: Alleviating invariance-related information loss in contrastive music representations Julien Guinot*, Elio Quinton, György Fazekas Abstract Contrastive learning has proven effective in self-supervised musical representation learning, particularly for Music Information Retrieval (MIR) tasks. However, reliance on augmentation chains for contrastive view generation and the resulting learnt invariances pose challenges when different downstream tasks require sensitivity to certain musical attributes. To address this, we propose the Leave One EquiVariant (LOEV) framework, which introduces a flexible, task-adaptive approach compared to previous work by selectively preserving information about specific augmentations, allowing the model to maintain task-relevant equivariances. We demonstrate that LOEV alleviates information loss related to learned invariances, improving performance on augmentation related tasks and retrieval without sacrificing general representation quality. Furthermore, we introduce a variant of LOEV, LOEV++, which builds a disentangled latent space by design in a self-supervised manner, and enables targeted retrieval based on augmentation related attributes. I Introduction, Related work Contrastive learning is a powerful paradigm for learning self-supervised representations. These representations have been proven to be effective on a range of downstream tasks, including in MIR. Since its first adaptation from computer vision [ 1 ] by Spijkervet et al. [ 2 ] , contrastive learning of musical representations has been successfully repurposed with different architectures [ 3 , 4 ] , notably MULE [ 5 ] ), and positive mining strategies [ 6 , 7 , 8 , 9 , 10 ] . Despite its success, the performance of contrastively learned representations on downstream tasks is sensitive to the positive mining strategy and the augmentation chain. Training a contrastive model involves maximizing (resp. minimizing) agreement between positive (resp. negative) augmented samples. Thus, the choice of positive sampling strategy and the augmentations the model learns to “ignore” significantly impact the usefulness of learnt representations w.r.t. downstream tasks. For example, in music, genre is transposition invariant: transposing a piece of music does not change its genre. So, including pitch shifting in contrastive training should benefit genre recognition by teaching the model to recognise similar representations across different keys. However, this would harm performance on tasks such as pitch detection, chord recognition or key detection, which rely on key-dependent features. Incorporating an augmentation in training renders representations invariant to this transformation, which may or may not be useful depending on the task. Furthermore, the positive sampling strategy implicitly guides the notion of similarity. That is, a contrastive model trained in this framework is never truly generic, and learnt invariances and similarities might not be task-appropriate for downstream applications. In practice and to the best of our knowledge, there exists no unified understanding or theory of how the augmentation chain and the positive mining strategy influence downstream performance or each other, nor any one-size-fits-all recipe for contrastive learning approaches. Some works have attempted to alleviate these issues by devising better positive mining strategies [ 11 ] , reducing false negatives within the mining strategy [ 12 , 13 ] , or influencing positivity and negativity with semantic weighing [ 8 , 11 , 14 ] . From the standpoint of the augmentation chain, a body of work on understanding the effect of certain augmentations on downstream tasks exists [ 15 , 16 ] , and similar studies have started to appear in MIR [ 17 ] . Information-theoretical work has attempted to understand the interplay of positive mining and augmentation chains in training contrastive models [ 15 ] . One work of particular interest to this study, Leave One Out Contrastive (LOOC) [ 16 ] , proposes an approach that does not discard information related to pre-training augmentations in representations. Our work adapts this paradigm to MIR by alleviating learnt invariances pertaining to musical pitch and tempo. Briefly, our contributions are as follows. Fig. 1 : Leave One EquiVariant framework. Subspace \mathcal{Z}^{k} is invariant to all transformations except T_{k} , forcing the embedding superspace \mathcal{V} to conserve information about all transformations. • We introduce Leave One EquiVariant (LOEV) and its variant LOEV++, an adaptation of the Leave One Out Contrastive (LOOC) framework, to mitigate invariance-related information loss with greater flexibility. • LOEV(++) enhances performance on augmentation-related tasks and retrieval without compromising general representation quality. • LOEV++ creates a disentangled latent space, enabling targeted retrieval and controlled storage of augmentation information. • Unlike [ 16 ] , where links between augmentations and downstream task are unclear, we demonstrate that LOEV(++) reduces performance loss for explicitly linked augmentations and MIR tasks. II Methods II-A Contrastive embedding space In the traditional contrastive learning framework, consider a batch of non-augmented samples \{\tilde{x}_{i}\},i\in[1...N] . These samples are augmented through an ordered stochastic chain of K nuclear augmentations \{T_{k}\},k\in[1...K] . These nuclear augmentations have a probability p_{k} of being applied. Parameters for each T_{k} are randomised for each augmented sample. Non-augmented samples \mathbf{\tilde{x}} are augmented M times into samples \textbf{x}_{i}=\{x_{i}\},i\in[1...NM] . The set of positives P(i) for a given index i is the set of samples originating from the same original sample x_{i} . The contrastive objective is given by the following loss for a sample i : \mathcal{L}_{i}=\frac{-1}{|P(i)|}\sum_{p\in P(i)}\frac{\exp(sim(z_{i},z_{p})/\tau)}{\sum_{j\neq i}\exp(sim(z_{i},z_{j})/\tau)} (1) z_{i} is the output of an encoder E:x\mapsto e\in\mathbb{R}^{d_{E}} and a projection head g:e\mapsto z\in\mathcal{Z}^{i}\subset\mathbb{R}^{d_{g}} , \tau is a temperature hyperparameter. In this framework, the network is taught to be invariant to all transformations (all-invariant) by mapping augmented samples to one embedding. We propose a novel, ad-hoc augmentation tracking framework that can be added to existing contrastive learning pipelines where the model maintains information about all augmentations of interest - L eave O ne E qui V ariant (LOEV). In addition to an all-invariant projection head g^{i} into an embedding space \mathcal{Z}^{i} , K projection heads g_{k} project the hidden representation e\in\mathcal{V} onto K spaces \{\mathcal{Z}^{k}\} . representations in \mathcal{Z}^{k} are invariant to all transformations except T_{k} . For each subspace to maintain variance w.r.t a transformation, the set of positives P_{k}(i) for sample i w.r.t T_{k} is samples from the same anchor that have not been augmented with T_{k} . This way, each projection space is explicitly taught to be variant to one augmentation, and the embedding space \mathcal{V} must contain information about all transformations to minimize all objectives. We employ a set of base augmentations \mathcal{T}_{B} without a dedicated subspace (i.e. the model learns to be invariant to them in all subspaces) and variant augmentations \mathcal{T}_{V} which each have a dedicated variant subspace. The contrastive objective for variant subspace \mathcal{Z}^{k} is: \mathcal{L}_{i}^{k}=\frac{-1}{|P_{k}(i)|}\sum_{p\in P_{k}(i)}\frac{\exp(sim(z_{i},z_{p})/\tau)}{\sum_{j\neq i}\exp(sim(z_{i},z_{j})/\tau)} (2) The global loss is then averaged over all subspaces: \mathcal{L}_{i}^{T}=\frac{1}{K+1}\Bigg(\mathcal{L}_{i}+\sum_{k=0}^{K}\mathcal{L}_{i}^{k}\Bigg) (3) II-B View generation To generate positive views, Leave-one-out contrastive (LOOC) [ 16 ] , keeps the clean anchor as a query; After generating an augmented sample, parameters from each nuclear augmentation except one per view are copied across K additional views. Our framework is similar in spirit but more flexible, employing an ad-hoc positive tracking method as well as a simCLR [ 1 ] contrastive pipeline versus MoCo [ 18 ] in [ 16 ] . Instead of always applying all augmentations but one for K augmentations, which induces some inflexibility to the implementation, we maintain our stochastic augmentation chain and simply track which augmentations are applied to each sample i within a binary vector t_{i}\in\{0,1\}^{K} (

Read the complete text