MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning
Aurian Quelennec, Pierre Chouteau, Geoffroy Peeters, Slim Essid · 2025
Read MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning by Aurian Quelennec, Pierre Chouteau, Geoffroy Peeters, Slim Essid (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12709.txt.
Introduction
Aurian Quelennec and co-authors improve a self-taught audio model in a 2025 paper. Such models learn sound patterns by hiding part of a clip and guessing the missing inner code. Past work paid little heed to the guessing module, though sound is often ambiguous when many sources mix.
Their upgrade lets the guesser offer several choices instead of one forced pick, which fits mixed sounds better. Tests across many sound tasks gave top scores, both when the model was frozen and when fully tuned on a large labeled set. A version trained only on music also led its field with less compute. Readers get a smarter route to general sound models.
Catalog record
| CREATOR | Aurian Quelennec, Pierre Chouteau, Geoffroy Peeters, Slim Essid |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-12709
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning Aurian Quelennec, Pierre Chouteau, Geoffroy Peeters, Slim Essid Abstract Masked latent prediction has emerged as a leading paradigm in self-supervised learning (SSL), especially for general audio and music representation learning. While recent methods have demonstrated strong performance, the role of the predictor module used at the output of such SSL systems remains mainly overlooked, despite being crucial for solving the pretext task at hand. In particular, this module should be able to deal with the ambiguity inherent in audio content, especially when it is composed of multiple sound sources. This work proposes a novel enhancement: integrating Multiple Choice Learning (MCL) to explicitly model prediction ambiguity and improve representation quality. We build on top of the recently proposed MATPAC system, improving its prediction and unsupervised classification pretext tasks with MCL. We extensively evaluate our method, MATPAC++, through both linear probing across multiple downstream tasks and fine-tuning on AudioSet, employing a unified protocol that enables rigorous and fair comparisons with state-of-the-art SSL approaches. Results show that our proposal achieves state-of-the-art when fine-tuned on AudioSet and overall state-of-the-art scores on downstream tasks. Additionally, we examine domain specialisation by training exclusively on music data, where our model achieves state-of-the-art performance with significantly improved efficiency. 1 Introduction With the rise of llm ( llm ), several multimodal systems have exhibited promising capabilities for solving vision or speech-related tasks [ 1 , 2 , 3 ] . More recently, solutions supporting non-speech audio have been also developed , where so-called alm ( alm ) [ 4 , 5 , 6 , 7 , 8 , 9 ] have attempted to improve the perception and understanding of general audio and music by combining llm s with powerful audio encoders. Therefore, there is a strong interest in building robust audio encoders and clarifying how well they perform before they can be used in multimodal approaches to enhance audio understanding. A powerful approach to achieve such a goal is Self-Supervised Learning ( ssl ( ssl )), which has become a cornerstone in representation learning, across various domains, including general audio and music [ 10 ] , enabling models to learn rich and general representations without requiring ground-truth labels. Among all the existing pretext tasks that have been developed to train ssl methods [ 11 ] mlp ( mlp ) has emerged as a leading paradigm, showing strong performance across different domains. Methods such as I-JEPA [ 12 ] , M2D [ 13 ] , or MATPAC [ 14 ] have demonstrated that learning to predict the latent representation of the masked parts of a partially visible input yields promising results. In these methods, the predictor module plays a central role for the resolution of the mlp pretext task. Figure 1 : met performances on reference datasets compared to leading SSL methods: MATPAC [ 14 ] , M2D [ 13 ] , BEATs Iter3 [ 15 ] , Data2vec [ 16 ] for general audio ssl methods, MERT [ 17 ] for music ssl methods. Best viewed in colour. It is responsible for predicting the latent representations of the masked part of the input, encoded by a teacher model, only from the latent representations of the visible parts of the input, encoded by a student model. However, predicting masked segments of an audio signal, based on visible parts, is inherently ambiguous: there is no guarantee that an audio event will persist across patches, that its “timbre” will remain unchanged, or that new sound sources will not concurrently appear. This makes the prediction task inherently one-to-many: multiple plausible latent completions may exist for a given input context, yet all existing mlp methods output a single one. We therefore propose here a novel enhancement to the mlp framework in order to address this limitation, by introducing a mcl ( mcl ) component [ 18 , 19 ] . mcl provides a simple way to take into account ambiguity by modelling multiple prediction hypotheses and selecting the most appropriate one, using a variant of the so-called wta ( wta ) strategy . This naturally captures the diversity of plausible latent representations of the masked parts, which leads to more robust and semantically richer representations. We implement this in the state-of-the-art ssl audio representation method mat [ 14 ] , resulting in a new model that we call met . Contributions. (i) We introduce met a novel method to solve the ambiguity of the mlp pretext task in self-supervised audio representation learning methods (multiple plausible latent completions may exist for a given input context). This is done by considering multiple output hypotheses in the predictor of the student branch and training using mcl . (ii) We conduct an extensive experimental evaluation of our method and other state-of-the-art ssl methods on diverse tasks, including general audio and music classification tasks . In contrast to previous works, we dedicate a lot of care to the experimental validation, ensuring all models are compared in a fair manner, using exactly the same train/test data splits and careful examination of the statistical significance of the evaluation metrics considered. We also perform a unified fine-tuning evaluation of all models using two distinct settings, revealing the limited reliability of previous evaluations. Overall, our method achieves ssl state-of-the-art results on all tasks, as summarised in Fig. 1 . (iii) We examine domain specialisation by training the models exclusively on music data, again achieving state-of-the-art performance with a lighter model than previous ones. This work marks, to our knowledge, the first use of mcl in general audio representation learning, and we provide insights into how it tackles ambiguity in the mlp pretext task. Model checkpoints and code will be made publicly available. 1 1 1 https://github.com/aurianworld/matpac 2 Related Works Masked Latent Prediction pretext tasks. Among ssl methods that rely on pretext tasks defined in a latent space, mlp ( mlp ) has been shown to be one of the most effective. In this case, a student encodes a masked part of the input and a predictor uses the student’s latent representation to match the teacher’s latent representation of either the full input or the complement of the masked input. Using this paradigm, M2D [ 13 ] for audio and I-JEPA [ 12 ] for images, have achieved a strong performance while increasing the model’s efficiency. Consequently, we adopt the mlp pretext task as one of the training objectives for our model, while addressing its limitation in handling prediction ambiguities. Unsupervised classification pretext tasks. Instead of matching latent representations, one can consider a classification pretext task. In ssl this classification task is unsupervised, hence such an approach requires “pseudo” labels generated by the models themselves or by an external ssl method. Methods such as BEATs and HuBERT, for general audio and speech, exploit “pseudo” labels generated by clustering methods. They need, however, to be trained iteratively to achieve optimal performance, which is a limitation. In contrast, methods developed in the vision domain, such as DINO [ 20 ] , pass different augmentations of an input image to the student and teacher networks, projects them into a probability distribution, and learns to match them through classification. This approach proved to be quite effective. However, choosing