AudioMosaic: Contrastive Masked Audio Representation Learning
Hanxun Huang, Qizhou Wang, Xingjun Ma, Cihang Xie, Christopher Leckie, Sarah Erfani · 2026
Read AudioMosaic: Contrastive Masked Audio Representation Learning by Hanxun Huang, Qizhou Wang, Xingjun Ma, Cihang Xie, Christopher Leckie, Sarah Erfani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14231.txt.
Introduction
Huang and five co-authors introduce AudioMosaic, a contrastive audio encoder for general audio understanding. Self-taught audio learning seeks general-purpose codes from large-scale untagged audio, yet recent progress leans on generative remake while contrastive methods stay less explored. Two hurdles explain the gap: making effective audio changes is hard, and contrastive training demands large batches. It builds positive pairs with structured time-frequency masking on spectrogram patches, cutting memory use for efficient large-batch training.
Catalog record
| CREATOR | Hanxun Huang, Qizhou Wang, Xingjun Ma, Cihang Xie, Christopher Leckie, Sarah Erfani |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50517 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14231
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AudioMosaic: Contrastive Masked Audio Representation Learning Hanxun Huang, Qizhou Wang, Xingjun Ma, Cihang Xie, Christopher Leckie, Sarah Erfani Abstract Audio self-supervised learning (SSL) aims to learn general-purpose representations from large-scale unlabeled audio data. While recent advances have been driven mainly by generative reconstruction objectives, contrastive approaches remain less explored, partly due to the difficulty of designing effective audio augmentations and the large batch sizes required for contrastive pre-training. We introduce AudioMosaic , a contrastive learning–based audio encoder for general audio understanding. During pre-training, AudioMosaic constructs positive pairs by applying structured time–frequency masking to spectrogram patches, which reduces memory usage and enables efficient large-batch training. Compared with generative approaches, the AudioMosaic encoder learns more discriminative utterance-level representations that demonstrate strong transferability across datasets, domains, and acoustic conditions. Extensive experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks under both linear probing and fine-tuning. We further show that integrating the pretrained AudioMosaic encoder into audio–language models improves performance on audio–language tasks. The code is publicly available in our GitHub repository . 1 Introduction Self-supervised learning (SSL) has become a cornerstone of representation learning across modalities, driving advances in natural language processing ( Radford et al. ; Devlin et al. 2019 ) , computer vision ( Chen et al. 2020c ; He et al. 2022 ; Oquab et al. 2024 ; Fan et al. 2025 ) , and audio processing ( Baevski et al. 2020 ; Fei et al. 2023 ; Ahmed et al. 2024 ; Kong et al. 2024 ; Lee et al. 2025 ; Ghosh et al. 2025 ; Goel et al. 2025 ; Dong et al. 2025 ) . Existing audio SSL methods largely fall into two paradigms: masked modeling and contrastive learning. Masked modeling approaches, particularly masked spectrogram modeling, have been extensively explored for general audio understanding ( Niizumi et al. 2022 ; Huang et al. 2022 ; Chong et al. 2023 ; Chen et al. 2024 ; Alex et al. 2025 ) . In contrast, contrastive learning has primarily focused on raw waveform inputs and speech-centric tasks ( Oord et al. 2018 ; Baevski et al. 2020 ; Hsu et al. 2021 ) . Despite its success in vision and language, contrastive learning over spectrogram representations for audio understanding remains relatively underexplored. This gap is not due to a lack of effort, but reflects fundamental challenges in applying contrastive learning to spectrograms. Effective contrastive learning relies on carefully designed data augmentations to generate informative positive pairs ( Wang et al. 2022 ; Zhai et al. 2024 ) . However, augmentation design is highly domain-dependent ( Blankemeier et al. 2023 ; Zhou et al. 2024 ) and often requires expensive search over combinations of transformations ( Chen et al. 2020c ) . Moreover, contrastive methods typically rely on large batch sizes to provide sufficient negative samples, leading to substantial computational cost. Although SpecAugment ( Park et al. 2019 ) is highly effective for supervised learning, it has been shown to be suboptimal for self-supervised masked modeling objectives ( Huang et al. 2022 ) . Together, these observations underscore the difficulty of directly applying existing methods to spectrogram. Figure 1 : Overview of AudioMosaic. (a) Comparison between commonly used unstructured masking in prior audio-SSL methods and the time–frequency masking strategy. (b) AudioMosaic first converts an input spectrogram into patch tokens with positional encoding, then applies time–frequency masking to construct a positive pair for contrastive learning. The masked patches are omitted, and only the visible ones are fed into the encoder in randomized order. In this work, we introduce AudioMosaic , an audio encoder pre-trained with a contrastive objective and a tailored augmentation strategy inspired by SpecAugment. Unlike SpecAugment, which directly masks spectrogram regions with zero values for supervised learning, AudioMosaic operates on spectrogram patches and constructs positive pairs by applying independent masking along the time and frequency dimensions. This produces complementary views of the same utterance, as illustrated in Figure 1 (a). In contrast to the unstructured random masking used in reconstruction-based models to capture local correlations, the structured masking in AudioMosaic is designed specifically for the contrastive setting to encourage the learning of global, discriminative representations. During pre-training, only visible patches are processed by the encoder, and contrastive learning is performed across masked views within a shared embedding space. An overview of the framework is shown in Figure 1 (b). This time–frequency masking strategy differs from prior waveform-based contrastive methods ( Oord et al. 2018 ; Baevski et al. 2020 ) , which focus on contrasting short-term temporal views, and from masked spectrogram modeling approaches ( Huang et al. 2022 ) , which rely on local context for reconstruction. We further analyze why this augmentation strategy is more effective. Spectrograms exhibit strong local correlations along both time and frequency dimensions. For reconstruction-based objectives, unstructured masking that preserves local structure is beneficial because accurate reconstruction relies heavily on local context. In contrastive learning, however, if positive views share too much local structure, the contrastive task becomes overly easy and fails to encourage the learning of informative features. This can lead to dimension collapse ( Jing et al. 2022 ; Huang et al. 2024 ) . Such collapse is characterized by low effective rank ( Roy & Vetterli 2007 ) and degraded representation quality. Structured time–frequency masking alleviates this issue by reducing shared local redundancy between positive views while preserving broader temporal–spectral patterns. By contrasting complementary time–frequency masked views of the same utterance, the model is encouraged to rely on more global structure, leading to the learning of utterance-invariant representations that are more discriminative and transferable across domains. Extensive experiments on standard benchmarks, including AudioSet ( Gemmeke et al. 2017 ) , ESC-50 ( Piczak 2015 ) , Speech Commands ( Warden 2018 ) , and Environmental Sound Deepfake Detection (EnvSDD) ( Yin et al. 2025a ; Yin et al. 2025b ) , show that AudioMosaic achieves state-of-the-art performance on several tasks, while reducing pre-training memory cost and model complexity. In addition, following Gong et al. 2024 , we show that aligning the AudioMosaic encoder with large language models further improves performance on audio–language tasks. In summary, our main contributions are as follows: • We introduce AudioMosaic , a contrastive audio pre-training framework that rethinks masking as a mechanism for constructing informative positive pairs rather than as noise for reconstruction. By using structured, independent time–frequency masking, AudioMosaic creates complementary but non-trivial views that enable effective contrastive learning on spectrograms while remaining computationally efficient. • We provide a representation-level analysis of masking strategies using effective rank, offering insight into how excessive shared local structure in positive pairs can lead to degenerate contrastive solutions. This analysis helps explain why structured time–frequency masking is particularly effective for contrastive learning on spectrograms. • We demonstrate that AudioMosaic learns discriminative utterance-level representations that generalize across datasets, domains, and acoustic conditions, achieving state-of-the-art results on several standard benchmarks and strong performance on others, while also supporting memory efficient large-batch pre-training and improving audio–language models. 2 Relate Work Masked Modeling. Masked modeling has emerged as a popular paradigm for self-supervised pre-training due to its simplicity and effectiveness: by reconstructing masked or missing content, models can learn context-aware representations. The concept was first popularized by masked