Rethinking Masking Strategies for Masked Prediction-based Audio Self-supervised Learning

Daisuke Niizumi † ‡ ¶ , Daiki Takeuchi ‡ , Masahiro Yasuda ‡ , Binh Thien Nguyen ‡ , Noboru Harada ‡ , and Nobutaka Ono † · 2026

Read Rethinking Masking Strategies for Masked Prediction-based Audio Self-supervised Learning by Daisuke Niizumi † ‡ ¶ , Daiki Takeuchi ‡ , Masahiro Yasuda ‡ , Binh Thien Nguyen ‡ , Noboru Harada ‡ , and Nobutaka Ono † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23810.txt.

Introduction

Niizumi and co-authors rethink how to mask audio when training self-supervised models. Masked prediction learns from spectrograms with parts hidden. Recent informed masking methods work well but cost much compute. Their method, dispersion-weighted masking, uses the sparse frequency shape of audio and stays light. Tests show a common block masking choice helps event understanding but hurts generalization. Their method eases that trade-off and lifts performance. They offer practical guidance for masking design in audio learning.

Catalog record

CREATORDaisuke Niizumi † ‡ ¶ , Daiki Takeuchi ‡ , Masahiro Yasuda ‡ , Binh Thien Nguyen ‡ , Noboru Harada ‡ , and Nobutaka Ono †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31341
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23810

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking Masking Strategies for Masked Prediction-based Audio Self-supervised Learning Thanks: This work was partially supported by JST Strategic International Collaborative Research Program (SICORP), Grant Number JPMJSC2306, Japan. Daisuke Niizumi † ‡ ¶ , Daiki Takeuchi ‡ , Masahiro Yasuda ‡ , Binh Thien Nguyen ‡ , Noboru Harada ‡ , and Nobutaka Ono † Abstract Since the introduction of Masked Autoencoders, various improvements to masking techniques have been explored. In this paper, we rethink masking strategies for audio representation learning using masked prediction-based self-supervised learning (SSL) on general audio spectrograms. While recent informed masking techniques have attracted attention, we observe that they incur substantial computational overhead. Motivated by this observation, we propose dispersion-weighted masking (DWM), a lightweight masking strategy that leverages the spectral sparsity inherent in the frequency structure of audio content. Our experiments show that inverse block masking, commonly used in recent SSL frameworks, improves audio event understanding performance while introducing a trade-off in generalization. The proposed DWM alleviates these limitations and computational complexity, leading to consistent performance improvements. This work provides practical guidance on masking strategy design for masked prediction-based audio representation learning. I Introduction Masked Autoencoders (MAE) [ 11 ] have had a significant impact on representation learning through masked prediction-based self-supervised learning (SSL), initially in the image domain, and have subsequently influenced various studies on general-purpose audio representations [ 1 , 17 , 12 ] . While the original MAE demonstrated the effectiveness of fully random masking, subsequent work in the image domain showed that more structured strategies can lead to improved representations, such as block masking in BEiT [ 3 ] and inverse block masking (IBM) in data2vec 2.0 [ 2 ] . In addition to masking strategies that rely purely on randomness regardless of data content, several recent studies have explored informed masking approaches [ 13 , 24 ] , which exploit attention patterns or patch-level representations derived from pretrained models or from the model itself during training. In the audio domain, many existing methods largely follow the random masking strategy introduced in MAE, whereas recent SSL approaches have reported performance gains by adopting IBM [ 6 , 29 ] . However, in these studies, the contributions of masking strategies are often entangled with other training components, and evaluations are typically conducted on a limited set of downstream tasks. As a result, the role of masking strategy design in masked prediction-based audio representation learning remains insufficiently understood, especially in terms of generalization performance, leaving room for further investigation into alternative masking strategies. Fig. 1: Masking examples on an audio spectrogram (hint ratio r_{h}=0.0 ). Object-centric masking (SGIM) almost entirely covers audio events (objects), whereas the proposed DWM applies dispersion-weighted random masking based on patch-wise spectral variability, yielding masking patterns that loosely align with object-centric behavior. In this study, we investigate how various masking strategies, ranging from random masking to informed masking, contribute to masked prediction-based representation learning for general audio (e.g., environmental sound, speech, and music) using spectrogram inputs. Using three masked prediction-based SSL frameworks, we conduct a comparative analysis of random masking, IBM, informed masking, and the proposed dispersion-weighted masking (DWM), as illustrated in Fig. 1 . DWM is motivated by the observation that typical audio events do not occupy all frequency components uniformly, resulting in blank or low-energy regions in spectrogram frequency bins. By leveraging this property, DWM assigns higher masking probabilities to regions with larger dispersion, thereby incorporating spectral structure into the masking process. Compared to previous informed masking approaches that require non-negligible computational overhead to identify candidate regions, DWM provides a simple and lightweight algorithm, enabling a spectrogram-friendly and efficient form of informed masking. Our experimental results provide insights into how different masking strategies affect representation learning. While IBM facilitates the learning of representations that are well suited for audio event semantics, it degrades performance on other downstream tasks, resulting in an overall trade-off. Informed masking approaches, on the other hand, incur non-negligible computational costs due to the need to compute masking candidates using the model during training. In contrast, the proposed DWM improves performance on audio event understanding without increasing computational complexity, while suppressing performance degradation on other tasks and preserving generalization performance. The findings of this study, together with the proposed DWM, are applicable to a broad range of masked prediction-based audio representation learning methods and contribute to masking strategy design in the following ways: • We clarify the impact of masking strategies on audio spectrogram-based representation learning, highlighting their effects across different downstream tasks. • We propose DWM, a lightweight masking strategy tailored to audio spectrograms, which leverages dispersion patterns with minimal additional computational cost. We publicly release our code 1 1 1 https://github.com/onolab-tmu/audio_ssl_masking to support reproducibility and to facilitate further advances in masked prediction-based audio representation learning. II Preliminary We introduce the background and prior work necessary to frame our study on masking strategies for masked prediction-based representation learning in this section. II-A Masked prediction-based representation learning for audio spectrograms Masked prediction-based representation learning operates by dividing an input into visible and masked patches, denoted as x_{v} and x_{m} , respectively. An encoder f(\cdot) processes only the visible patches to produce latent representations z_{v}=f(x_{v}) . A predictor g(\cdot) then infers the masked part of the input as \hat{z}_{m}=g(\mathrm{concat}(z_{v},m)) , where m denotes mask tokens concatenated at the masked positions. The form of the prediction target \hat{z}_{m} and the role of the predictor g(\cdot) depend on the specific method. For example, in MAE [ 11 ] , \hat{z}_{m} corresponds to the reconstructed input signal in the masked regions, and g(\cdot) acts as a reconstruction decoder. In Masked Modeling Duo (M2D) [ 19 ] , also used in our experiments, \hat{z}_{m} represents latent representations of the masked regions, and g(\cdot) serves as a predictor in the representation space. For audio spectrograms, MAE-AST [ 1 ] , MSM-MAE [ 17 ] , Audio-MAE [ 12 ] , and CED [ 7 ] are MAE-based methods. Prior to MAE, SSAST [ 10 ] also learns representations using a reconstruction-based objective. In contrast, M2D [ 19 ] , ATST [ 14 ] , EAT [ 6 ] , and AaPE [ 29 ] learn by predicting masked patch representations, while BEATs [ 5 ] predicts tokenized labels. M2D-CLAP [ 20 ] further extends M2D by combining masked prediction with contrastive language-audio pre-training (CLAP), enabling the model to directly learn semantic information from natural language descriptions of audio. Among these methods, most adopt a random masking strategy. More recently, EAT and AaPE have demonstrated performance improvements by leveraging IBM. II-B Masking strategies Among existing masking strategies, we consider random masking, inverse block masking (IBM) [ 2 ] , and self-guided informed masking (SGIM) [ 24 ] in our evaluation. Random Masking In MAE, several masking variants, such as block and grid masking, were investigated, and random masking was found to be the most effective, enabling higher masking ratios while maintaining strong performance. Inverse Block Masking IBM starts with a complete mask over all patches and iteratively restores original content in block-shaped regions, which may overlap, until the number of masked embeddings matches a target masking ratio. As a result, visible patches remain in contiguous block regions; however, the selection of these regions

Read the complete text