Layer-wise Investigation of Large-Scale Self-Supervised Music Representation Models
Yizhi, Haina, Hangting · 2025
Read Layer-wise Investigation of Large-Scale Self-Supervised Music Representation Models by Yizhi, Haina, Hangting (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16306.txt.
Introduction
Yizhi Zhou, Haina Zhu, and Hangting Chen present a 2025 study of what self-supervised music models learn, layer by layer. Such models train on raw music and now aid tasks like tagging and search. Yet what each layer holds, and which to use, was little studied.
The team probed two models, MusicFM and the newer MuQ. They checked the gains of self-supervised models across many tasks. They then mapped how layers specialize: early and late layers carry different kinds of information. Picking the right layer for a task changed scores in clear ways. Their analysis shows the structure inside these models and how to use them better for music tasks.
Catalog record
| CREATOR | Yizhi, Haina, Hangting |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16306
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Layer-wise Investigation of Large-Scale Self-Supervised Music Representation Models Yizhi, Haina, Hangting Abstract Recently, pre-trained models for music information retrieval based on self-supervised learning (SSL) are becoming popular, showing success in various downstream tasks. However, there is limited research on the specific meanings of the encoded information and their applicability. Exploring these aspects can help us better understand their capabilities and limitations, leading to more effective use in downstream tasks. In this study, we analyze the advanced music representation model MusicFM and the newly emerged SSL model MuQ. We focus on three main aspects: (i) validating the advantages of SSL models across multiple downstream tasks, (ii) exploring the specialization of layer-wise information for different tasks, and (iii) comparing performance differences when selecting specific layers. Through this analysis, we reveal insights into the structure and potential applications of SSL models in music information retrieval. Zhou Zhou Chen \dagger NanJing UniversityChina Shanghai Jiao Tong UniversityChina Tencent AI Lab China keywords Self-supervised learning, representation analysis, music information retrieval † † email: [email protected], [email protected], [email protected] † † Work performed during an internship at Tencent AI Lab. \dagger Corresponding Author. 1 Introduction Self-superviesed learning (SSL) has been used to train large fundamental models, including audio signal processing models [ 1 , 2 ] . One of the most significant advantages of SSL models is their ability to leverage unlabeled audio data, enabling the possibility of training with large-scale datasets. Recently, numerous SSL models and methods leveraging SSL models for downstream tasks have been proposed, including those in the field of music information retrieval [ 3 , 4 , 5 ] . However, the features extracted by these models and the information contained within them have not been thoroughly explored, which may increase the difficulty and error rate in developing downstream tasks using these features. We aim to enhance our understanding of the information encoded in pre-trained models by investigating how features evolve across layers, and how these features can be better utilized. In this work, we analyze the recent state-of-the-art music SSL model, MusicFM, and the newly emerged MuQ model. Our main findings are shown as follows: 1. SSL models in Music Information Retrieval (MIR) tasks was demonstrated effective through comprehensive experiments. 2. Feature representations progressively evolve from Acoustic-level to Semantic-level in the model. 2 Related work Music Information Retrieval (MIR) tasks primarily aims to extract useful information from raw audio signals, including Genre classification, Key detection, and so on. However, research in this area has been hindered by the limitations in data access [ 6 ] . To address this challenge, self-supervised methods [ 4 ] have been introduced to the MIR field due to their ability to utilize unlabeled data and establish a unified upstream and downstream architecture [ 5 ] , including wav2vec [ 7 , 8 ] that based on Contrastive Learning, MERT [ 1 ] that adapted Hu-BERT [ 9 ] to conduct a framework for music representation. More recently, researchers introduced MusicFM [ 2 ] and MuQ [ 10 ] base on BEST-RQ [ 11 ] , proving its generalization ability on sequence-level and frame-level MIR tasks. Some research has been dedicated to analyzing the reasons behind the high performance of speech SSL models [ 12 ] , and there are studies that have conducted layer-wise analysis on speech SSL models of feature information changes and the performance differences in downstream speech tasks [ 13 ] . Additionally, recent studies have comprehensively evaluated the performance of various mainstream SSL models on speech tasks [ 14 ] . We utilize MARBLE benchmark [ 15 ] to analyze music SSL models, including comprehensive and layer-wise analysis. To our knowledge, this is the first work to analyze music SSL models on the layer-wise granularity. 3 Analysis Methods and Details Figure 1 : Model Framework, illustrates the paradigm of using SSL models for downstream tasks. 3.1 Foundation Model The foundation models are pre-trained on a large amount of unlabeled data to learn high-dimensional representations of audio. It aims to compress and denoise long raw audio signals, retaining as much information in signals as possible. The training objective is to predict tokens that have been randomly masked within a sequence. similar to BERT [ 16 ] . In audio processing, authors used a variety of feature extractors to generate the tokens for training. MusicFM use random projection quantizer as feature extractor following the settings in BEST-RQ . To further explore the shared characteristics of the SSL models, we also used a SSL model called MuQ with the same structure [ 10 ] . The training labels for this model were tokens extracted from Mel Residual Vector Quantization (Mel-RVQ), which quantizes the Mel spectrum using a linear RVQ [ 17 ] . After the pre-training, the models are considered capable of extracting features from previously unseen music audio. During evaluating, the Mel spectrum is extracted from raw audios, resampled, then fed into multiple layers of encoders for context learning as in ( 1 ),( 2 ). At each layer i from 1 to L ( L is the number of encoder layers), we can obtain sequence embeddings z_{i} for audio signals. These embeddings represent the features of the music. Then we can use different methods to aggregate the features as the input of downstream models. The entire process is illustrated in the Figure 1 . \boldsymbol{h_{0}}=Mel(\boldsymbol{x}) (1) \boldsymbol{h_{i}}=Encoder_{i}(h_{i-1})+\boldsymbol{h_{i-1}}\quad i=1,2,...,L (2) In the evaluating procedure, to demonstrate the representational ability of the extracted features, we employ a simple probing model to perform downstream tasks, which is consistent with prior work [ 4 ] . This probing model is structured as a shallow neural network with a single 512-dimensional hidden layer and an output layer. The model’s parameters are frozen during the downstream training. 3.2 Feature extractor Details We adopted the following settings for our testing approach: Baseline feature We concatenate FBANK and Chroma representation as the baseline for evaluating due to their wide adoption in audio and music related tasks. FBANK features represent the energy of different frequency bands. We use 80-dimensional Mel filter bank as FBANK features, with a frame length of 25ms and an overlap of 10ms; Chroma features represent the energy distribution of different pitch classes (like the 12 notes in Western music) regardless of the octave. Chroma features are particularly useful to analyze harmonic content. MusicFM We use the checkpoint pre-trained on the MSD dataset [ 18 ] with 330M parameters, which is publicly available. The embeddings’ hidden dim d=1024 , token sample rate is 25K Hz. MuQ The MuQ model consists of 12 layers of Conformer, with 310M parameters totally. Mel-RVQ takes the Mel-spectrogram ( d=128 ) of the audio as input and outputs the tokens to be predicted. The embeddings’ hidden dim d=1024 , token sample rate is 25K Hz. 3.3 Canonical Correlation Analysis Canonical Correlation Analysis (CCA) [ 19 ] is a statistical technique for relating two sets of observations arising from an underlying process. CCA has been used as a method to compare representations within and across neural network models [ 20 , 21 ] . Specifically, we use the variant PWCCA [ 22 ] , which is a improved CCA method that computes a weighted mean of the