Unsupervised Evaluation of Deep Audio Embeddings for Music Structure Analysis
Axel Marmoret ( [email protected] )( 0000-0001-6928-7490 ) · 2026
Read Unsupervised Evaluation of Deep Audio Embeddings for Music Structure Analysis by Axel Marmoret ( [email protected] )( 0000-0001-6928-7490 ) (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27218.txt.
Introduction
Marmoret tests whether generic deep audio embeddings can reveal musical structure without labels. Music structure analysis is usually taught, but labels are scarce and section edges are ambiguous. Nine open pre-trained models supply bar-level embeddings for the test. Three unsupervised segmentation methods then hunt for section boundaries. Modern ones beat spectrogram baselines, though not in every case. Correlation block matching proved the strongest segmentation method overall. The unsupervised pipeline also topped recent linear probing baselines. Marmoret urges trimming labels to stop padding standard metrics.
Catalog record
| CREATOR | Axel Marmoret ( [email protected] )( 0000-0001-6928-7490 ) |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35203 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.27218
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unsupervised Evaluation of Deep Audio Embeddings for Music Structure Analysis Axel Marmoret ( [email protected] )( 0000-0001-6928-7490 ) Abstract Music Structure Analysis (MSA) aims to uncover the high-level organization of musical pieces. State-of-the-art methods are often based on supervised deep learning, but these methods are bottlenecked by the need for heavily annotated data and inherent structural ambiguities. In this paper, we propose an unsupervised evaluation of nine open-source, generic pre-trained deep audio models, on MSA. For each model, we extract barwise embeddings and segment them using three unsupervised segmentation algorithms (Foote’s checkerboard kernels, spectral clustering, and Correlation Block-Matching (CBM)), focusing exclusively on boundary retrieval. Our results demonstrate that modern, generic deep embeddings generally outperform traditional spectrogram-based baselines, but not systematically. Furthermore, our unsupervised boundary estimation methodology generally yields stronger performance than recent linear probing baselines. Among the evaluated techniques, the CBM algorithm consistently emerges as the most effective downstream segmentation method. Finally, we highlight the artificial inflation of standard evaluation metrics and advocate for the systematic adoption of “trimming”, or even “double trimming” annotations to establish more rigorous MSA evaluation standards. 1 Introduction Beyond just sound, music is an intricate organization of scales and rhythms. At its core, musical composition relies on repetition, contrast, and variation to organize sound into coherent forms. These elements create an internal logic that dictates a song’s shape, establishing recognizable sections separated by distinct transitions. Music Structure Analysis (MSA) is the task of identifying this high-level organization. Specifically, it aims to partition a musical piece into meaningful, non-overlapping sections ( e.g. , intro, verse, chorus, or bridge), and in particular locating their precise temporal boundaries [ 20 ] . Beyond its established value in guiding applications like music summarization, cover song identification, and data-driven musicology, we believe that MSA will find renewed significance in the era of generative AI. By providing a structural framework, accurate MSA can enhance the editability, user control, and long-term coherence of AI-generated audio [ 17 , 5 ] . Yet, despite its practical importance, MSA remains a notoriously challenging task: musical form is inherently hierarchical, and multiple structural interpretations can be simultaneously valid depending on the listener’s focus, the annotation protocol, and the desired level of granularity. Early work on MSA focused on self-similarity matrices (SSMs) [ 20 , 6 ] , i.e. square matrices indicating the pairwise similarity of all time instances in a song. From this viewpoint, sections are often understood as large, homogeneous and/or repeating regions, and boundaries as salient ruptures between subsequent sections. Comparing time instances to detect change points was the root of MSA for years, with significant research in the previous decade dedicated to refining the similarity computation between these instances [ 27 , 16 , 14 ] . This line of work was significantly advanced by the rise of deep learning, particularly representation learning [ 15 , 26 , 30 , 3 ] . Researchers have begun to use deep models in conjunction with standard downstream self-similarity segmentation algorithms to improve audio understanding. For instance, Salamon et al. [ 26 ] used deep embeddings to replace the traditional acoustic features in [ 16 ] , improving the performance. Parallel to these developments, supervised deep learning models [ 9 , 11 , 4 ] emerged as a dominant approach, explicitly learning to predict boundaries from labeled data and achieving state-of-the-art performance in MSA. However, this success comes with significant trade-offs. Supervised models rely heavily on large, meticulously annotated datasets, which are time-consuming and costly to produce. Furthermore, the inherent ambiguity of musical structure becomes a major bottleneck here: supervised training often forces the model to bias its estimates towards a single “ground truth” annotation protocol. This effectively discards other potentially valid, equivalent structural interpretations and limits the model’s ability to generalize across different musical distributions. Recent Self-Supervised Learning (SSL) paradigms offer a promising path to alleviate these issues. By leveraging pretext tasks (such as Masked Acoustic Modeling, where random masks are applied to segments of the input signal and the model is optimized to reconstruct the corrupted regions), these networks bypass the annotation bottleneck and learn directly from massive, unlabeled audio corpora. SSL dedicated to MSA has been shown to be effective in recent work [ 3 ] . However, an intriguing open question remains: do broad, general-purpose audio representations naturally encode structure-related information as a byproduct of their pre-training, even without being explicitly designed for structure? To succeed in their pre-training tasks, these generic SSL models must intrinsically capture the underlying structural, timbral, and temporal characteristics of acoustic signals. Recent work by Toyama et al. [ 29 ] explored whether generic deep audio models (SSL-based but not only) could benefit MSA. However, their study evaluated learned representations using linear probing, which involves training a supervised linear head on top of the frozen embeddings to estimate structure. While computationally efficient, linear probing remains fundamentally supervised. Consequently, it remains sensitive to the manual biases that limit fully supervised models. Our work shares the fundamental premise of Toyama et al. [ 29 ] , but we employ strictly unsupervised downstream segmentation. In this paper, we study unsupervised boundary estimation using frozen embeddings extracted from nine open-source, generic, pre-trained audio models [ 10 , 13 , 31 , 33 , 21 , 24 , 12 , 22 , 32 ] . Following our previous work [ 14 ] , these embeddings are computed at the bar scale. We evaluate these embeddings by computing barwise SSMs, and segment them using three standard and state-of-the-art unsupervised segmentation algorithms: Foote’s checkerboard kernels [ 6 ] , spectral clustering (LSD) [ 16 ] , and the Correlation Block-Matching (CBM) algorithm [ 14 ] . We focus on the boundary retrieval task only, and do not attempt to label the sections. Our goal is two-fold: on one hand, to assess to what extent modern deep audio embeddings are rich enough to provide structured information about the data, and in turn improve unsupervised MSA; on the other hand, to compare the potential of traditional unsupervised segmentation algorithms to be improved when complemented with deep representation learning. The remainder of this paper details our methodology for extracting and segmenting barwise embeddings (Section 2 ), outlines the experimental setup and evaluation metrics (Section 3 ), discusses our comparative results (Section 4 ), and concludes (Section 5 ). 2 Methodology 2.1 Deep Audio Models To evaluate the structural information captured by modern representation learning, we extract embeddings from nine distinct open-source deep audio models [ 10 , 13 , 31 , 33 , 32 , 21 , 24 , 12 , 22 ] . These deep audio models differ significantly in their architectures and training objectives. Three of these nine models (MERT [ 13 ] , MusicFM [ 31 ] , and MuQ [ 33 ] ) are only trained on music data, and their respective authors explicitly noted their potential utility for MSA. The majority of the evaluated architectures rely on Masked Acoustic Modeling, an SSL paradigm where the network optimizes the reconstruction of masked portions of the audio signal. Within this category, AudioMAE [ 10 ] learns by reconstructing masked spectrogram patches. Meanwhile, MERT [ 13 ] , MusicFM [ 31 ] , and MuQ