Adopting State-of-the-Art Pretrained Audio Representations for Music Recommender Systems Journal: TORS Volume: 1 1 1 DOI: 10.1145/3808222 CCS: Information systems Recommender systems CCS: Information systems Music retrieval CCS: Computing methodologies Neural networks
Yan-Martin Tamm, Anna Aljanaki · 2026
Read Adopting State-of-the-Art Pretrained Audio Representations for Music Recommender Systems Journal: TORS Volume: 1 1 1 DOI: 10.1145/3808222 CCS: Information systems Recommender systems CCS: Information systems Music retrieval CCS: Computing methodologies Neural networks by Yan-Martin Tamm, Anna Aljanaki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23077.txt.
Introduction
Tamm and Aljanaki test nine pretrained audio backend models for music recommender systems, a use case music information retrieval papers rarely explore. They test the models with five recommendation methods, including nearest neighbors, a shallow neural network, contrast multi-modal projection, a hybrid model, and BERT4Rec, in hot and cold-start scenarios. Pretrained representations show a large performance gap between old retrieval tasks and music recommendations. The findings suggest backend models capture musical information differently by task, laying groundwork for further study.
Catalog record
| CREATOR | Yan-Martin Tamm, Anna Aljanaki |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77637 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23077
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adopting State-of-the-Art Pretrained Audio Representations for Music Recommender Systems Journal: TORS Volume: 1 1 1 DOI: 10.1145/3808222 CCS: Information systems Recommender systems CCS: Information systems Music retrieval CCS: Computing methodologies Neural networks Yan-Martin Tamm, Anna Aljanaki Abstract . Over the years, Music Information Retrieval (MIR) research community has released various models pretrained on large amounts of music data. Transfer learning showcases the proven effectiveness of pretrained backend models for a broad spectrum of downstream tasks, including auto-tagging and genre classification. However, MIR papers generally do not explore the efficiency of pretrained models for Music Recommender Systems (MRS). In addition, the Recommender Systems community tends to favour traditional end-to-end neural network training. Our research addresses this gap and evaluates the performance of nine pretrained backend models (MusicFM, Music2Vec , MERT, EncodecMAE, Jukebox, MusiCNN, MULE, MuQ and MuQ-MuLan ) in the context of MRS. We assess them using five recommendation approaches: K-Nearest Neighbours (KNN), Shallow Neural Network, Contrastive Multi-Modal projection, a Hybrid model, and BERT4Rec both for the hot and cold-start scenarios. Our findings suggest that pretrained audio representations exhibit significant performance disparity between traditional MIR tasks and both hot and cold music recommendations, indicating that valuable aspects of musical information captured by backend models may differ depending on the task. This study establishes a foundation for further exploration of pretrained audio representations to enhance music recommendation systems. † † cc-license: by 1. Introduction Music Recommender Systems (MRS) are naturally suitable for a hybrid recommendation setting because both collaborative interactions and audio data are equally important, and usually readily available, allowing us to gain deeper insight into user preferences. Apart from performance improvement, hybrid recommender systems also have the potential to address the cold start problem for new items, which is an acute problem for music streaming platforms due to huge catalogue size, short duration of consumption and a long tail of less popular content ( 69 ) . In content-aware MRS, Convolutional Neural Networks (CNN) are commonly trained on Mel-Spectrograms extracted from music segments, as popularized by ( 89 ) . This approach proved highly effective in processing audio and incorporating content information into recommender systems. The Music Information Retrieval (MIR) community has introduced numerous backend models that are pretrained on extensive amounts of music data. Some of these models, such as musiCNN ( 63 ) , are trained in a supervised manner, usually on auto-tagging task, while others, like Jukebox ( 19 ) , are self-supervised. Regardless of the approach, the crucial factor is that these models can be effectively utilized for downstream tasks through transfer learning, yielding results comparable to state-of-the-art models specifically designed for those tasks. One way to do this is to fine-tune all model weights for the target task and dataset, which is computationally costly. The other way is to take the embeddings produced by a backend model and train a separate small network that adapts them to downstream tasks. We refer to these frozen embeddings as pretrained audio representations (PAR). This approach opens up the possibility of using large quantities of unlabeled music data to address problems with limited labelled examples, which could be particularly beneficial in MRS research, where access to large datasets containing both music data and user play history is limited due to copyright. However, pretrained audio representations were not explored within the context of MRS both in MIR and RS scientific communities. The MIR community tends to focus on established tasks and benchmarks that do not include recommendations, while the Recommender Systems (RS) community tends to lean towards traditional end-to-end neural network training over these models. One notable exception is ( 59 ) , where the authors utilized three pretrained encoders: CLMR ( 78 ) , MEE ( 35 ) , and Jukebox ( 19 ) . However, the primary focus of that paper was to study the role of negative preferences in user music tastes, and the use of different pretrained models emphasized the stability of the proposed method rather than being integral to the research. Our paper compares the performance of pretrained embeddings in the context of MRS using nine recently released pretrained backend models. It should be noted that we leave the problem of finding the best architecture for combining content and collaborative information for future research. More specifically, we do not propose a new recommendation model, instead we conduct a wide comparison of pretrained representations. Our goal is to highlight the most promising ones to inspire their usage in MRS research. We outline the following research questions: • RQ1: How do different recommendation approaches compare when initialized with pretrained audio representations? • RQ2: How do different pretrained audio representations compare in the context of MRS? • RQ3: How does the performance of pretrained audio representations in MRS correspond to performance in MIR tasks? In this paper, we extend and improve our previous work on the topic ( 84 ) , adding more recommendation models, additional PARs, and results for the cold-start scenario. The rest of the paper is organized as follows: first, we lay out the approaches used in MRS to incorporate music data into recommendations to explain background for the paper. Next, we describe the experiment and dataset. We continue by introducing recent pretrained models from MIR that will be used to compute audio representations and briefly describe each of them. We describe the recommendation models we used, outline training details 1 1 1 The code is available at github.com/Darel13712/adopting-pretrained-audio , and conclude with results and discussion. 2. Background To start our discussion, we turn our attention to different ways to incorporate audio in MRS that have been used throughout the years. To this end, we take recent reviews on the topic ( 18 ; 71 ; 54 ) covering papers from 2006 to 2022, select all papers that use audio content and study the way it is processed. This funnel produced a total of 45 papers; the full list can be found in the appendix in Table 14 . It is important to note that our overview is only as comprehensive as the reviews we selected, and we can expect some relevant papers to be missing, but we believe it to be a reasonable estimation of the global trend. We group the models into four distinct approaches that use different methods to represent musical data: Low-level Features , Spotify Features , Spectrograms and raw Waveform . We proceed with the description of each group. The distribution of the methods can be seen in Figure 1 . Low-level Features include handcrafted features such as MFCC, Chroma, ZCR; Spotify refers to pre-computed audio descriptors that were available through Spotify API such as danceability and acousticness, Spectrogram refers to image input such as Mel-Scale spectrogram or Constant-Q transform, Waveform refers to using raw audio (time-series of amplitude samples) as an input. Figure 1 . Number of MRS papers using different types of input data to represent audio files per year. 2.1. Audio features and representations 2.1.1. Low-level Acoustic Features These features originate from speech processing and MIR and involve calculating manually crafted statistics and transformations over the input signal to describe its content. We include spectral features and musical characteristics in this category. Spectral features are local short-time