On the Robustness of Cover Version Identification Models: A Study Using Cover Versions from YouTube

Simon Hachmeier, Robert Jäschke · 2025

Read On the Robustness of Cover Version Identification Models: A Study Using Cover Versions from YouTube by Simon Hachmeier, Robert Jäschke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-01333.txt.

Introduction

Simon Hachmeier and Robert Jäschke test how well software can spot cover versions of songs in this two thousand twenty-five paper. Systems that find covers score well on standard test sets. Those sets lean on one online database of cover songs. But covers on video platforms like YouTube may differ in ways the test sets miss. The authors sampled songs from YouTube and labeled a subset by hand.

They ran top current models on this new set. The models scored far lower than on the standard community set. Some version types, such as instrumental takes, proved extra hard to rank. The authors also built a taxonomy of the changes found in web covers. Their work warns that lab scores may overstate how these tools do in the wild.

Catalog record

CREATORSimon Hachmeier, Robert Jäschke
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-01333

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Robustness of Cover Version Identification Models: A Study Using Cover Versions from YouTube Simon Hachmeier, Robert Jäschke Abstract Recent advances in cover song identification have shown great success. However, models are usually tested on a fixed set of datasets which are relying on the online cover song database SecondHandSongs. It is unclear how well models perform on cover songs on online video platforms, which might exhibit alterations that are not expected. In this paper, we annotate a subset of songs from YouTube sampled by a multi-modal uncertainty sampling approach and evaluate state-of-the-art models. We find that existing models achieve significantly lower ranking performance on our dataset compared to a community dataset. We additionally measure the performance of different types of versions (e.g., instrumental versions) and find several types that are particularly hard to rank. Lastly, we provide a taxonomy of alterations in cover versions on the web. 1 Introduction In the context of western popular music a cover version is a derivative of an original performance of a musical work. Artists perform versions to convey their subjective interpretations of musical works, which is a long-standing practice in musical culture. Usually, different versions of the same work share similar changes of individual notes (melody) or groups of notes (harmony) over time [ 35 ] . The research field of version identification (VI) deals with the automatic detection of cover versions in music collections. Recent approaches in VI aim to encode versions into representations retaining only relevant information in the context of cover versions [ 9 , 10 , 11 , 20 , 25 , 36 , 37 ] . For instance, Abrassart and Doras [ 1 ] report that melody, harmony and lyrics are generally more relevant than rhythm. However, the actual relevance of each characteristic is non-trivial to predict and might strongly vary for different musical pieces. In contrast, characteristics irrelevant in the VI context are usually well agreed upon; such as the tempo or the key/scale [ 10 , 11 , 25 , 35 , 36 , 39 ] . Online video platforms feature various application scenarios for VI such as copyright infringement detection and music recommendation. Hence, the robustness of methods against noise and variance on the platform is important. One key peculiarity of VI in online videos is the alignment problem. In VI, this was addressed by summarization of musical content along the time axis including pooling mechanisms [ 9 , 11 , 36 , 39 ] and more recently by the matching of smaller chunks of the pairs [ 10 , 25 ] . Since YouTube is a collection of videos rather than versions, 1 1 1 Except for YouTube’s proprietary music streaming service YouTube Music . the relationship between videos and versions is an m -to- n relationship. This makes the alignment problem in online videos particularly challenging. For instance, a video might contain multiple versions (e.g., concert recordings) or only chunks of a version (e.g., guitar solo covers or tutorials [ 19 ] ). Additionally, videos might include noise such as commentary (e.g., people reacting to music [ 28 ] ). Beside the alignment problem, other challenges might arise for VI in online videos such as the absence of the main melody (e.g., karaoke or instrumental versions [ 2 , 33 ] ), low fidelity in amateur recordings and versions occurring only in the background as accompaniment [ 26 ] . VI research has made great progress in recent years, mainly measured in metrics from MIREX 2 2 2 See https://www.music-ir.org/mirex/wiki/2021:Audio_Cover_Song_Identification and reported on community datasets like SHS100K-Test [ 34 ] and Da-Tacos [ 38 ] . However, both of these datasets are based on the platform SecondHandSongs (SHS) 3 3 3 https://secondhandsongs.com/ curated by a community of volunteers 4 4 4 https://secondhandsongs.com/page/About which makes present cover song collections subject to the selection policies of the platform. For example, web covers are considered an individual category of versions characterized by being released non-commercially. 5 5 5 See https://secondhandsongs.com/page/Guidelines/Entities/WebCover At the same time, they appear to be less relevant for collaborators, since the amount of web covers is usually much lower than for commercially released covers as can be seen for the example ‘‘Enter Sandman’’ by Metallica. 6 6 6 See https://secondhandsongs.com/work/6616 What is more, due to a technical limitation of the application interface of SHS, the created datasets do not actually contain web covers . This poses the question whether VI models trained and evaluated on data from SHS are considering all relevant characteristics of versions and motivates our first research question: RQ1 Do cover version datasets based on the platform SecondHandSongs represent the distributions of cover versions and their characteristics on YouTube? We assume that there exists a subset of versions with specific characteristics on YouTube which are relevant in the context of VI but not found on the platform SHS: out-of-distribution data. Consequently, recent VI models are neither trained nor evaluated on data with regard to these characteristics. We therefore propose our second research question: RQ2 Which characteristics of versions drive the uncertainty of existing VI models? In this paper, we aim to explore the success and the challenges of VI on out-of-distribution data. Rather than relying on the cover version collection SecondHandSongs, we leverage the richness of creativity of the YouTube community. Applying a multi-modal uncertainty sampling approach, we identify the most uncertain version candidates. Subsequently, we obtain human annotations by workers on the crowdsourcing platform Mechanical Turk (MTurk). Lastly, two music experts curate a subset of the dataset and provide annotations of uncertainties in the problem context, together with a taxonomy of these. In summary, the main contributions are: • we provide a benchmark dataset SHS-YT 7 7 7 https://github.com/progsi/SHS-YT created with a multi-modal uncertainty sampling approach followed by human annotations. It includes labels on an ordinal scale to reflect the complexity of VI on online video platforms (e.g., videos without musical content and identical audio tracks). • two experts curate the provided dataset to gather insights into uncertainties in the VI context of online video platforms. We also provide a taxonomy extending an existing one [ 35 ] . • our benchmarks show that even the current state-of-the-art model under-performs on our proposed dataset. Additionally, we identify challenging alterations such as the isolation of single instruments or the vocal track which would be better addressed in the field of query-by-humming. This uncovers potential boundaries of cover version definitions. Table 1 : Popular VI benchmark datasets and the seed dataset and our annotated datasets in bold text. Dataset Works Versions Remarks Covers80 [ 12 ] 80 160 Provides raw audio files. Da-Tacos [ 38 ] 1,000 15,000 Provides extracted audio features. Discogs-VI-YT-Test [ 5 ] 9,878 116,197 Based on metadata of Discogs. SHS100K-Test [ 34 ] 1,692 10,547 YouTubeCovers [ 32 ] 50 350 Currently unavailable. SHS-SEED 100 2,404 A subset of versions from [ 34 ] . SHS-YT 100 900 Includes 513 rich annotations. SHS-YT+2Q 100 1,092 SHS-YT with two versions from [ 34 ] per work. SHS-YT+AllQ 100 3,289 SHS-YT with all versions from [ 34 ] per work. 2 Related Work 2.1 Version Identification VI datasets are composed of versions which are grouped by musical

Read the complete text