MIRFLEX: Music Information Retrieval Feature Library for Extraction
2024
Read MIRFLEX: Music Information Retrieval Feature Library for Extraction (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00469.txt.
Description
Abstract This paper introduces an extendable modular system that compiles a range of music feature extraction models to aid music information retrieval research. The features include musical elements like key, downbeats, and genre, as well as audio characteristics like instrument recognition, vocals/instrumental classification, and vocals gender detection. The integrated models are state-of-the-art or latest open-source.
Catalog record
| YEAR | 2024 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 9 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-00469
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MIRFLEX: Music Information Retrieval Feature Library for Extraction Abstract This paper introduces an extendable modular system that compiles a range of music feature extraction models to aid music information retrieval research. The features include musical elements like key, downbeats, and genre, as well as audio characteristics like instrument recognition, vocals/instrumental classification, and vocals gender detection. The integrated models are state-of-the-art or latest open-source. The features can be extracted as latent or post-processed labels, enabling integration into music applications such as generative music, recommendation, and playlist generation. The modular design allows easy integration of newly developed systems, making it a good benchmarking and comparison tool. This versatile toolkit supports the research community in developing innovative solutions by providing concrete musical features. 1 Introduction Music Information Retrieval (MIR) is a complex field focused on computational analysis and processing of musical data, with tasks like similarity estimation, genre classification, and recommendation. While recent advances in machine learning have led to powerful feature extraction methods, the fragmented nature of these tools poses challenges for researchers who must integrate multiple disparate systems. To address this, we present MIRFLEX, a unified feature extraction library designed for MIR research. MIRFLEX offers a diverse set of extractors covering key musical aspects such as key, beats, and genre, using both signal processing and machine learning techniques to generate comprehensive audio representations. The primary objectives of this work are threefold: 1. To offer a centralized and easily accessible collection of feature extraction tools, reducing the burden on researchers to implement and integrate disparate feature extraction techniques. 2. To provide a comprehensive feature set that captures the multifaceted nature of musical data, enabling researchers to explore a wide range of music-related applications and queries. 3. To contribute to the advancement of music information retrieval research by facilitating the rapid prototyping and development of new applications that leverage easily accessible and readily available musical features. The proposed feature extraction library is available at 1 1 1 https://github.com/AMAAI-Lab/megamusicaps 2 Integrated Feature Extractors Our feature extractor comprises of the following features. We also detail our selection process for the exact approach for feature extraction: 2.1 Key Detection We consider following approaches for key detection, Inception Key Net and CNNs with Directional Filters. Inception Key Net [ 1 ] adapts the Inception V3 architecture and uses the Constant-Q transform to convert the audio to a time-frequency representation. Achieves state-of-the-art performance. CNNs with Directional Filters [ 2 ] approach compares shallow, domain-specific architectures with directional filters to deep VGG -style architectures with square filters, using constant-Q magnitude spectrograms. Approach Dataset Accuracy(%) InceptionKeyNet [ 1 ] GiantSteps [ 3 ] 75.5 CNNs with Directional Filters [ 2 ] GiantSteps [ 3 ] 67.9 Table 1 : Key Detection Extractor candidates. In spite of the performance in Table 1 , [ 1 ] does not have model weights publicly available. We integrate [ 2 ] using weights they provided 2 2 2 https://github.com/hendriks73/key-cnn . 2.2 Chord Detection We consider the following options for chord detection: Bidirectional Transformers Uses a self-attention mechanism to focus on regions of chords [ 4 ] . Self-attention is applied forward (on preceding frames) and backward (on succeeding frames) in parallel. It achieves a weighted chord symbol recall (WCSR) score of 83.9 for the Root chord and 83.1 for the maj-min label. CNN-MCTC with HMM [ 5 ] combines Convolutional Neural Networks with Multi-Class Temporal Classification and Hidden Markov Models. It can be used for both chord recognition and local key estimation. Fully Convolutional Networks with CRF [ 6 ] uses a fully convolutional deep auditory model for feature extraction, followed by CRF for decoding to output the chord sequence. d. Semi-Supervised Jointly Trainable CNN [ 7 ] combines CNN with a semi-supervised learning approach , that can jointly train on labelled and unlabelled data to improve chord estimation accuracy. Overall, we select the Bidirectional Transformer as it achieves the best performance. Approach Dataset Accuracy Bidirectional Transformer [ 4 ] Isophonics Subset + Robbie Williams 83.9 (WCSR) CNN-MCTC with HMM [ 5 ] Schubert Winterreise Dataset 0.818 (F1 Score) Fully Convolutional Network with CRF [ 6 ] Isophonics Subset 82.9 (WCSR) Semi-supervised Jointly Trainable CNN [ 7 ] Isophonics + RWC + uspop2002 + McGill Billboard 82.8 (Accuracy) Table 2 : Chord transcription candidates. 2.3 Down-beat Transcription / Tempo Estimation We consider the following approaches for the Downbeat transcription/tempo estimation. CNNs with Directional Filters [ 2 ] uses Convolutional Neural Networks, with directional convolutional kernels instead of square ones. Single-Step Tempo Estimation CNN [ 8 ] frames tempo estimation as a multi-class classification problem, and uses conventional Convolutional Neural Network for the architecture . It can be used for audio clips of 11.9s and therefore is suitable for both local and global tempo estimation. 1D State Space HMM [ 9 ] utilizes a 1D state space and a semi-Markov model for music rhythmic analysis. This approach can reduce the computation cost and provides 30 times speedup in processing. BeatNet: CRNN and Particle Filtering [ 10 ] combines Convolutional-Recurrent Neural Networks with particle filtering for online joint beat, downbeat and meter tracking . It can achieve real-time processing, with high accuracy, albeit computationally expensive. Based on its performance, we choose BeatNet. Approach Dataset Accuracy (%) CNNs with Directional Filters [ 2 ] GiantSteps [ 3 ] 88.7 Single Step Tempo Estimation CNN [ 8 ] GiantSteps [ 3 ] 86.4 1D State Space HMM [ 9 ] GTZAN [ 11 ] 76.48 (F-measure) BeatNet: CRNN and Particle Filtering [ 10 ] GTZAN 80.64 (F-measure) Table 3 : Downbeat transcription/tempo estimation. 2.4 Vocals / Instrumental Detection The EfficientNet model, trained on Discogs, is used for instrument/vocals and vocals gender detection. The implementation and weights are from the Essentia library [ 12 ] . We compare this to the vocals & gender detection implementation available at 3 3 3 https://github.com/x4nth055/gender-recognition-by-voice . Based on results, we choose Discogs-Effnet for vocals / instrumental detection. 2.5 Instrument, Mood / Theme, Genre Detection Not many approaches open-source weights and implementations, we are not able to use any major latest model available. Consequently, the feature extraction system incorporates available weights and techniques implemented in the Essentia library. The Essentia library employs an array of Convolutional Neural Networks as the model architecture, utilizing the Jamendo baseline for the tasks of instrument detection, mood/theme detection, and genre detection. Approach Dataset F1 Score Number of instruments CNN [ 13 ] Philharmonica + UIowa-MIS 0.98 20 Decoupled CNN per instrument [ 14 ] Slakh 0.93 4 SVM classifier w 10-fold cross-validation [ 15 ] IRMAS 0.81 6 U-Net Reprogramming [ 16 ] OpenMIC 0.81 20 Attention-based Embedding Network [ 17 ] OpenMIC 0.81 20 Leveraging Hierarchical Structures for few-shot detection [ 18 ] MedleyDB 0.81 24 Table 4 : Instrument detection candidates. Approach Dataset Accuracy Frequency-Aware RF-Regularized CNNs [ 19 ] MTG-Jamendo [ 20 ] 0.1546 Noisy student training and Harmonic Pitch Class profiles [ 21 ] MTG-Jamendo [ 20 ] 0.1356 Table 5 : Mood / Emotion Detection candidates. 3 Call To Contribute MIRFLEX is an extendable toolkit freely available on GitHub 4 4 4 https://github.com/AMAAI-Lab/megamusicaps . Moreover, we invite the research community to contribute new feature extractors or extend the existing ones. By providing