Is MixIT Really Unsuitable for Correlated Sources? Exploring MixIT for Unsupervised Pre-training in Music Source Separation
2025
Read Is MixIT Really Unsuitable for Correlated Sources? Exploring MixIT for Unsupervised Pre-training in Music Source Separation (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07631.txt.
Introduction
Kohei Saijo and Yoshiaki Bando ask whether a self-teaching trick written off for music can still prime music separation models, in a 2025 paper. Splitting a song into stems is costly to label, so pre-training on raw music tempts. The trick, MixIT, learns by unmixing mixtures of mixtures, and it presumes the sources stand apart. Music's parts move together, so the field assumed a mismatch.
The authors blame a different flaw: stem labels shift by use, and the model is never told what counts as a part. Their early tests showed MixIT pulling instruments apart anyway, at least in part. They pre-trained a top separation model on wild, unlabeled archive music, then fine-tuned it on a labeled set. The primed model beat one trained from scratch. The pair pitch MixIT as a cheap head start for music work.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-07631
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Is MixIT Really Unsuitable for Correlated Sources? Exploring MixIT for Unsupervised Pre-training in Music Source Separation Abstract In music source separation (MSS), obtaining isolated sources or stems is highly costly, making pre-training on unlabeled data a promising approach. Although source-agnostic unsupervised learning like mixture-invariant training (MixIT) has been explored in general sound separation, they have been largely overlooked in MSS due to its implicit assumption of source independence. We hypothesize, however, that the difficulty of applying MixIT to MSS arises from the ill-posed nature of MSS itself, where stem definitions are application-dependent and models lack explicit knowledge of what should or should not be separated, rather than from high inter-source correlation. While MixIT does not assume any source model and struggles with such ambiguities, our preliminary experiments show that it can still separate instruments to some extent, suggesting its potential for unsupervised pre-training. Motivated by these insights, this study investigates MixIT-based pre-training for MSS. We first pre-train a model on in-the-wild, unlabeled data from the Free Music Archive using MixIT, and then fine-tune it on MUSDB18 with supervision. Using the band-split TF-Locoformer, one of the state-of-the-art MSS models, we demonstrate that MixIT-based pre-training improves the performance over training from scratch. 1 Introduction Music source separation (MSS) has seen remarkable progress in recent years, largely driven by advances in neural networks [ 1 , 2 , 3 , 4 , 5 ] . These models are typically trained using supervised learning with large collections of mixture-source pairs. However, acquiring isolated sources (or stems) for MSS is extremely costly, and publicly available datasets are scarce. For example, the most widely used dataset in the community, MUSDB18 [ 6 ] , contains only 150 tracks, approximately 10 hours for each of the vocals, drums, bass, and other (VDBO) stem. Given the data scarcity in MSS, recent studies have explored leveraging unlabeled data. In [ 7 ] , HuBERT-style pre-training [ 8 ] for MSS, called Pac-HuBERT, has been proposed. It uses cluster labels derived from primitive auditory features such as repetition structure [ 9 ] and melody contour [ 10 ] . By attaching a decoder to the pre-trained Pac-HuBERT encoder and fine-tuning with labeled data, the model outperformed training from scratch. However, its performance heavily depends on the choice of auditory features. In [ 7 ] , the selected primitives were closely tied to vocals and the ”other” stem, leading to notable improvements in their separation. Still, little improvement has been reported for bass and drums in low-resource settings. In general sound separation, several source-agnostic unsupervised methods have been proposed [ 11 , 12 , 13 ] . A representative method is mixture-invariant training (MixIT) [ 14 ] , where the model is trained to separate a mixture of mixtures (MoM), created by adding multiple mixtures together. The model is trained to reconstruct the individual mixtures in the MoM by remixing the separated sources with the optimal permutation (Fig. 1 ). Since the model does not know which mixture each source in the MoM originally belonged to, it needs to separate the sources to reconstruct the original mixtures well. Although MixIT has been shown to be effective for speech and environmental sound separation, it has been argued that MixIT does not perform well in MSS [ 15 , 16 ] . The rationale is that high inter-source correlation in musical mixtures may allow the model to associate sources with specific mixtures, leading the training to fall into a sub-optimal solution where the model separates mixtures directly rather than disentangling individual sources (Fig. 1 bottom). However, such claims are based on qualitative properties, and no quantitative analysis has been reported. Fig. 1: Top : Overview of MixIT. The model is trained to separate sources in an MoM with a mixture reconstruction loss. The model needs to separate each source well since it does not know which source originally belonged to which mixture. Bottom : Behavior of MixIT commonly assumed when inter-source correlation is high. In this case, the model may identify the combinations of the sources by finding a certain shared property among the sources in each mixture, leading to a sub-optimal solution to output the individual mixtures directly. Although this is the common assumption in MSS, we revisit MixIT and investigate its effectiveness for MSS pre-training. We argue that it is premature to conclude without any experimental validation that MixIT is ineffective for MSS solely due to the high inter-source correlation. This is because a non-negligible portion of data may exhibit relatively low inter-source correlation, making it feasible to learn separation from such data. We hypothesize that MixIT’s limited performance in MSS arises from the ill-posed nature of MSS itself, rather than from high inter-source correlation. In MSS, the definition of each stem is often application-dependent, and the model lacks explicit knowledge of what should or should not be separated. For example, the ”other” stem sometimes consists of multiple instruments in the commonly considered VDBO setup, and separating it into multiple sources is penalized by standard intrusive metrics. Consequently, models trained with MixIT may appear to perform poorly, even if they can separate sources to some extent. Indeed, in our preliminary experiments, we have found that the model trained with MixIT on music data somewhat separates sources, although the signal-to-distortion ratio (SDR) remains low. Although MixIT does not assume any source model and struggles with the ambiguities of the stem definition, these preliminary results suggest its potential for pre-training. Taking these considerations into account, this study investigates the effectiveness of the MixIT for pre-training MSS models. We first pre-train the model on a large-scale, unlabeled music dataset and then fine-tune it using labeled data. In the experiments, we demonstrate that fine-tuning the pre-training model yields notable performance gains over training from scratch. In addition, by fine-tuning only the encoder and decoder of an encoder-separator-decoder model, we show that the MixIT-pretrained model already achieves good performance. We also examine the impact of pre-training data scale. We will publish the source code and model weights once the paper is accepted. 2 Related work 2.1 Semi-supervised learning in music source separation One prevailing approach to address the data scarcity is semi-supervised learning, where a model pre-trained on labeled data is used to generate pseudo-labels through source activity detection or separation [ 3 , 17 , 4 ] . Although these methods have shown promise, their effectiveness may heavily depend on the quality of the unlabeled data. Prior work on MSS has relied on in-house collections [ 3 , 4 ] , but we do not have access to such high-quality unlabeled data. While semi-supervised learning can potentially be combined with the unsupervised pre-training, we leave it for future work and focus on pre-training. 2.2 Unsupervised music source separation Neri et al. trained a variational autoencoder (VAE) by maximizing the evidence lower bound (ELBO) [ 18 ] . In [ 16 ] , parametric source model-based MSS is trained with a mixture reconstruction objective [ 16 ] . Despite their promising results, they inherently constrain the model to be based on a VAE or a parametric source model, while recent state-of-the-art MSS models are discriminative models. For the purpose of pre-training, we are interested in a