A correlation-permutation approach for speech-music encoders model merging
Fabian Ritter-Gutierrez Nanyang Technological University Singapore [email protected] Yi-Cheng Lin National Taiwan University Taiwan [email protected] Jeremy H.M Wong Institute for Infocomm Research Singapore [email protected] Hung-yi Lee National Taiwan University Taiwan tlkagkb939 · 2025
Read A correlation-permutation approach for speech-music encoders model merging by Fabian Ritter-Gutierrez Nanyang Technological University Singapore [email protected] Yi-Cheng Lin National Taiwan University Taiwan [email protected] Jeremy H.M Wong Institute for Infocomm Research Singapore [email protected] Hung-yi Lee National Taiwan University Taiwan tlkagkb939 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11403.txt.
Introduction
Six researchers merge speech and music encoders into one audio model in a 2025 paper. Training a joint model from scratch is costly, and naive merging fails when the two models' internal weights are not aligned. Their method aligns the music encoder to the speech encoder layer by layer.
A permutation is computed for each layer to maximize feature cross correlation between the models, extending earlier re-basin ideas to Transformer layers. After alignment, the merged model keeps its speech skill and gains sharply on music tasks. Average scores rose by nearly fifteen points over plain weight averaging. The approach builds unified audio models from separately trained parts at low cost.
Catalog record
| CREATOR | Fabian Ritter-Gutierrez Nanyang Technological University Singapore [email protected] Yi-Cheng Lin National Taiwan University Taiwan [email protected] Jeremy H.M Wong Institute for Infocomm Research Singapore [email protected] Hung-yi Lee National Taiwan University Taiwan tlkagkb939 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-11403
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A correlation-permutation approach for speech-music encoders model merging Fabian Ritter-Gutierrez Nanyang Technological University Singapore [email protected] Yi-Cheng Lin National Taiwan University Taiwan [email protected] Jeremy H.M Wong Institute for Infocomm Research Singapore [email protected] Hung-yi Lee National Taiwan University Taiwan [email protected] Eng Siong Chng Nanyang Technological University Singapore [email protected] Nancy F. Chen Institute for Infocomm Research Singapore [email protected] Abstract Creating a unified speech and music model requires expensive pre-training. Model merging can instead create an unified audio model with minimal computational expense. However, direct merging is challenging when the models are not aligned in the weight space. Motivated by Git Re-Basin, we introduce a correlation-permutation approach that aligns a music encoder’s internal layers with a speech encoder. We extend previous work to the case of merging transformer layers. The method computes a permutation matrix that maximizes the model’s features-wise cross-correlations layer by layer, enabling effective fusion of these otherwise disjoint models. The merged model retains speech capabilities through this method while significantly enhancing music performance, achieving an improvement of 14.83 points in average score compared to linear interpolation model merging. This work allows the creation of unified audio models from independently trained encoders. I Introduction Speech and music are two fundamental domains of audio, each with unique characteristics and complexities. Significant advancements have been made in self-supervised learning (SSL) for both areas, with encoders like HuBERT [ 1 ] demonstrating exceptional performance in speech representation and MERT [ 2 ] tailored for music understanding. These models have proven effective in capturing domain-specific features through large-scale pre-training. However, current SSL models often treat speech and music in isolation [ 3 ] . While this benefits domain-specific tasks, the increasing demand for unified audio understanding in applications such as audio large language models necessitates a single, general model capable of processing both domains [ 4 ] . Training such a comprehensive model is computationally expensive [ 5 ] . An alternative to costly pre-training is model merging, which involves combining parameters from different models to create a single model that inherits their capabilities [ 6 , 7 , 8 ] . Model merging has been successfully explored in the speech technology domain for supervised fine-tuning scenarios [ 9 , 10 , 11 , 12 , 3 ] . Nonetheless, there is a key limitation in all previous work on model merging for speech, which is the assumption of starting from a commonly initialized model. Such limitation arises from the requirement that for effective merging, models’ weight spaces must be similar enough to ensure their weights are linearly mode connected [ 13 ] . This assumption is feasible in the case of fine-tuning the same model, as their weight space does not deviate much from the initialized models. Nonetheless, this is not the case for the situation where 2 completely independently-trained models are attempted to be merged. Merging independently trained models, however, is not a trivial task. Naive approaches like simple weight averaging often lead to a significant degradation in performance, as shown in Table I . This is because models trained independently, even with the same architecture, tend to converge to different, symmetrically equivalent minima in the loss landscape [ 13 , 14 , 15 ] . Direct interpolation of weights in such cases can place the merged model in a suboptimal region, leading to a collapse in performance. As highlighted by the ZipIt [ 15 ] framework, these minima are often connected by low-loss paths only when their internal representations, such as neurons or channels, are appropriately permuted and aligned. The present paper addresses the problem of merging independently trained models by maximizing channel-wise cross-correlations between the feature activations of the models to find the best channel alignments between them; such an approach allows the merging of models that were trained with their own loss functions and pre-training data. The foundational works of Git-Re Basin [ 14 ] and Zipit [ 15 ] demonstrated the feasibility of merging independently trained models for MLP-based and CNN-based architectures on the computer vision domain. A subsequent work [ 16 ] provided insights on how transformer [ 17 ] layers for the text domain should be merged when following a permutation framework. While they provided evaluations on loss barriers of transformer layers, they left unaddressed whether such merging can work on downstream tasks. This work aims to close the gap and unify the findings of [ 14 , 15 , 16 ] but for the speech and music domain. Different from the image and text domain, speech/music is a temporal signal with a high frame rate. We then simplify the pipeline so that the framework can be feasible in the speech/music domain. We test this correlation-based permutation approach on pre-trained models. In particular, it is noted that we test on models that were 1) initialized on different random seeds, 2) pre-trained with different pre-training data, and 3) optimized to learn different representation capabilities. Specifically, we focus on HuBERT and MERT as our case study, but this framework can be easily extended to any other models as long as the models to be merged share the same or very similar architecture, such as Whisper variants [ 18 ] . We show that it is possible to merge a MERT encoder and a HuBERT encoder when we do channel-wise permutation of MERT towards HuBERT before performing linear interpolation. Results shown in Table I suggest our method can retain the speech performance of HuBERT while also improving downstream task performance on music tasks. The best performing model is achieved when the channels of the CNN encoder are permuted as well as the attention heads ordering and the channels within each attention head (ID 5 in Table I ). This setup increases the average SUPERB score by 14.83 points. Additionally, ablations on which parts of a model are more suitable to permute are also provided with their respective statistical significance analysis. Fig. 1: Feature‐wise Permutation Framework for Merging HuBERT and MERT. (a) Permutation Computation: For each layer i , compute a permutation \pi_{i}^{*} that maximizes channel‐wise cross‐correlation between MERT’s feature activations (blue) and HuBERT’s feature activations (red). The toy 4\times 4 correlation matrices illustrate raw correlations (Channel‐wise) on the left and reordered, near‐diagonal correlations (Maximized Channel‐wise) on the right. (b) Model Merging: For all layer i\in\mathcal{S} , apply \pi_{i}^{*} to reorder MERT’s weight matrix for layer i and linearly interpolate the permuted MERT weights with HuBERT’s weights, producing a merged layer. (c) Merged Model: The resulting merged encoder (purple) incorporates speech representations from HuBERT and music representations from MERT into a single unified model. II Related Work Early automatic-speech-recognition (ASR) research showed that combining multiple Hidden Markov Model (HMM) based acoustic models improves robustness. The work [ 19 ] proposed to combine observation likelihoods and state transition probabilities. A Bayesian HMM-state merging is proposed by treating the problem via Bayesian model inference to decide which pair of states is best to merge at each step. Similarly, [ 20 ] performs merging of HMM tied-states of a standard triphone-based HMM acoustic model to adapt from standard Chinese to specific dialects. Both works present a way of merging before the usage of neural networks in speech. Their focus is on better modeling of the