Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalization
Tzu-Quan Lin, Wei-Ping Huang, Hao Tang, Hung-yi Lee · 2025
Read Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalization by Tzu-Quan Lin, Wei-Ping Huang, Hao Tang, Hung-yi Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12672.txt.
Introduction
Tzu-Quan Lin, Wei-Ping Huang, Hao Tang, and Hung-yi Lee address in 2025 a common frustration with speech models: fine-tuning one for a single task often makes it worse at every other task. The tuning shifts the model's internal features so far that knowledge from pre-training is lost.
Their method, Speech-FT, fine-tunes in two stages designed to keep the tuned model close to the original one's features. Earlier fixes, such as limiting weight changes, did not keep features similar enough. By protecting that similarity, Speech-FT aims to gain skill on the target task while keeping broad ability across speech tasks.
Catalog record
| CREATOR | Tzu-Quan Lin, Wei-Ping Huang, Hao Tang, Hung-yi Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12672
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalization Tzu-Quan Lin, Wei-Ping Huang, Hao Tang, Hung-yi Lee Abstract Fine-tuning speech representation models can enhance performance on specific tasks but often compromises their cross-task generalization ability. This degradation is often caused by excessive changes in the representations, making it difficult to retain information learned during pre-training. Existing approaches, such as regularizing weight changes during fine-tuning, may fail to maintain sufficiently high feature similarity with the pre-trained model, and thus could possibly lose cross-task generalization. To address this issue, we propose Speech-FT, a novel two-stage fine-tuning framework designed to maintain cross-task generalization while benefiting from fine-tuning. Speech-FT first applies fine-tuning specifically designed to reduce representational drift, followed by weight-space interpolation with the pre-trained model to restore cross-task generalization. Extensive experiments on HuBERT, wav2vec 2.0, DeCoAR 2.0, and WavLM Base+ demonstrate that Speech-FT consistently improves performance across a wide range of supervised, unsupervised, and multitask fine-tuning scenarios. Moreover, Speech-FT achieves superior cross-task generalization compared to fine-tuning baselines that explicitly constrain weight changes, such as weight-space regularization and LoRA fine-tuning. Our analysis reveals that Speech-FT maintains higher feature similarity to the pre-trained model compared to alternative strategies, despite allowing larger weight-space updates. Notably, Speech-FT achieves significant improvements on the SUPERB benchmark. For example, when fine-tuning HuBERT on automatic speech recognition, Speech-FT is able to reduce phone error rate from 5.17% to 3.94%, lower word error rate from 6.38% to 5.75%, and increase speaker identification accuracy from 81.86% to 84.11%. Speech-FT provides a simple yet powerful solution for further refining speech representation models after pre-training. I Introduction Speech representation models have become a cornerstone of modern speech processing [ 1 , 2 , 3 , 4 ] , offering strong generalization across a wide range of downstream tasks such as phoneme recognition, speaker identification, and emotion recognition [ 5 ] . These models learn general speech representations from large-scale unlabeled data via self-supervised pre-training. Recently, there has been growing interest in further enhancing these representations through fine-tuning, either using supervised signals [ 6 ] or by continuing the self-supervised pre-training process [ 7 ] . Despite its potential benefits, fine-tuning often leads to a degradation in cross-task generalization ability [ 6 ] . A fine-tuned model may become overly specialized to the target task, resulting in poor transferability to other tasks. While this issue has been partially attributed to changes in the pre-trained features [ 8 ] , the underlying causes remain unclear. One common belief is that large deviations in weight space lead to representational drift [ 9 , 10 , 11 ] . However, this is not necessarily the case. Even with substantial weight updates, a model can retain high feature similarity to the original representations, depending on how fine-tuning is conducted [ 12 ] . A straightforward approach to preserving cross-task generalization is weight-space regularization, which penalizes large deviations from the pre-trained model during fine-tuning. While this constraint reduces deviations in weight space, it does not necessarily maintain feature similarity. As a result, models trained with weight-space regularization could possibly lose the relatively general representations learned during pre-training, leading to degraded performance on unrelated tasks. In this work, we investigate how fine-tuning can enhance pre-trained speech representation models while preserving cross-task generalization ability . To achieve this, we propose Speech-FT ( Speech F ine- T uning), a novel two-stage fine-tuning framework designed to maintain cross-task generalization while benefiting from fine-tuning. In the first stage, Speech-FT applies a fine-tuning strategy specifically designed to reduce representational drift with the pre-trained model. We refer to this approach as stable fine-tuning throughout this paper. In the second stage, Speech-FT further restores feature similarity and enhances cross-task generalization through weight-space interpolation between the pre-trained and fine-tuned models. This interpolation step can be interpreted as a form of model merging [ 13 ] , which has been shown to effectively integrate information from different models. In the context of Speech-FT, interpolation combines task-specific information acquired during fine-tuning with the relatively general representations of the pre-trained model. As we will analyze in Section V-D , Speech-FT achieves higher feature similarity with the pre-trained model compared to weight-space regularization, despite allowing larger parameter updates during fine-tuning. This aligns with prior work [ 12 ] which demonstrates the existence of low-impact subspaces in weight space where substantial weight changes induce minimal functional change; our findings provide empirical evidence suggesting that such invariant subspaces exist in the context of speech representation models as well. We conduct extensive experiments primarily on HuBERT [ 3 ] , across a variety of supervised and unsupervised fine-tuning scenarios. In addition, we validate the generality of Speech-FT on other speech representation models including wav2vec 2.0 [ 1 ] , DeCoAR 2.0 [ 2 ] , and WavLM Base+ [ 4 ] . Our evaluations on the SUPERB benchmark [ 5 ] reveal that Speech-FT consistently outperforms other baselines that constrain weight deviation during fine-tuning, such as weight-space regularization and LoRA fine-tuning [ 14 ] . In summary, the main contributions of this work are: • We propose Speech-FT , a novel two-stage fine-tuning framework for speech representation models that designed to maintain cross-task generalization while benefiting from fine-tuning. • We evaluate Speech-FT across diverse supervised and unsupervised fine-tuning scenarios, using various speech representation models, demonstrating its generality. • Speech-FT offers an efficient and effective approach to further improve general speech representations after pre-training. • We conduct extensive comparisons against a variety of baselines, including regularization-based methods and LoRA fine-tuning, and show that Speech-FT consistently achieves stronger performance. • We conduct an in-depth analysis showing that Speech-FT’s interpolation step plays a crucial role in restoring feature similarity with the pre-trained model, enabling it to maintain cross-task generalization ability while improving task-specific performance. II Methodology II-A Problem Formulation Let the pre-trained speech representation model be denoted as \theta_{0} , whose representations are evaluated on a set of downstream tasks \mathcal{T}=\{t_{1},t_{2},\dots,t_{n}\} . Given a particular fine-tuning task \hat{t} , our objective is to design a fine-tuning algorithm that produces a representation model \hat{\theta} which improves performance across tasks in \mathcal{T} . During fine-tuning, we attach a task prediction model D to the speech representation model. The task prediction model D takes the speech representations as input and produces task-specific predictions. Both D and \theta_{0} are optimized jointly during fine-tuning. After fine-tuning, the tuned task prediction model D is discarded (Step 1 in Figure 1 ). It is worth noting that we evaluate the cross-task performance of the representation model, rather than that of each tuned task prediction model D . To examine whether the representation model preserves its cross-task generalization ability, we adopt the SUPERB framework [ 5 ] (Step 3 in Figure 1 ), where downstream models are re-tuned individually without relying on the task prediction models trained during the initial fine-tuning stage. Fig. 1: The pipeline of Speech-FT for representation learning and evaluation. Step 1: A pre-trained representation model \theta_{0} undergoes stable fine-tuning on a specific task \hat{t} , producing a tuned representation model \theta^{\prime} while discarding the task prediction model D . Step 2: The pre-trained and tuned models are merged in weight space to obtain the final representation model \hat{\theta} . Step 3: The