Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging
Haobo Zhang, Jiayu Zhou · 2025
Read Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging by Haobo Zhang, Jiayu Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22934.txt.
Introduction
Haobo Zhang and Jiayu Zhou tackle a snag in merging small fine-tuned models into one that can do many tasks. Large language models are often adapted cheaply with a method called LoRA, which trains only a small set of added weights. Merging several such models should give one multi-task model for free, yet merged LoRA models often lose much of their skill.
The pair trace the loss to a clash between the trained weights and the data each task uses. Their fix, called OSRM, confines each task's LoRA updates to its own separate subspace before training starts, so tasks cannot push each other's outputs around. Added to standard merging tools, the fix raised merged scores on eight data sets and kept single-task skill. It also held up better when merging settings changed.
Catalog record
| CREATOR | Haobo Zhang, Jiayu Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22934
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging Haobo Zhang, Jiayu Zhou Abstract Fine-tuning large language models (LMs) for individual tasks yields strong performance but is expensive for deployment and storage. Recent works explore model merging to combine multiple task-specific models into a single multi-task model without additional training. However, existing merging methods often fail for models fine-tuned with low-rank adaptation (LoRA), due to significant performance degradation. In this paper, we show that this issue arises from a previously overlooked interplay between model parameters and data distributions. We propose O rthogonal S ubspaces for R obust model M erging ( OSRM ) to constrain the LoRA subspace prior to fine-tuning, ensuring that updates relevant to one task do not adversely shift outputs for others. Our approach can seamlessly integrate with most existing merging algorithms, reducing the unintended interference among tasks. Extensive experiments on eight datasets, tested with three widely used LMs and two large LMs, demonstrate that our method not only boosts merging performance but also preserves single-task accuracy. Furthermore, our approach exhibits greater robustness to the hyperparameters of merging. These results highlight the importance of data-parameter interaction in model merging and offer a plug-and-play solution for merging LoRA models. 1 Introduction Pre-trained language models (LMs) have achieved remarkable success across diverse tasks, with fine-tuning approaches enabling strong downstream performance ( Radford et al., 2019 ; Touvron et al., 2023 ) . However, maintaining a separate fine-tuned model for each task becomes prohibitively expensive in terms of both storage and deployment. While multi-task learning ( Zhang and Yang, 2021 ) attempts to address this issue by training a unified model for multiple tasks, it demands simultaneous access to all task data and high computational overhead, thereby limiting its scalability. An appealing alternative is model merging, which combines multiple task-specific models into a single multi-task model without further training ( Ilharco et al., 2022 ; Huang et al., 2024 ; Yadav et al., 2024 ) . Early merging techniques primarily average parameters, often guided by Fisher information ( Matena and Raffel, 2022 ) or inner-product-based metrics ( Jin et al., 2022 ) . Another line of work employs task vectors—the difference between pre-trained and fine-tuned model weights—and manipulates them before summation ( Yadav et al., 2024 ; Du et al., 2024 ; Jiang et al., 2023 ) . Despite these promising developments, merging models that were fine-tuned with low-rank adaptation (LoRA) ( Hu et al., 2021 ) , remains challenging and can severely degrade performance ( Stoica et al., 2024 ; Tang et al., 2023a ) . We argue that this degradation stems from parameter interference and how each model’s parameters interact with out-of-task data. While prior work has focused on preserving orthogonality among task vectors ( Ortiz-Jimenez et al., 2023 ; Gao et al., 2024 ; Yoshida et al., ) or aligning them in a shared space ( Stoica et al., 2024 ) , these data-free strategies often overlook the crucial interplay between latent features and parameter updates. Specifically, consider a pre-trained layer W_{0} and two sets of learned LoRA blocks \{B_{1},A_{1}\} and \{B_{2},A_{2}\} . The merged model is then formulated as W_{m}=W_{0}+B_{1}A_{1}+B_{2}A_{2} . Given a latent feature vector \mathbf{h}_{1} from task T_{1} , the merged model produces W_{m}\mathbf{h}_{1}=W_{1}\mathbf{h}_{1}+B_{2}A_{2}\mathbf{h}_{1} , where the first term represents the intended response while the second term is the undesired shift. Notably, existing data-free approaches that focus solely on resolving parameter conflicts are insufficient to mitigate such interference. In this paper, we propose a novel approach named OSRM ( O rthogonal S ubspaces for R obust model M erging) that restricts the LoRA subspace before fine-tuning, making it largely orthogonal to irrelevant out-of-task data distributions. Concretely, we aim to reduce the interference between data and parameters by minimizing \|A_{2}\mathbf{h}_{1}\|_{F} and derive an analytical solution under the assumption that A_{2} has an orthogonal basis. Our method integrates seamlessly with existing merging algorithms and mitigates the unintentional output shifts that arise when multiple LoRA modules are combined (c.f. Fig. 1 ). We further present practical extensions to ensure robust performance in real-world scenarios. Extensive experiments on eight datasets using three widely used LMs and two large LMs confirm that our approach consistently outperforms existing merging baselines on multi-task evaluations, while preserving strong single-task performance. Additionally, our empirical results show the robustness of our method against hyperparameters, such as the scaling coefficient, the sample size, the number of tasks, and the choice of learnable blocks. Our findings highlight the importance of considering data-parameter interplay in model merging and demonstrate a generalizable strategy for combining LoRA models more effectively. Figure 1: Overview of OSRM, which seeks a data-driven subspace to initiate LoRA fine-tuning and thereby greatly improves model performance when merging multiple LoRA models from different tasks. W_{0} is the pre-trained weight. \{B_{i},A_{i}\} are LoRA fine-tuned on the i -th task. Purple : (W_{0}+B_{1}A_{1})*h_{1} is the required output. Light blue : Decompose the sample covariance matrix to initialize A_{2} . Dark blue : Reduce the output shift induced by B_{2}A_{2} . 2 Related Work Model Merging. A significant body of work explores merging models that have been fine-tuned independently on different datasets to obtain a unified multi-task model. Ilharco et al. (2022) introduced Task Arithmetic (TA), which defines task vectors for models and combines them using a unified weight. Building on TA, several methods have been proposed to address weight entanglement in task-specific models by aligning them before merging ( Gao et al., 2024 ; Ortiz-Jimenez et al., 2023 ; Yoshida et al., ; Tam et al., 2024 ; Stoica et al., 2024 ; Hazimeh et al., ; Ainsworth et al., 2022 ) . Other approaches improve TA by designing better weighting schemes for model averaging ( Matena and Raffel, 2022 ; Jin et al., 2022 ; Wang et al., 2024a ; Zhou et al., 2024 ) . A different research direction focuses on manipulating task vectors to enhance merging performance ( Yadav et al., 2024 ; Du et al., 2024 ; Jiang et al., 2023 ; He et al., 2024 ; Wang et al., 2024b ; Davari and Belilovsky, 2024 ) . Alternatively, some methods perform adaptive model merging using dynamic inference, employing either a router ( Lu et al., 2024 ; Stoica et al., 2023 ) or masks ( Huang et al., 2024 ) . Merging LoRA Models. LoRA ( Hu et al., 2021 ) has become a widely used technique for parameter-efficient fine-tuning. However, most existing model merging methods struggle to effectively transfer to LoRA models ( Stoica et al., 2024 ; Tang et al., 2023a ) . KnOTS ( Stoica et al., 2024 ) highlights the importance of merging LoRA models within a shared space, while Tang et al. (2023a) propose that linearized LoRA models reduce weight entanglement. Zhao et al. (2024) and Prabhakar et al. (2024) further improve LoRA merging by learning optimal merging weights and clustering LoRA modules, respectively. Despite these advancements, existing methods often overlook the interaction between model parameters and data, leading to suboptimal merging performance. Moreover, prior work primarily focuses on the phases during and after fine-tuning. In contrast, our method explicitly addresses parameter-data interactions and focuses on the merging phase before fine-tuning. 3 Preliminaries and Background Notation. Let