LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation

Farzad Farhadzadeh, Debasmit Das, Shubhankar Borse, Fatih Porikli, Qualcomm AI Research, San Diego, CA 92121, USA {ffarhadz, debadas, sborse, fporikli}@qti.qualcomm.com · 2025

Read LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation by Farzad Farhadzadeh, Debasmit Das, Shubhankar Borse, Fatih Porikli, Qualcomm AI Research, San Diego, CA 92121, USA {ffarhadz, debadas, sborse, fporikli}@qti.qualcomm.com (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16559.txt.

Introduction

Farzad Farhadzadeh, Debasmit Das, Shubhankar Borse, and Fatih Porikli solve a costly problem when base models get replaced. Their twenty twenty-five paper introduces LoRA-X for cross-model adaptation. Small adapter modules tune large foundation models cheaply. When a base model is retired, every adapter normally needs retraining.

Original training data is often locked away by privacy or licensing rules. Synthetic data may fail to match the real data well. LoRA-X instead moves adapter settings between models with no training at all. It forces the adapter to work inside the source model subspace. Tests support the transfer approach. Teams can keep useful tuning work when platforms upgrade their base models.

Catalog record

CREATORFarzad Farhadzadeh, Debasmit Das, Shubhankar Borse, Fatih Porikli, Qualcomm AI Research, San Diego, CA 92121, USA {ffarhadz, debadas, sborse, fporikli}@qti.qualcomm.com
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-16559

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation Farzad Farhadzadeh, Debasmit Das, Shubhankar Borse, Fatih Porikli, Qualcomm AI Research, San Diego, CA 92121, USA {ffarhadz, debadas, sborse, fporikli}@qti.qualcomm.com Abstract The rising popularity of large foundation models has led to a heightened demand for parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), which offer performance comparable to full model fine-tuning while requiring only a few additional parameters tailored to the specific base model. When such base models are deprecated and replaced, all associated LoRA modules must be retrained, requiring access to either the original training data or a substantial amount of synthetic data that mirrors the original distribution. However, the original data is often inaccessible due to privacy or licensing issues, and generating synthetic data may be impractical and insufficiently representative. These factors complicate the fine-tuning process considerably. To address this challenge, we introduce a new adapter, Cross-Model Low-Rank Adaptation (LoRA-X), which enables the training-free transfer of LoRA parameters across source and target models, eliminating the need for original or synthetic training data. Our approach imposes the adapter to operate within the subspace of the source base model. This constraint is necessary because our prior knowledge of the target model is limited to its weights, and the criteria for ensuring the adapter’s transferability are restricted to the target base model’s weights and subspace. To facilitate the transfer of LoRA parameters of the source model to a target model, we employ the adapter only in the layers of the target model that exhibit an acceptable level of subspace similarity. Our extensive experiments demonstrate the effectiveness of LoRA-X for text-to-image generation, including Stable Diffusion v1.5 and Stable Diffusion XL. 1 1 footnotetext: Corresponding author 2 2 footnotetext: Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. 1 Introduction Figure 1: LoRA-X training-free transfer (a) from source SD-v1.5 to targets SD Eff-v1.0 and RV-v3.0, (b) from source SDXL to targets SSD-1B and RVXL-v3.0. Large foundation models (LFMs) have demonstrated outstanding performance across various domains, including natural language processing ( OpenAI et al. 2023 ; Gemini Team et al. 2023 ; Anthropic 2024 ; AI@Meta 2024 ) and computer vision ( Ho et al. 2020 ; Rombach et al. 2022 ) . Due to their remarkable capabilities, fine-tuning LFMs for a wide array of downstream tasks has become common practice. In the full fine-tuning approach, each new model tailored to a specific task generally retains the same number of parameters as the original model. As models increase in size and customization requirements grow, the need to store such fully fine-tuned checkpoints also rises, leading to substantial storage and memory costs. To address this challenge, Parameter-Efficient Fine-Tuning (PEFT) ( Xu et al. 2023 ) methods, such as Low-Rank Adaptation (LoRA) ( Hu et al. 2022 ) , offer a promising solution. PEFT methods aim to reduce the number of parameters that need to be updated during fine-tuning, making the process more computationally efficient and less resource-intensive. LoRA, in particular, addresses this by representing the weight changes in the model using two low-rank matrices, {\bm{A}} and {\bm{B}} . Specifically, the weight update is expressed as {\bm{W}}_{0}+\Delta{\bm{W}}={\bm{W}}_{0}+{\bm{B}}{\bm{A}} , where {\bm{W}}_{0} is the original weight matrix, and \Delta{\bm{W}} is the change applied during fine-tuning. By decomposing the weight changes into these low-rank matrices, LoRA significantly reduces the number of parameters that need to be adjusted, thus making the fine-tuning process more efficient. Nevertheless, a consequential limitation of the LoRA approach is its dependency on the base model. A LoRA adapter fine-tuned for a specific task is intrinsically tied to its base model and cannot function independently of it. Moreover, it cannot be directly transferred to another base model without additional training. This dependency poses a profound challenge when the base model is deprecated and replaced by a newer version. In such cases, applications utilizing LoRA adapters from the deprecated model must transition their adapters to the newer base model. This process can be cumbersome, resource-intensive, and, crucially, infeasible if the original fine-tuning data is no longer available. This naturally leads us to an important question: Can we design a LoRA adapter that can be transferred between different base models without requiring additional training or access to the original data? A solution addressing this problem allows the longevity of LoRA adapters and eliminates the need for repeated fine-tunings as base models evolve. In this work, we introduce LoRA-X, a compact adapter that can be transferred between different versions of a base model without the need for fine-tuning with the original data. The core idea behind LoRA-X is to maintain the adapter within the column-row subspace of the base model weights. This innovative strategy ensures that the adapter remains compatible with alternative versions of the base model, simplifying the transition process and enhancing the adaptability of fine-tuned models. To the best of our knowledge, LoRA-X is the first adapter designed for transferability without additional training. Our main contributions are (See Figure 1 ): • Concept of LoRA-X : We introduce LoRA-X, a versatile cross-model adapter that can be seamlessly transferred across various base models without the need for additional training. This innovation simplifies the process of adapting models to new tasks or domains. • Subspace Similarity and Transferrability Metrics : We propose a subspace similarity metric to identify pairs of modules between source and target base models that share a common subspace. Additionally, we compute a scalar transferability cost metric to quantify the difficulty of transferring adapters. This metric is derived from the optimal transport solution to pair-wise subspace similarities between different modules of the source and target models. • Transfer Method : We present a practical yet highly effective method to transfer LoRA-X from a source model to a target model. This approach ensures that the target model inherits the capabilities of the source model, enabling efficient knowledge transfer and improved performance. Throughout this paper, we refer to “Source” as the case where the LoRA-X adapter is trained from scratch on a specific base model (pretrained weights) using a training dataset. Conversely, “Target” denotes the case where the LoRA-X adapter, transferred from a different base model without any additional training, is applied to a new base model. 2 Related Work Parameter Efficient Finetuning (PEFT) ( Xu et al. 2023 ) has emerged as a pivotal research area, particularly in transfer learning, where the challenge lies in adapting large pretrained models to specific tasks without extensive retraining. The literature on PEFT includes various strategies, each aiming to modify a minimal number of parameters while maintaining competitive performance. Several PEFT methods have been introduced, such as Adapter Modules ( Sung et al. 2022 ) , Prompt Tuning ( Lester et al. 2021 ) , and popular Low-Rank Adaptation techniques like LoRA ( Hu et al. 2022 ) and VeRA ( Kopiczko et al. 2023 ) . Among these, recent methods such as SVDiff ( Han et al. 2023 ) , PiSSA ( Meng et al. 2024 ) , SVFT ( Lingam et al. 2024 ) and LoRA-XS ( Bałazy et al. 2024 ) fine-tune the singular values of base model weight matrices. However, SVDiff and its follow-up works primarily aim to reduce the number of parameters during fine-tuning rather than the

Read the complete text