Efficient Multi-Source Knowledge Transfer by Model Merging

Marcin Osial, Bartosz Wójcik, Bartosz Zieliński, Sebastian Cygert · 2025

Read Efficient Multi-Source Knowledge Transfer by Model Merging by Marcin Osial, Bartosz Wójcik, Bartosz Zieliński, Sebastian Cygert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19353.txt.

Introduction

Marcin Osial, Bartosz Wojcik, Bartosz Zielinski, and Sebastian Cygert merge many trained models into one to transfer their knowledge cheaply in a 2025 paper. The web holds countless ready models, yet standard transfer learning uses just one source. Prior merging methods are coarse and scale poorly to many or large models.

Their method first splits each source model into elementary rank-one parts using a classic matrix tool. It then keeps only the most salient parts across all sources and fine-tunes just their weights for the target task. The result is a fine-grained, scalable merge that works for both vision and language models. Tests show it stays robust when inputs or parameters are perturbed.

Catalog record

CREATORMarcin Osial, Bartosz Wójcik, Bartosz Zieliński, Sebastian Cygert
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-19353

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Multi-Source Knowledge Transfer by Model Merging Marcin Osial, Bartosz Wójcik, Bartosz Zieliński, Sebastian Cygert Abstract While transfer learning is an advantageous strategy, it overlooks the opportunity to leverage knowledge from numerous available models online. Addressing this multi-source transfer learning problem is a promising path to boost adaptability and cut re-training costs. However, existing approaches are inherently coarse-grained, lacking the necessary precision for granular knowledge extraction and the aggregation efficiency required to fuse knowledge from either a large number of source models or those with high parameter counts. We address these limitations by leveraging Singular Value Decomposition (SVD) to first decompose each source model into its elementary, rank-one components. A subsequent aggregation stage then selects only the most salient components from all sources, thereby overcoming the previous efficiency and precision limitations. To best preserve and leverage the synthesized knowledge base, our method adapts to the target task by fine-tuning only the principal singular values of the merged matrix. In essence, this process only recalibrates the importance of top SVD components. The proposed framework allows for efficient transfer learning, is robust to perturbations both at the input level and in the parameter space (e.g., noisy or pruned sources), and scales well computationally. Introduction The increasing complexity of models and the immense computational costs associated with their training necessitate the efficient utilization of existing resources. Transfer learning ( Zhuang et al. 2020 ) , which involves initializing networks with weights from a pretrained model, has emerged as a standard practice. This practice relies on foundational models, such as large-scale vision transformers ( Awais et al. 2025 ) and self-supervised models ( Caron et al. 2021 ) , which learn robust and generalized representations from vast, general-purpose datasets (e.g., ImageNet, LAION-5B). By effectively leveraging this broad pre-existing knowledge, transfer learning significantly reduces the demand for extensive task-specific data, accelerates training, and enhances overall model performance across a wide range of computer vision tasks. Figure 1: Accuracy versus the number of trainable parameters for our method and aTLAS, averaged over all target tasks with ViT-B-32 architecture. Each data point corresponds to a parameter budget defined by the top N singular values (N=10%, 20%, and 40%). The solid line denotes the mean accuracy, while the shaded area represents the standard deviation. The variation is calculated over all source task vectors. However, the wealth of specialized knowledge residing in other fine-tuned models remains largely untapped. Each model represents a valuable knowledge asset, with hundreds of thousands of versions publicly available on platforms like Hugging Face. Each new adaptation typically requires training from its original, pre-trained state, neglecting the specialized knowledge already acquired by previously fine-tuned models for distinct tasks. This gap has sparked considerable interest in developing methods for combining multiple models into a unified model ( Shu et al. 2021 ; Yang et al. 2022 ) . Among these is model merging ( Yang et al. 2024 ) , which presents a notable opportunity to fuse capabilities at low cost, with an example of an aTLAS method ( Zhang et al. 2024 ) , which addresses the multi-source knowledge transfer for a new target task. It learns to scale and combine task vectors anisotropically ( Ilharco et al. 2022 ) , which are the weight differences between fine-tuned models and their pre-trained state. The method operates by learning a distinct coefficient for each of the T tasks, across each of the L layers, and for each of P partitions within a weight matrix. These coefficients collectively form a learned tensor with dimensions T\times L\times P , allowing for adjustments to the model’s behavior for new tasks. While holding significant promise, aTLAS lacks mechanisms for granular parameter selection, which restricts the precision of knowledge fusion. Furthermore, aTLAS’s memory footprint scales linearly with the number of added sources due to its reliance on using full task vectors. This design prevents the aggregation of larger models or a greater number of source models. As a result, its training is confined to multi-GPU environments, undermining its parameter-efficient benefits. This coarse-grained approach lacks a robust knowledge composition mechanism, making it susceptible to perturbations from both corrupted or pruned parameters and degraded inputs. In this paper, we present a unified method that efficiently combines specialized knowledge from multiple fine-tuned source models in the parameter space to facilitate transfer to a new, unseen target task. We depart from the methodology proposed in the aTLAS paper, which assumes that the entire set of full-rank task vectors is used throughout the entire training process. Instead, we propose a more scalable approach that first aggregates knowledge and then allows for its efficient refinement during adaptation. First, we leverage Singular Value Decomposition (SVD) to decompose each task vector into its elementary, rank-one components. This allows us to identify and isolate granular patterns learned for each source task. A subsequent combination stage aggregates these components from all source models, performing a joint ranking to retain only a small, fixed number of the most significant ones. We term this strategy AXIS , as it embodies the principle of A ggregation by e X traction of I mportant S ingular components. Such selective aggregation ensures a stable memory usage and constant wall-time footprint during training, irrespective of the number of source models or original task matrix sizes (see Figure 5 ). Consequently, the proposed design is not only more parameter-efficient, but it also proves to be more robust. Our key contributions include: • We introduce a scalable approach, AXIS, which outperforms the state-of-the-art method, aTLAS, across a wide spectrum of evaluation conditions, including 21 distinct tasks and various parameter budgets. • The computational efficiency of AXIS is a key advantage, allowing for the scaling of knowledge transfer from a large number of source tasks and larger models. • We demonstrate that AXIS exhibits robustness against degradations at both the parameter and input levels. • Through ablation studies, we offer insights into the underlying structure of knowledge composition and how it can be leveraged. Related Works Model merging is gaining traction as a promising approach to leverage fine-tuned models without requiring access to training data or incurring increased model size and inference costs. The merging stage itself demands low computational resources and could be entirely training-free. While numerous works explore combining models’ weights with diverse architectures ( Du et al. 2025 ) or those trained without a shared initialization ( Rinaldi et al. 2025 ; Stoica et al. 2023 ; Ainsworth, Hayase, and Srinivasa 2022 ) , these often prove less effective than approaches that assume all considered models originate from the same base model ( Akiba et al. 2025 ; Yang et al. 2023 ; Yadav et al. 2023 ) . This greater effectiveness is largely built upon the concept of a task vector, introduced by ( Ilharco et al. 2022 ) , which operates on full-rank weight matrices, in contrast to merging low-rank approximations, such as LoRA modules ( Zhao et al. 2024 ) . Model merging can enhance single-task performance ( Wortsman et al. 2022a ; Ramé et al. 2023 ; Jang, Yun, and Han 2024 ) or be utilized in the creation of multitask models ( Marczak et al. 2025 ; Gargiulo et al. 2025 )

Read the complete text