Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts
Samin Yeasar Arnob, Zhan Su, Minseon Kim, Oleksiy Ostapenko, Riyasat Ohib, Esra’a Saleh, Doina Precup, Lucas Page-Caccia, Alessandro Sordoni · 2025
Read Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts by Samin Yeasar Arnob, Zhan Su, Minseon Kim, Oleksiy Ostapenko, Riyasat Ohib, Esra’a Saleh, Doina Precup, Lucas Page-Caccia, Alessandro Sordoni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07140.txt.
Introduction
Samin Yeasar Arnob and co-authors explore, in a 2025 machine learning paper, sparse adapters as building blocks for modular models that can be merged on the fly. Adapters train only a small part of a base network for each task, keeping costs low. The popular alternative uses low-rank updates, but sparse adapters had not been tested at scale for merging.
The team first gives a simple training recipe that beat both low-rank adapters and full fine-tuning in their setting. They then merged adapters for up to twenty language tasks, well past the usual scale. Merged sparse adapters led on tasks seen in training. Strong results on unseen tasks, though, stayed hard for every method tried.
Catalog record
| CREATOR | Samin Yeasar Arnob, Zhan Su, Minseon Kim, Oleksiy Ostapenko, Riyasat Ohib, Esra’a Saleh, Doina Precup, Lucas Page-Caccia, Alessandro Sordoni |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-07140
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts Samin Yeasar Arnob, Zhan Su, Minseon Kim, Oleksiy Ostapenko, Riyasat Ohib, Esra’a Saleh, Doina Precup, Lucas Page-Caccia, Alessandro Sordoni Abstract Merging parameter-efficient task experts has recently gained growing attention as a way to build modular architectures that can be rapidly adapted on the fly for specific downstream tasks, without requiring additional fine-tuning. Typically, LoRA serves as the foundational building block of such parameter-efficient modular architectures, leveraging low-rank weight structures to reduce the number of trainable parameters. In this paper, we study the properties of sparse adapters, which train only a subset of weights in the base neural network, as potential building blocks of modular architectures. First, we propose a simple method for training highly effective sparse adapters, which is conceptually simpler than existing methods in the literature and surprisingly outperforms both LoRA and full fine-tuning in our setting. Next, we investigate the merging properties of these sparse adapters by merging adapters for up to 20 natural language processing tasks, thus scaling beyond what is usually studied in the literature. Our findings demonstrate that sparse adapters yield superior in-distribution performance post-merging compared to LoRA or full model merging. Achieving strong held-out performance remains a challenge for all methods considered. 1 Introduction Multitask training, e.g. ( Raffel et al. 2019 ) , is an effective method to improve the performance of large language models (LLMs) across different tasks. However, for multitask training, all task-specific datasets need to be available simultaneously requiring data sharing during training. Model merging has emerged as an efficient alternative to building multi-task models ( Wortsman et al. 2022 ) , which allows tasks to be trained separately and then combined at the end of the training process, thus ensuring privacy of data and savings at training time. Recent work shows that model averaging can improve out-of-distribution performance over multitask training ( Yadav et al. 2024b ; Ostapenko et al. 2024 ) . However, merging models do not achieve the same performance as true multitask training due to weight interference, and requires careful weight manipulation ( Ilharco et al. 2023 ; Yadav et al. 2023 ; Akiba et al. 2024 ; White 2016 ; Davar 2024 ) during the merging process to resolve conflicts. Merging has recently been the focus of modular architectures that re-use parameter-efficient experts such as LoRA ( Hu et al. 2021b ) readily available on platforms such as Huggingface Hub ( Huang et al. 2024 ; Ostapenko et al. 2024 ; Muqeeth et al. 2024 ) . Recent evidence suggests that carefully composing LoRA ( Hu et al. 2021a ) modules can even outperform multi-task training on some tasks ( Prabhakar et al. 2024 ) . The widespread adoption of LoRA is due to the fact that it reduces the number of task-specific trainable parameters via low-rank decomposition while still maintaining good task performance. However, merging task-specific LoRA experts result in significant parameter interference given that all parameters of a given layer are modified for each task. In contrast, sparse fine-tuning methods — an alternative parameter-efficient fine-tuning methods that train a smaller sub-network within the base model for each task, have been proposed ( He et al. 2022 ; Ansell et al. 2022 ; Ansell et al. 2024 ; Panda et al. 2024 ) and have shown some promising results in composability ( Ansell et al. 2022 ) , albeit in a constrained setting of merging a language expert with a task expert for multi-lingual tasks. In addition, Su et al. 2024 have shown that sparse adapters are more suitable for concurrent serving scenarios than LoRA further motivating their use for modular scenarios. Figure 1: Left: The base model’s weights remain frozen during training, with only the sparse adapter modules updated; during each forward pass, the adapter’s outputs are combined with the fixed base weights. Right: Multiple sparse adapters—each trained separately on different tasks—are merged at inference time with the frozen base model weights. Figure 2: Left: Visualization of a weight matrix with sparsity ratio 0.5 . Trainable parameters are highlighted in blue . Element-sparse and block-sparse with block size B=2 and N_{B}=2 . Right: A visual representation of sparse-adapter reconfiguration during fine-tuning. Color transitions indicate updates to the weight-space via masking. Subsets of trainable parameters are updated periodically. Once the mask is fixed after a fixed number of steps, the weight-space remains constant, discarding unmasked weights. In this paper, we study the properties of sparse adapters as potential building blocks for modular architectures. We begin by introducing a simple yet straightforward method for training sparse adapters, which simplifies prior approaches ( Ansell et al. 2024 ; Panda et al. 2024 ) while delivering superior performance compared to LoRA and full fine-tuning. We leverage connection sensitivity ( Mozer & Smolensky 1988 ; Lee et al. 2018 ) to identify an important subset of parameters that are essential for the task. While structural sparse adapters are well-suited for concurrent serving ( Su et al. 2024 ) , we hypothesize that block-sparsity can also facilitate more efficient merging, analogous to how block matrix-matrix multiplication accelerates expert routing in Gale et al. 2023 . Motivated by this, we propose a simple approach to learn block-sparse adapters and find that they perform comparably to unstructured sparse adapters. We then explore the merging behavior of sparse adapters by conducting experiments across a set of 20 FLAN ( Longpre et al. 2023 ) tasks—a significant expansion over previous work, which typically focuses on just a few tasks ( Ansell et al. 2022 ; Panda et al. 2024 ) . We compare the performance of sparse adapters with LoRA and full fine-tuning. We evaluate performance on both the test sets of the 20 in-distribution tasks (held-in) and a set of unseen 10 tasks (held-out). Additionally, we benchmark recent merging methods, including Task Arithmetic ( Ilharco et al. 2023 ) , Ties ( Yadav et al. 2023 ) and Breadcrumbs ( Davar 2024 ) . Our results show that sparse adapters outperform both LoRA and full fine-tuning in a single fine-tuning experiment across 20 tasks. Moreover, merging sparse adapters retains strong held-in performance while maintaining competitive held-out results. Unlike full-finetuning merging methods, which degrade in performance when scaled to 20 experts, sparse-adapters prove to be more effective. Overall, our work demonstrates that sparse adapters offer a scalable and efficient approach to merging modular architectures for multitask learning, especially when extending beyond the typical two-task setting explored in previous research. While we show improved generalization for model-merging, a notable gap remains compared to multitask performance. 2 Related Work Parameter-Efficient Finetuning (PEFT) enable the efficient adaptation of LLMs through updating only a small subset of parameters ( Han et al. ) . PEFT approaches directly update the pre-trained weights in a parameter-efficient manner ( Hu et al. 2021a ; Zhang et al. 2023 ; Hayou et al. ; Liu et al. 2024 ; Dettmers et al. 2024 ) . The most prominent method is Low-Rank Adaptation (LoRA) ( Hu et al. 2021b ) , which parameterizes incremental weight updates \Delta is the product of two low-rank matrices. LoRA achieves performance comparable to or even surpassing that of full fine-tuning. More recently ( Hu et al.