LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
Wenbing Li, Zikai Song, Hang Zhou, Yunyao Zhang, Junqing Yu, Wei Yang · 2025
Read LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing by Wenbing Li, Zikai Song, Hang Zhou, Yunyao Zhang, Junqing Yu, Wei Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00029.txt.
Introduction
Wenbing Li and five co-authors present a way to mix small task adapters inside large language models, in a paper from 2025. Each adapter is cheap to train for one task, and a router picks which ones to use. Prior mixes swapped whole layers or added side branches, which wasted capacity.
Their router sends task adapters into the core attention layers, giving fine control at the level of single tokens. A new training loss keeps loads balanced while still letting adapters specialize. Across fifteen test sets, the method beat prior mixes while using about half the trained values. It also works with frozen adapters taken from public sources.
Catalog record
| CREATOR | Wenbing Li, Zikai Song, Hang Zhou, Yunyao Zhang, Junqing Yu, Wei Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00029
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing Wenbing Li, Zikai Song, Hang Zhou, Yunyao Zhang, Junqing Yu, Wei Yang Abstract Recent efforts to combine low - rank adaptation (LoRA) with mixture - of - experts (MoE) for adapting large language models (LLMs) to multiple tasks, yet exhibit prevailing limitations: they either swap entire attention/feed - forward layers for switch experts or bolt on parallel expert branches, diluting parameter efficiency and task fidelity. We propose the LoRA-Mixer , a modular and lightweight MoE framework that integrates LoRA experts. Our core innovation lies in replacing the projection matrices of the attention module’s input/output linear layers with dynamically routed, task-specific LoRA experts. This design ensures seamless compatibility with diverse foundation models—including transformers and state space models (SSMs)—by leveraging their inherent linear projection structures. The framework supports two operational paradigms: (1) joint optimization of LoRA experts and routing mechanisms via a novel hard-soft routing strategy, or (2) direct deployment of pre-trained, frozen LoRA modules sourced from external repositories. To enable robust router training with limited data while ensuring stable routing decisions and maximizing expert reuse, we introduce an adaptive Specialization Balance Loss (SBL) that jointly optimizes expert balance and task-specific alignment. Extensive experiments on seven benchmark datasets, including MedQA, CoLA, SST-2, GSM8K, ARC-E, ARC-C, and HumanEval, demonstrate the effectiveness of LoRA-Mixer . On datasets such as GSM8K, HumanEval, and MedQA, LoRA-Mixer achieves significant improvements of 7.61% , 4.88% , and 3.08% over the base models, respectively. Compared with the state-of-the-art methods, LoRA-Mixer achieves additional improvements of 1.09% , 1.45% , and 1.68% , respectively, using only 48% of the parameters, demonstrating its efficiency and strong performance. 1 Introduction Large Language Models (LLMs) have achieved unprecedented proficiency in general-purpose reasoning and generation, yet their adaptation to specialized downstream domains remains computationally prohibitive, requiring significant resources for full-scale fine-tuning [ 1 , 2 ] . To mitigate these resource demands, parameter-efficient fine-tuning (PEFT) methods [ 37 , 38 , 39 , 40 , 41 , 42 ] have emerged as a scalable paradigm for task-specific adaptation. Among these, Low-Rank Adaptation (LoRA) [ 4 , 5 ] has demonstrated particular efficacy, operating through low-rank decomposition of updates to the pretrained weights—enabling efficient tuning with minimal parameter overhead. Recent work has explored modularly composing independently trained LoRA modules as a promising strategy for multitask adaptation; however, naive composition can result in interference between task-specific subspaces, limiting their synergistic potential [ 15 , 14 ] . This limitation has motivated exploration of mixture-of-experts (MoE) architecture [ 3 , 6 ] , which treat each task-specific LoRA as an expert and sparsely activate and fuse the experts. Recent studies demonstrate promising directions in hybrid Lo- Figure 1 : MoE and LoRA-MoE integration methods. (1) and (2): replace the attention or feedforward layers with switch experts; (3) and (4): introduce LoRA experts branches in parallel with the attention or feedforward layers, and fuse the output into the main branch. Our LoRA-Mixer (right) applies mixture of LoRA experts to the projection layers, which can effectively leverages the attention mechanism. RA-MoE frameworks [ 14 , 13 , 15 , 16 , 35 , 34 , 23 , 24 ] , aiming to enhance model performance on complex tasks across multi-domain datasets while preserving the parameter efficiency of fine-tuning. The central challenge in composing multiple pre-trained LoRAs lies in effectively synergizing them—achieving enhanced performance across constituent tasks while minimizing training overhead and preserving their distinct, task-specific characteristics. Conventional methods integrating LoRA into mixture-of-experts (MoE) architectures typically follow one of two paradigms: (1) directly substituting attention or feedforward layers with LoRA-based switchable experts [ 13 , 23 ] , emulating the classical MoE structure [ 6 , 3 ] ; or (2) introducing parallel branches of LoRA experts whose outputs are subsequently fused into the primary model pipeline [ 14 , 15 ] , as illustrated in Fig. 1 . While these strategies have demonstrated promising empirical results [ 14 , 35 , 52 , 53 , 54 , 55 ] , they still encounter fundamental limitations. Specifically, the vanilla MoE-inspired paradigm necessitates joint training across all expert modules, significantly increasing training data demands and restricting the modular reuse and transferability of pre-trained LoRAs. Conversely, the parallel LoRA-expert approach circumvents the inherent attention or state transition mechanisms, resulting in simplistic output fusion and suboptimal overall integration. Additionally, auxiliary losses commonly employed for routing optimization inherently promote uniform load balancing among experts, diminishing the capacity for nuanced task-awareness [ 13 ] . These limitations motivate the development of a more flexible, plug-and-play framework that is model-architecture agnostic, compatible with both Transformer and state-space model (SSM) architectures [ 7 ] , capable of training an efficient and discriminative routing mechanism with minimal computational and data demands, and maximizing the reuse and transferability of independently pre-trained LoRA modules. In this paper, we introduce LoRA-Mixer , a novel framework designed to efficiently synergize multiple pre-trained LoRA modules by treating them as dynamic, pluggable memory cells. LoRA-Mixer equips the linear projection layers of the original model with mixed LoRA experts, enabling these experts to directly leverage the effectiveness of the core attention or state-transition mechanisms. LoRA-Mixer supports LoRA modules sourced from external repositories, independently trained, or jointly trained through hard routing strategies, allowing seamless plug-and-play usage across various tasks and domains. Importantly, our method significantly reduces the necessity for training data or extensive re-adaptation, requiring only minimal additional data to effectively train the routing mechanism. Consequently, LoRA-Mixer is particularly suitable for constructing large-scale, modular language models characterized by task-specific memory, computational efficiency, and strong transferability. To further enhance efficiency and maintain routing effectiveness, we propose a novel Router Specialization Balancing Loss ( RSBL ). RSL aligns routing decisions with token-level expert usage, maintaining moderate entropy to encourage exploratory behavior. During inference, we employ sparse top- K fusion, effectively balancing computational cost and scalability without compromising expert selectivity. Extensive evaluations conducted on seven benchmark datasets—MedQA, CoLA, SST-2, GSM8K, ARC-E, ARC-C, and HumanEval—demonstrate that integrating LoRA-Mixer substantially improves model performance across all evaluated tasks. Additionally, cross-domain experiments confirm the versatility and adaptability of our proposed framework. Compared to state- Figure 2 : The overall architecture of LoRA-Mixer. LoRA-Mixer is applied to the linear projection layers in serial with the Attention and SSM modules and support all major LLM structures. LoRA-Mixer reueses the LoRA experts sourced from Internet, trained individually or jointly trained using hard routing. The routing training is guided by RSL loss for balancing experts loads and specificity. of-the-art approaches, LoRA-Mixer achieves significant efficiency, using only 48% of the training parameters while improving performance by 1.09%, 1.45%, 1.19%, and 0.85% on GSM8K, HumanEval, SST-2, and CoLA benchmarks, respectively. 2 Related Work PEFT For LLMs Low-rank adaptation (LoRA) [ 12 , 11 , 4 , 10 , 47 , 49 , 50 , 51 ] effectively fine-tunes large models by learning a low-rank matrix and freezing the original weights. While effective for a single task, its task-specific nature limits generalization. Recent work combines LoRA with mixture of experts (MoE) [ 13 , 14 , 25 , 15 , 16 ] to achieve dynamic adaptation. For example, MixLoRA [ 13 ] uses LoRA experts for top-k routing in FFN, improving multi-task