MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning
Dacao Zhang, Kun Zhang, Shimao Chu, Le Wu, Xin Li, Si Wei · 2025
Read MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning by Dacao Zhang, Kun Zhang, Shimao Chu, Le Wu, Xin Li, Si Wei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22694.txt.
Introduction
Dacao Zhang and co-authors (2025) present MoRE, a method for teaching a large language model many tasks without paying full price. Fine-tuning models is expensive, so practitioners use low-rank adapters: trainable matrices slipped into the frozen model, one per task by default. Separate adapters waste capacity and ignore what tasks share. MoRE builds a mixture of low-rank experts. Adapters of ranks are treated as experts with different capacities, and a routing mechanism assigns each task, and each input, its share.
Small tasks can ride on low-rank experts; harder ones draw on bigger ones; related tasks share experts and transfer knowledge. The authors argue this aligns the cost with the difficulty of each task, rather than charging all tasks alike. For teams fine-tuning one model across many jobs, the saving is the point: multi-task skill at closer to single-task cost.
Catalog record
| CREATOR | Dacao Zhang, Kun Zhang, Shimao Chu, Le Wu, Xin Li, Si Wei |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22694
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning Dacao Zhang, Kun Zhang, Shimao Chu, Le Wu, Xin Li, Si Wei Abstract With the rapid development of Large Language Models (LLMs), Parameter-Efficient Fine-Tuning (PEFT) methods have gained significant attention, which aims to achieve efficient fine-tuning of LLMs with fewer parameters. As a representative PEFT method, Low-Rank Adaptation (LoRA) and its variants introduce low-rank matrices to approximate the incremental tuning parameters and achieve impressive performance over multiple scenarios. However, these methods either focus on single-task scenarios or separately train multiple LoRA modules for multi-task scenarios, limiting the efficiency and effectiveness of LoRA in multi-task scenarios. To better adapt to multi-task fine-tuning, in this paper, we propose a novel M ixture o f Low- R ank E xperts (MoRE) for multi-task PEFT. Specifically, instead of using an individual LoRA for each task, we align different ranks of LoRA module with different tasks, which we named low-rank experts . Moreover, we design a novel adaptive rank selector to select the appropriate expert for each task. By jointly training low-rank experts, MoRE can enhance the adaptability and efficiency of LoRA in multi-task scenarios. Finally, we conduct extensive experiments over multiple multi-task benchmarks along with different LLMs to verify model performance. Experimental results demonstrate that compared to traditional LoRA and its variants, MoRE significantly improves the performance of LLMs in multi-task scenarios and incurs no additional inference cost. We also release the model and code to facilitate the community 1 1 1 https://github.com/NLPfreshman0/MoRE . 1 Introduction Task/Rank r=1 r=2 r=4 r=8 r=16 r=32 MRPC 89.7 89.2 88.7 89.2 89.2 89.5 RTE 77.6 78.7 80.5 77.6 80.1 79.1 SST-2 94.4 94.6 94.8 94.5 94.4 94.5 CoLA 60.9 60.0 61.9 63.3 62.3 60.5 Table 1: LoRA-based Fine-tuning Performance of T5-base with varying ranks on different tasks Recent advancements in Large Language Models (LLMs) have revolutionized various domains, offering unprecedented performance across numerous tasks ( Raffel et al., 2020 ; Brown et al., 2020 ; Touvron et al., 2023 ) . Plenty of tuning strategies are designed to extend the application of LLMs, such as Instruction Tuning ( Wei et al., 2022 ; Zhang et al., 2023b ) , Continual Pre-Training ( Ke et al., 2023 ) , and Parameter-Efficient Fine-Tuning (PEFT) ( Houlsby et al., 2019 ; Liu et al., 2023b ; Lester et al., 2021 ; Hu et al., 2022 ) . Among these strategies, PEFT has drawn the most attention due to its fewer parameter tuning and lower computational cost. As the representative PEFT method, Low-Rank Adaptation (LoRA) ( Hu et al., 2022 ) introduces low-rank matrices to approximate the incremental tuning parameters and demonstrate good performance in many scenarios, which has become a standard paradigm for LLM fine-tuning and inspired many improvements ( Liu et al., 2024 ; Valipour et al., 2023 ; Ding et al., 2023 ) . Despite the achieved progress, LoRA relies on a fixed and unalterable intrinsic rank, making it not flexible enough in multi-task scenarios. Taking Table 1 as an example, when dealing with different tasks, LoRA requires different ranks to achieve the best performance (e.g., best ranks for MRPC and CoLA tasks are 1 and 8 ). Considering the high computational cost and storage cost of LLM fine-tuning, training multiple LoRAs is sub-optimal for applying LLMs to multi-task scenarios. Meanwhile, searching the best rank of LoRA during LLM fine-tuning is also time-consuming and computationally expensive ( Valipour et al., 2023 ) , which highlights the limitations of a one-size-fits-all approach in LoRA. This phenomenon also emphasizes the need for adaptive mechanisms that dynamically adjust ranks based on task requirements. To overcome the limitations of fixed ranks in LoRA, one promising direction is to explore adaptive mechanisms. For example, DyLoRA ( Valipour et al., 2023 ) dynamically trained all ranks during training to avoid separate rank tuning for each task. AdaLoRA ( Zhang et al., 2023a ) allocated the parameter budget based on the importance scores of the weight matrices and pruned insignificant singular values to exclude unimportant rank spaces. SoRA ( Ding et al., 2023 ) introduced a trainable gating unit and used proximal gradient descent to optimize the sparsity of the update matrices, thereby dynamically adjusting the intrinsic rank size during training. While these improvements enable dynamic adjustment of rank space, they are primarily designed for single-task scenarios. They do not consider the distinctions and connections among different tasks in multi-task scenarios, prohibiting the effectiveness of LoRA in multi-task scenarios. In the meantime, there also exist other strategies that try to exploit the connections among different tasks. However, they are still far from satisfied. For example, HyperFormer ( Mahabadi et al., 2021 ) enhanced adapter-based methods by utilizing a shared hypernetwork to facilitate cross-task knowledge sharing, while incorporating task-specific adapters to tailor the model for individual tasks. However, they face limitations due to their inherent performance constraints and additional inference latency. Prompt Tuning methods ( Vu et al., 2022 ; Asai et al., 2022 ; Wang et al., 2023b ) are proposed to use learned prompts on source tasks to initialize the learning of target tasks. Despite the effectiveness, these approaches typically require a two-stage training process (i.e., first on the source task and then on the target task), which requires higher data quality and results in training efficiency decrease. Meanwhile, parallel LoRA strategies ( Wang et al., 2023a ; Li et al., 2024 ; Liu et al., 2023a ; Huang et al., 2023 ) can effectively address the above shortcoming, offering a better adaptability in multi-task scenarios. Nonetheless, the usage of parallel LoRA modules increases the overall parameter count and resource consumption, contradicting the original purpose of LoRA to reduce the training parameters. Thus, one important question should be considered: “ How to achieve efficient LLM fine-tuning in multi-task scenarios remains challenging? ” To this end, in this paper, we design a novel Mixture of Low-Rank Experts (MoRE) for efficient LLM fine-tuning in multi-task scenarios. Since different tasks require different ranks of LoRA, we propose to build connections between the ranks and the tasks in a Mixture-of-Expert (MoE) manner. Specifically, we propose to treat each rank in the LoRA module as an expert and design a novel Adaptive Rank Selector . Thus, the different experts corresponding to different tasks can share common information and maintain distinctive information simultaneously (i.e., the ranks r_{i} and r_{j} can share some common parameters). Meanwhile, our proposed selector uses a gating mechanism to select the appropriate rank expert for each task. Moreover, to fully exploit the distinctions and connections among different tasks for accurate rank selection, we develop a novel CL-based Task Embedding module, which assigns a task embedding to each task and uses a Contrastive Learning (CL) optimization to ensure the quality of learned task embeddings. Furthermore, we incorporate the Balanced Dataset Sampling strategy to address the severe dataset imbalance in multi-task scenarios. Along this line, MoRE can fully exploit the potential of LoRA and realize efficient LLM fine-tuning in multi-task scenarios. Finally, extensive experiments on multi-task benchmarks demonstrate the efficiency and effectiveness of MoRE. 2 Related Work 2.1 Parameter-Efficient Fine-Tuning (PEFT) PEFT methods are designed to adapt LLMs to new tasks with minimal additional parameters. Representative works include