Mixture of Routers

Jia-Chen Zhang, Yu-Jie Xiong, Xi-He Qiu, Chun-Ming Xia, Fei Dai · 2025

Read Mixture of Routers by Jia-Chen Zhang, Yu-Jie Xiong, Xi-He Qiu, Chun-Ming Xia, Fei Dai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23362.txt.

Introduction

Jia-Chen Zhang and co-authors propose Mixture of Routers in 2025, a fine-tuning method for large language models. Adapter tuning is cheap but limited, while expert mixtures add power by routing each input to fit specialists. Routing itself can misfire, though, sending inputs to wrong experts and starving others.

Borrowing redundancy ideas from fault-tolerant design, the method staffs the router with several sub-routers. A learnable main router weighs their votes for each selection. The design plugs into existing setups with few added parameters. Across most test tasks it beat baseline tuners by about one percent on average. Code is shared for reuse.

Catalog record

CREATORJia-Chen Zhang, Yu-Jie Xiong, Xi-He Qiu, Chun-Ming Xia, Fei Dai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23362

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mixture of Routers Jia-Chen Zhang, Yu-Jie Xiong, Xi-He Qiu, Chun-Ming Xia, Fei Dai Abstract Supervised fine-tuning (SFT) is a milestone in aligning large language models with human instructions and adapting them to downstream tasks. In particular, Low-Rank Adaptation (LoRA) has gained widespread attention due to its parameter efficiency. However, its impact on improving the performance of large models remains limited. Recent studies suggest that combining LoRA with Mixture-of-Experts (MoE) can significantly enhance fine-tuning performance. MoE adapts to the diversity and complexity of datasets by dynamically selecting the most suitable experts, thereby improving task accuracy and efficiency. Despite impressive results, recent studies reveal issues in the MoE routing mechanism, such as incorrect assignments and imbalanced expert allocation. Inspired by the principles of Redundancy and Fault Tolerance Theory. We innovatively integrate the concept of Mixture of Experts into the routing mechanism and propose an efficient fine-tuning method called Mixture of Routers (MoR). It employs multiple sub-routers for joint selection and uses a learnable main router to determine the weights of the sub-routers. The results show that MoR outperforms baseline models on most tasks, achieving an average performance improvement of 1%. MoR can serve as a plug-and-play, parameter-efficient fine-tuning method suitable for a wide range of applications. Our code is available here: https://anonymous.4open.science/r/MoR-DFC6 . Figure 1: A schematic diagram of the MoR plugin, where the green expert represents the correct expert. MoR effectively corrects misallocations through joint multi-routing assignments. 1 Introduction Large Language Models (LLMs) have gradually become the cornerstone of natural language processing (NLP) Devlin et al. 2019 ; Liu et al. 2021 ; He et al. 2021 ; Radford et al. 2019 . As model parameters increase, LLMs demonstrate impressive emergent abilities and transfer learning capabilities Wei et al. 2022 ; Chowdhery et al. 2023 ; Zhang et al. 2024 ; Jiang et al. 2024 . However, the computational resources required for full fine-tuning are enormous Lv et al. 2024 , and more and more research is focusing on parameter-efficient fine-tuning (PEFT) Mangrulkar et al. 2022 . The main goal is to significantly reduce the resources required for fine-tuning. For example, P-tuning converts prompts into learnable embedding layers Liu et al. 2022 , while LoRA uses a set of low-rank matrices to learn incremental updates Hu et al. 2022 . DCFT Zhang et al. 2025 further reduces the number of parameters by deconvolution. Despite the high efficiency of PEFT methods, their fine-tuning performance often falls short of meeting the increasingly complex demands of downstream tasks. Mixture-of-Experts (MoE) is designed to improve overall model performance by integrating the advantages of multiple expert networks Jiang et al. 2024 . The core idea of this approach is that different expert networks can specialize in handling different subsets or features of the data, while a gating mechanism is responsible for determining which expert should process each input Jacobs et al. 1991 ; Shazeer et al. 2017a . The model typically includes multiple experts and a gating network. The task of the gating network is to evaluate the relevance of each expert based on the input data and dynamically allocate the input to the expert most suited to handle that data. This dynamic allocation mechanism enables MoE models to be more flexible and efficient, especially when dealing with large-scale and diverse datasets Lepikhin et al. 2021 ; Du et al. 2022 ; Xue et al. 2021 ; Zuo et al. 2022 . Moreover, the MoE architecture offers higher computational efficiency and scalability. By parallelizing the computation across multiple experts, MoE can optimize resource usage, accelerating both training and inference speed. This is particularly important in scenarios that require processing massive datasets. Recent research has shown that combining PEFT and MoE allows for the advantages of both methods Zadouri et al. 2024 ; Liu et al. 2024 ; Dou et al. 2023 . LoRAMoE Dou et al. 2023 is a plugin version of MoE, learns multiple sets of low-rank matrices as experts and uses a softmax layer as a router to compute the input data for each expert. During training, the pre-trained weights are kept frozen, and only the experts and the router are trained. MoLA Gao et al. 2024 further investigates the number of experts at different layers, assigning fewer experts to lower layers and increasing the number of experts as the layer depth increases. While it demonstrates impressive results, recent studies have found issues with the MoE routing mechanism, including incorrect assignments and imbalances in expert allocation Shazeer et al. 2017b ; Fedus et al. 2022 . In system design, the principles of Redundancy and Fault Tolerance Theory emphasize the importance of using multiple components to enhance reliability and robustness. By introducing redundancy, systems can mitigate the impact of individual component failures and improve overall performance. Inspired by this theory, we propose a new parameter-efficient MoE method to address the aforementioned issues. Specifically, our approach employs multiple sub-routers for joint decision-making, where each sub-router contributes to the final decision, thereby reducing the risk of errors from any single sub-router. A main router is then used to select the top-r sub-routers based on their scores, ensuring that only the most reliable decisions are prioritized. By adjusting the number of sub-routers, MoR can flexibly adapt to tasks of varying complexity. Ultimately, the weighted cooperation of the sub-routers determines the scores for each expert, and the top-k experts with the highest combined scores are selected for the final inference. We conducted experiments on six benchmarks, including NLP and Commonsense Reasoning (CR) tasks, to demonstrate the effectiveness of MoR. Our main contributions are as follows: \bullet We propose a new fine-tuning method called MoR, which selects expert models through multiple sub-routers and uses a main router to determine the selection of sub-routers. MoR can replace the router layer in MoE-style models, thereby making it a plug-and-play and parameter-efficient solution. \bullet We conduct numerous experiments on Llama2-7B to validate the effectiveness of MoR. We compare it against benchmarks across six different tasks, and MoR outperforms in most of them. Figure 2: Here is a schematic illustration of MoR. On the left is a schematic diagram of the integration of LoRA and MoE, incorporating LoRA as an expert into the MoE model. On the right is the MoR plugin we propose, which can flexibly replace the router layer in MoE-style models. 2 Related Work When performing SFT tasks, full fine-tuning not only requires substantial computational and storage resources but can also lead to catastrophic forgetting. In contrast, PEFT Mangrulkar et al. 2022 achieves similar results to full fine-tuning by freezing most of the model parameters and training only a small subset of them. Low-Rank Approximation Hu et al. 2022 is a popular and efficient fine-tuning method for LLMs, dubbed as LoRA. It utilizes low-rank approximation theory to effectively adjust the model’s behavior with smaller parameter increments. The forward formula is as follows: W=W^{(0)}+\Delta=W^{(0)}+BA, (1) where \Delta\in\mathbb{R}^{din\times dout} , A\in\mathbb{R}^{r\times dout} , and B\in\mathbb{R}^{din\times r} , with r^{\in(din,dout)} . The dimensions of din and dout are the same as those of the pre-trained matrix W . Although LoRA significantly reduces the number of parameters, its impact on SFT performance is noticeable. The integration of MoE and LoRA represents a notable trend in

Read the complete text