Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs
Seyed Alireza Molavi, Zhan Su, Yan Hu, Peyman Sheikholharam Mashhadi, Stefan Byttner, Prayag Tiwari · 2026
Read Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs by Seyed Alireza Molavi, Zhan Su, Yan Hu, Peyman Sheikholharam Mashhadi, Stefan Byttner, Prayag Tiwari (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31413.txt.
Introduction
Molavi and five colleagues compose frozen reasoning adapters instead of mixing them softly. LoRA adapters trained apart can give one language model many domain skills, useful when training data cannot be shared. Soft mixture routing, though, rescales each adapter's update and distorts the behaviour it was trained for.
Their Hard-Routed MoR-LoRA trains each reasoning expert alone with reinforcement learning from verifiable feedback, then freezes them all. A light shared router plus a small attention adapter learn to pick exactly one expert per token, trained through a straight-through estimator. Across five benchmarks and several model families, expert behaviour survives while trainable parameters fall far below soft-routing rivals. Analysis shows soft mixtures already pile most weight on one expert, so hard picks lose little.
Catalog record
| CREATOR | Seyed Alireza Molavi, Zhan Su, Yan Hu, Peyman Sheikholharam Mashhadi, Stefan Byttner, Prayag Tiwari |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.31413
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs Seyed Alireza Molavi, Zhan Su, Yan Hu, Peyman Sheikholharam Mashhadi, Stefan Byttner, Prayag Tiwari Abstract Composing independently trained LoRA adapters into a single large language model is useful for multi-domain adaptation, especially when the original training data cannot be shared. A common approach is to use MoE-style routing over LoRA experts, but for frozen pretrained adapters, soft weighted combinations can change the unit-scale additive update under which each LoRA module was originally trained. We propose Hard-Routed MoR-LoRA , a two-stage framework for composing frozen reasoning LoRA experts through unit-scale hard selection. First, domain-specific LoRA adapters are trained independently using reinforcement learning from verifiable feedback to obtain reasoning experts. Then, all experts are frozen, reasoning traces are distilled from them, and only a lightweight shared router together with a small attention LoRA is trained for integration. The router selects exactly one expert per token using hard top-1 routing, while a straight-through estimator enables gradient-based training. Experiments across five benchmarks, multiple model scales, and additional model families show that Hard-Routed MoR-LoRA preserves expert behavior while requiring substantially fewer trainable parameters than soft-routing mixture baselines. Our analysis further shows that normalized soft mixtures often concentrate most routing mass on a single expert, suggesting that hard unit-scale routing provides a simple and efficient abstraction for frozen LoRA expert composition 1 1 1 Our code is available at github.com/sar-molavi/hard-routed-mor-lora . 1 Introduction Large language models (LLMs) have demonstrated strong performance across diverse language understanding and reasoning tasks 34 ; 35 ; 2 . However, adapting these models to specialized domains remains expensive because full fine-tuning requires substantial computation and access to training data. Parameter-efficient fine-tuning (PEFT) methods address this issue by modifying only a small subset of parameters 11 . Among them, Low-Rank Adaptation (LoRA) 12 is widely used because it adds lightweight trainable matrices while keeping the backbone frozen. In practice, domain adaptation is often done independently, where different groups train LoRA adapters on their own datasets but cannot share the original data due to privacy or regulatory constraints. Instead, they may release only the trained LoRA modules. A natural goal is therefore to compose multiple independently trained LoRA experts into a single model that supports heterogeneous tasks. Mixture-of-Experts (MoE) architectures 27 ; 8 provide a natural mechanism for such modular reuse by routing each token to selected experts. However, when the goal is to reuse frozen independently trained LoRA adapters, directly applying standard soft MoE routing introduces a scale mismatch. LoRA modules are trained assuming a unit additive update, whereas MoE combines experts using routing weights. Soft routing, therefore, blends expert updates and changes the effective function applied by each adapter. For example, LoRA-Mixer 18 addresses this by retraining the adapters with auxiliary losses, but this increases trainable parameters, requires more data, and can alter the knowledge encoded in the original experts. This challenge becomes more pronounced for instruction-tuned models with chain-of-thought (CoT) reasoning ability. Conventional fine-tuning encourages imitation of short answers and may weaken internal reasoning behavior in large models, while reinforcement learning from verifiable feedback (RLVF) can produce stronger reasoning experts by optimizing correctness rather than matching reference outputs 26 ; 10 . However, existing LoRA composition methods primarily focus on supervised multi-task fine-tuning and do not study the training and integration of experts in instruction-tuned models that exhibit CoT reasoning. In this work, we propose Hard-Routed MoR-LoRA , a framework that separates reasoning acquisition from expert selection. First, domain-specific LoRA adapters are trained independently using RLVF to obtain reasoning. Then, the experts are frozen and integrated using a hard-routed mixture that selects exactly one expert per token through a straight-through estimator (STE). We distill reasoning traces from the experts and train only a shared lightweight router and a small attention adaptation using a small amount of data (1000 samples per dataset). Hard routing preserves the original LoRA assumption by applying each expert with unit scale, enabling modular composition without retraining the experts. Experiments across five benchmarks and multiple model scales show that RLVF produces stronger reasoning experts for capable models, while conventional fine-tuning can degrade reasoning behavior in large instruction-tuned models. We further demonstrate that hard routing achieves comparable or better performance than soft-routing baselines in our setting while using substantially fewer trainable parameters and keeping the pretrained experts frozen. Analysis of routing probabilities reveals that normalized soft top- k mixtures implicitly behave like near hard top-1 selection, indicating that hard routing captures the intended behavior more directly. Our contributions are summarized as follows: • We identify a scaling mismatch that arises when soft MoE routing is applied to independently trained frozen LoRA adapters, and we use hard unit-scale routing to preserve standalone expert application without retraining experts. • We introduce a two-stage framework that trains reasoning LoRA experts using RLVF and integrates them through distillation while keeping experts frozen. • We provide empirical evaluation across model scales and tasks showing improved performance and parameter efficiency. 2 Related Work Modular LoRA adaptation. LoRA enables efficient fine-tuning by learning small low-rank updates while keeping the backbone model frozen. Although effective for adapting a model to a single task, independently trained LoRA modules are usually task-specific and difficult to reuse across different domains, especially when the original training data cannot be shared. There is a rich body of research have Several works, therefore, studiedy how to combine multiple pretrained adapters into one model 24 ; 20 ; 22 ; 31 . LoraHub 13 learns weights that merge a set of LoRA modules into a single composed adapter, while LoRA-LEGO 36 decomposes adapters into smaller rank components and reconstructs a merged adapter for cross-task reuse. These methods demonstrate that pretrained adapters can be combined without retraining, but the resulting model applies the same merged adapter to every input and cannot dynamically select different experts at the token level. Mixture-of-Experts with LoRA. MoE architectures 27 ; 8 ; 21 ; 29 increase model capacity by routing tokens to a sparse subset of experts. Instead of merging adapters into one module, MoE-based approaches keep multiple experts and select among them during inference. Recent work applies this idea to parameter-efficient tuning by treating LoRA modules as experts. MixLoRA 16 inserts multiple LoRA modules into feed-forward layers (FFN) with top- k routing. MoLE 32 learns hierarchical gating across layer-wise adapters, while SiRA 37 and GOAT 17 study alignment and sparse routing strategies. The closest to our work, LoRA-Mixer, integrates LoRA modules into attention projections and trains a lightweight router. However, soft routing produces a weighted sum of LoRA updates, whereas standalone LoRA training assumes unit-scale application. In LoRA-Mixer, this issue is addressed by further fine-tuning the experts with regularization losses, which increases the number of trainable parameters and introduces additional integration-stage computation. Instruction-tuned Reasoning and Verifiable Feedback. Instruction tuning and alignment stages shape how language models produce and structure reasoning 23 . RLVF improves reasoning by optimizing correctness using automatically checkable rewards instead of matching reference outputs 26 ; 10 . DeepSeek-R1 10 shows that strong reasoning ability can emerge from this training paradigm, and Group Relative Policy Optimization (GRPO) 26 stabilizes training without requiring a learned value critic. In our setting, adapters are added to instruction-tuned