Ro uting M anifold A lignment Improves Generalization of Mixture-of-Experts LLMs

Zhongyang Li, Ziyue Li, Tianyi Zhou · 2025

Read Ro uting M anifold A lignment Improves Generalization of Mixture-of-Experts LLMs by Zhongyang Li, Ziyue Li, Tianyi Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07419.txt.

Introduction

Mixture-of-experts models switch on only a few parts for each input, saving compute. Zhongyang Li, Ziyue Li, and Tianyi Zhou show that the switching decisions, called routing, are often far from ideal. Broad tests reveal accuracy gaps of ten to twenty percent versus the best possible routing.

Their fix, Routing Manifold Alignment, nudges routing weights to resemble task embedding patterns. Only the routers are lightly retrained while the rest of the model stays frozen. That small change closes much of the gap and improves performance on new tasks. The result suggests smarter routing, not bigger models, can unlock gains.

Catalog record

CREATORZhongyang Li, Ziyue Li, Tianyi Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07419

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ro uting M anifold A lignment Improves Generalization of Mixture-of-Experts LLMs Zhongyang Li, Ziyue Li, Tianyi Zhou Abstract Sparse Mixture-of-Experts (MoE) have been widely adopted in recent large language models since it can efficiently scale up the model capability without increasing the inference cost. However, evaluations on broad downstream tasks reveal a consistent suboptimality of the routers in existing MoE LLMs, which results in a severe performance gap (e.g., 10-20% in accuracy) to the optimal routing. In this paper, we show that aligning the manifold of routing weights with that of task embedding can effectively reduce the gap and improve MoE LLMs’ generalization performance. Our method, “ Ro uting M anifold A lignment (RoMA)”, introduces an additional manifold regularization term in the post-training objective and only requires lightweight finetuning of routers (with other parameters frozen). Specifically, the regularization encourages the routing weights of each sample to be close to those of its successful neighbors (whose routing weights lead to correct answers) in a task embedding space. Consequently, samples targeting similar tasks will share similar expert choices across layers. Building such bindings between tasks and experts over different samples is essential to achieve better generalization. Moreover, RoMA demonstrates the advantage of unifying the task understanding (by embedding models) with solution generation (by MoE LLMs). In experiments, we finetune routers in OLMoE, DeepSeekMoE, and Qwen3-MoE using RoMA. Evaluations on diverse benchmarks and extensive comparisons with baselines show the substantial improvement brought by RoMA. 1 Introduction Sparse Mixture-of-Experts (MoE) have emerged as a cornerstone architecture in scaling large language models (LLMs), enabling significant capacity increases without proportional computational overhead during inference ( 13 ; 18 ) . At the core of this mechanism lies the router, which assigns input tokens to a small subset of experts through routing weights in each layer. Despite the small portion of router parameters in MoE LLMs (e.g., 0.03% in a 7B model), they are the key to the success of expert usage in MoE ( 32 ) . However, evaluations across broad downstream tasks reveal that routers in existing MoE LLMs cause major failures. As shown in Table 1 , their suboptimal routing weights lead to a performance gap of 10-20% in accuracy when compared to the optimal routing weights (oracle). This gap underscores a major untapped bottleneck in MoE LLMs, suggesting that improving routing is critical to boosting MoE LLMs’ generalization performance on downstream tasks. Our analysis further investigates the reasons behind the performance gap and the poor generalization capabilities of pretrained routers. As illustrated in Figures 3 (a) and (b), pretrained routers assign semantically similar samples in the task embedding space to distinct experts with dramatically different routing weights. Such misalignment between the task embedding manifold and routing weight manifold hinders effective knowledge sharing across tasks and underutilizes the collective expertise of the experts. This misalignment between the targeted tasks and the assigned experts undermines the generalization of MoE and its core principle, which is to leverage specialized experts, share skills, and transfer knowledge for related inputs. A natural solution is to finetune the routers. Existing approaches, such as Dense BP ( 27 ) developed more effective pretraining objectives for routers but do not address the manifold misalignment between the targeted tasks and the routing weights across samples. Figure 1: RoMA on OLMoE-7B-A1B vs. 7-34B dense LLMs across eight benchmarks. RoMA leads to 7-15% accuracy improvement, consistently outperforming all models over eight benchmarks, demonstrating the effectiveness of post-training by RoMA. This limitation motivates our exploration of incorporating manifold alignment into the fine-tuning objective. Specifically, our manifold alignment aims to enforce the consistency between task understanding (encoded by an embedding model) and task solving in an MoE LLM (encoded by the routing weights). As illustrated in Figure 2 , for each training sample, in addition to minimizing its loss defined on the output, we encourage its intermediate layers’ routing weights to move to those of its “ successful neighbors ” (samples with correct MoE predictions) in the task embedding space. These neighbors are weighted by their similarity to the sample. This training objective can be formulated as manifold regularization ( 2 ) , a well-established technique in machine learning that aims to preserve the local neighborhood structure of high-dimensional inputs on the manifold of low-dimensional representations or outputs. Unlike its original setting, we apply such a regularization to the routing weights across MoE layers rather than the final outputs, and establish coherent bindings between the expert choices (weights) and the task embedding instead of the raw inputs. Figure 2: Overview of RoMA. RoMA finetunes routers in MoE LLM (bottom, yellow) with a training objective defined on each sample (x_{i},y_{i}) , which is composed of (1) the task loss \mathcal{L}_{\text{task}}(i) defined on the model output f(x_{i},r_{i}) ; and (2) the manifold alignment regularization \mathcal{L}_{\text{manifold}}(i) , which aligns the manifolds of routing weights (right, green) and the task embedding (left, blue). It improves MoE’s generalization by unifying solution generation in MoE with task understanding. To this end, we propose “ Ro uting M anifold A lignment (RoMA)”, a router post-training method that aligns the manifold of routing weights with task embeddings through lightweight fine-tuning of a few routers in MoE LLMs. RoMA introduces a manifold regularization term to the training objective that encourages routing weights of each sample to approximate those of its successful neighbors with similar task embedding, thereby promoting consistent expert selection for semantically related inputs. Extensive experiments on three recent MoE LLMs (OLMoE, DeepSeekMoE, Qwen3-MoE) demonstrate that RoMA brings substantial improvements (7-15% in accuracy) across diverse benchmarks and outperforms SOTA routing methods, as shown in Figure 1 , by merely finetuning 0.0095\% parameters of the base model, without affecting inference cost. Notably, RoMA-finetuned MoE LLMs with only 1-3B active parameters achieve competitive or superior performance over much larger dense models with 34B parameters. We conduct comprehensive ablation studies that further investigate the effects of key designs in RoMA, including layer selection, neighborhood configuration, and regularization strategies, validating the effectiveness of RoMA in bridging the performance gap between pretrained routers and optimal routing for MoE LLMs. 2 Related Work MoE LLMs Mixture of Experts (MoE) architectures have been extensively incorporated into large language models (LLMs) to enhance computational efficiency and task‑specific specialization ( 32 ) . Recent work such as OLMoE ( 26 ) and DeepSeekMoE ( 9 ) demonstrate the effectiveness of sparse MoE layers in reducing active parameters while maintaining model capacity. These MoE models fundamentally rely on routers to determine expert selection, typically employing token-choice routing that selectively activates subsets of experts for each input token ( 13 ; 18 ) . However, the quality of these routing decisions remains a critical bottleneck. Our study shows current routers often produce suboptimal routing weights that fail to fully leverage expert specialization, resulting in load imbalance and expert underutilization. Manifold Regularization of LLMs Recent work reveals that LLM embeddings exhibit stratified manifold structures with varying dimensions across semantic domains ( 21 ; 28 ) . While traditional manifold regularization assumes smooth global structures ( 2 ) , LLMs require more sophisticated approaches. Methods like I-STAR ( 29 ) control isotropy in embedding spaces, while CROW ( 25 ) enforces consistency across layers. However, these techniques

Read the complete text