Teacher-Guided Routing for Sparse Vision Mixture-of-Experts
Masahiro Kada 1 Ryota Yoshihashi 1 Satoshi Ikehata 2,3 Rei Kawakami 1 Ikuro Sato 1,2 1 Institute of Science Tokyo 2 DENSO IT Laboratory 3 National Institute of Informatics Tokyo, Japan [email protected] · 2026
Read Teacher-Guided Routing for Sparse Vision Mixture-of-Experts by Masahiro Kada 1 Ryota Yoshihashi 1 Satoshi Ikehata 2,3 Rei Kawakami 1 Ikuro Sato 1,2 1 Institute of Science Tokyo 2 DENSO IT Laboratory 3 National Institute of Informatics Tokyo, Japan [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21330.txt.
Introduction
Masahiro Kada and four colleagues stabilize sparse mixture-of-experts routing for vision models, where the router learns only through experts chosen in the forward pass. That gradient blocking leaves unselected routes uninformative, breeding unstable, fluctuating assignments. Their TGR-MoE method builds a teacher router from a pretrained dense teacher's intermediate representations and uses its outputs as pseudo-supervision for the student router. On ImageNet-1K and CIFAR-100 it improves accuracy and routing consistency, holding stable even under highly sparse settings.
Catalog record
| CREATOR | Masahiro Kada 1 Ryota Yoshihashi 1 Satoshi Ikehata 2,3 Rei Kawakami 1 Ikuro Sato 1,2 1 Institute of Science Tokyo 2 DENSO IT Laboratory 3 National Institute of Informatics Tokyo, Japan [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39116 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21330
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Teacher-Guided Routing for Sparse Vision Mixture-of-Experts Masahiro Kada 1 Ryota Yoshihashi 1 Satoshi Ikehata 2,3 Rei Kawakami 1 Ikuro Sato 1,2 1 Institute of Science Tokyo 2 DENSO IT Laboratory 3 National Institute of Informatics Tokyo, Japan [email protected] Abstract Recent progress in deep learning has been driven by increasingly large-scale models, but the resulting computational cost has become a critical bottleneck. Sparse Mixture of Experts (MoE) offers an effective solution by activating only a small subset of experts for each input, achieving high scalability without sacrificing inference speed. Although effective, sparse MoE training exhibits characteristic optimization difficulties. Because the router receives informative gradients only through the experts selected in the forward pass, it suffers from gradient blocking and obtains little information from unselected routes. This limited, highly localized feedback makes it difficult for the router to learn appropriate expert-selection scores and often leads to unstable routing dynamics, such as fluctuating expert assignments during training. To address this issue, we propose TGR-MoE: Teacher-Guided Routing for Sparse Vision Mixture-of-Experts, a simple yet effective method that stabilizes router learning using supervision derived from a pretrained dense teacher model. TGR-MoE constructs a teacher router from the teacher’s intermediate representations and uses its routing outputs as pseudo-supervision for the student router, suppressing frequent routing fluctuations during training and enabling knowledge-guided expert selection from the early stages of training. Extensive experiments on ImageNet-1K and CIFAR-100 demonstrate that TGR consistently improves both accuracy and routing consistency, while maintaining stable training even under highly sparse configurations. 1 Introduction Sparse Mixture-of-Experts (MoE) architectures 38 enable compute-efficient scaling by activating only a small subset of experts per input, allowing model capacity to grow without a proportional increase in computation. Originally popularized in large-scale language models 8 ; 21 , MoE has recently been extended to vision models for recognition and generation tasks 33 ; 30 ; 42 ; 9 . In this work, we focus on sparse Vision MoE (VMoE) ( e.g . , 33 ; 30 ), where individual patch tokens are fed into MoE layers in parallel. Figure 1 : Comparison between standard sparse MoE routing (left) and our Teacher-Guided Routing in TGR-MoE (right). The bar plots denote the router’s per-expert routing scores. In a conventional sparse MoE, the router receives gradient feedback only from the experts it selects, leaving the majority of experts without informative signals. This limited and highly localized feedback can lead to locally optimal routing behaviors, such as expert collapse or frequent changes in expert selection. TGR-MoE introduces a global prior derived from the intermediate features of a pretrained dense teacher model. This prior provides additional supervision to the router, complementing the sparse gradient signals and encouraging more coherent routing preferences even for experts that are not selected at each step. Despite their computational advantages, sparse MoE models remain challenging to train stably 50 . Unlike dense ( i.e . , non-MoE) models, which update all parameters on every input, sparse MoE architectures activate and update only a small subset of experts and the router for each input. We argue that an important yet under-discussed challenge in this training regime is gradient blocking : as illustrated in Fig. 1 (left), because only selected experts participate in the forward and backward passes, the router receives no informative feedback from unselected routes. As a result, it is difficult for the router to learn appropriate expert-selection scores for each input, especially in the early stage of training when expert specialization has not yet emerged. This lack of informative, data-dependent supervision can easily lead to unstable routing dynamics. In particular, the expert assigned to a fixed input can vary in an excessively frequent fashion over the course of training, a phenomenon referred to as routing fluctuation 5 . Although auxiliary load-balancing losses 38 are commonly used to mitigate expert-usage imbalance, they are not sufficient to effectively suppress routing instability during training and can even amplify changes in expert assignment over the course of training. Such temporal inconsistency complicates optimization because the same sample may update different experts across iterations, even though only one or a small fixed set of experts will be used at inference. Overly frequent changes in expert assignment hinder stable expert specialization and can leave some experts undertrained, particularly in settings with a large number of experts. These considerations motivate us to provide the router with an external prior that supplies informative guidance beyond the sparsely activated routes, as illustrated in Fig. 1 (right). We propose TGR-MoE: Teacher-Guided Routing for Sparse Vision Mixture-of-Experts , a framework that stabilizes sparse MoE routing by leveraging a pretrained dense teacher. Rather than modifying the task objective, TGR-MoE attaches an auxiliary teacher router to the frozen teacher backbone. The teacher router is trained with a load-balancing loss and an entropy loss, which were formerly used as regularizers for the target router, to produce routing distributions that are both balanced across experts and sufficiently confident. The resulting teacher router yields stable routing distributions that are used to guide the target router. The MoE student learns its router by distilling these distributions: for each MoE layer, the student gating distribution is trained to match the teacher’s via a Kullback–Leibler divergence term combined with the standard task loss. Because the teacher backbone is frozen and only the lightweight teacher router is optimized, teacher routing converges quickly and remains stable, and the distillation term encourages temporally consistent, semantically meaningful expert assignments in the student. The proposed training strategy introduces a prior to the router’s softmax outputs with different abstraction levels, relieving the gradient blocking problem without freezing the target router, while no additional cost is required at test time. We evaluate TGR-MoE against the standard VMoE 33 and its variants. Across ImageNet-1K 36 pre-training and downstream tasks including CIFAR-10/100 19 and Oxford-IIIT Pets 28 , TGR-MoE consistently outperforms VMoE at all model scales (Tiny, Small, Base) under a comparable compute budget. It is competitive with, and often superior to, advanced MoE variants such as expert-choice routing 49 , SoftMoE 30 , and z-loss regularization 50 . Applying TGR-MoE during fine-tuning further yields the strongest transfer performance on CIFAR-100, indicating that explicit routing guidance remains beneficial beyond pre-training. TGR-MoE also scales more reliably with the number of experts, maintaining steady gains even in regimes where conventional VMoE saturates (Sec. 4.3 ), and routing-consistency analysis shows that it substantially reduces routing fluctuations and reaches stable expert assignments earlier than VMoE (Sec. 5.1 ). 2 Related Work Mixture of Experts. While MoE was originally introduced in the early 1990s 14 ; 27 , it has recently emerged as an efficient way to scale model capacity without increasing computation proportionally 38 ; 21 . Driven by scaling laws 18 ; 13 , MoE has been widely adopted in large language models 21 ; 7 ; 8 ; 22 ; 4 ; 16 ; 44 and vision models 33 ; 25 ; 9 ; 2 ; 29 ; 47 ; 15 ; 39 ; 17 . Despite this progress, MoE architectures face several training challenges. A well-known issue is expert-usage imbalance, where only one or a few experts receive most routing traffic; auxiliary load-balancing losses 38 ; 8 ; 33 and alternative designs such as expert-choice routing 49