Self-Routing: Parameter-Free Expert Routing from Hidden States
Jama Hussein Mohamud, Drew Wagner, Mirco Ravanelli, [0.5em] Mila - Quebec AI Institute · 2026
Read Self-Routing: Parameter-Free Expert Routing from Hidden States by Jama Hussein Mohamud, Drew Wagner, Mirco Ravanelli, [0.5em] Mila - Quebec AI Institute (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00421.txt.
Introduction
Mohamud and colleagues ask whether mixture-of-experts models really need a trained router. In these layers, each token wakes just a few experts, picked by a learned router from the token's hidden state. Self-Routing removes that router projection and instead reads expert scores straight from a chosen slice of the hidden state, leaving the rest of the layer untouched. Tests on language modelling across expert counts and model sizes, plus ImageNet-1K classification, compare it against the learned-router baseline, random routing, and dense models.
Catalog record
| CREATOR | Jama Hussein Mohamud, Drew Wagner, Mirco Ravanelli, [0.5em] Mila - Quebec AI Institute |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25243 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.00421
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-Routing: Parameter-Free Expert Routing from Hidden States Jama Hussein Mohamud, Drew Wagner, Mirco Ravanelli, [0.5em] Mila - Quebec AI Institute Abstract Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token, and typically rely on a learned router to map hidden states to expert assignments. In this work, we ask whether a dedicated learned router is strictly necessary in the MoE settings we study. We propose Self-Routing , a parameter-free routing mechanism that uses a designated subspace of the token hidden state directly as expert logits, eliminating the router projection entirely while leaving the rest of the MoE layer unchanged. We evaluate Self-Routing on GPT-2-scale language modeling and ImageNet-1K classification by comparing it against a standard learned router, random-routing baselines, and dense non-MoE baselines. Our results show that Self-Routing remains competitive with the learned-router baseline while removing all dedicated routing parameters, and yields more balanced expert utilization, with about 17% higher average normalized routing entropy and no explicit load-balancing loss. On ImageNet-1K with DeiT-S/16, Self-Routing also slightly improves over the corresponding learned-router MoE. These findings suggest that effective MoE routing can emerge from the hidden representation itself without requiring a separate learned router module. 1 Introduction Mixture-of-Experts (MoE) layers have become a standard mechanism for increasing model capacity without activating all parameters for every token ( 18 ; 7 ; 11 ; 3 ) . By replacing a dense feed-forward block with a set of experts and activating only a small subset per token, MoE models can scale parameter count while keeping the per-token compute relatively small. A central component of this design is the router , which maps token representations to expert assignments. In most modern MoE architectures, routing is performed by a learned projection from the hidden state to expert logits. This design is simple and effective, and has become the default choice in practice. At the same time, it introduces a dedicated routing module in every MoE layer, together with additional parameters, implementation complexity, and often router-specific training considerations such as load balancing. In this work, we study whether the hidden representation already exposes enough information to route tokens effectively. We therefore propose a simple alternative that we call Self-Routing . Instead of learning a router matrix, Self-Routing uses a designated subspace of the token hidden state directly as expert logits. In our implementation, the last N coordinates of the hidden state are used to route among N experts, eliminating router parameters entirely while preserving the standard top- k dispatch and aggregation mechanism. Prior work has examined the role of routing in sparse models from several directions, including learned versus random routing ( 5 ) , hash-based routing ( 16 ) , and frozen or randomly initialized routers variants ( 6 ) . We discuss these connections, along with adjacent parameter-free sparse-routing ideas, in Section 5 . Our goal is not to argue that learned routers are unnecessary in general, nor to claim a universal replacement for standard MoE routing. Rather, we ask a narrower empirical question: in the MoE settings studied here, how much does a dedicated learned router actually matter? To answer this, we compare Self-Routing against a standard learned router, random routing baselines, and dense non-MoE baselines in controlled GPT-2-scale language-model experiments, and we additionally report an ImageNet-1K classification comparison with DeiT-S/16. We evaluate not only downstream quality, but also routing behavior through expert utilization and load-balancing behavior. Our results suggest that a dedicated learned router may be less essential than standard practice assumes. Self-Routing remains competitive with the learned-router baseline on several downstream evaluations while removing all router parameters. Beyond task quality, Self-Routing also yields more balanced expert utilization, achieving higher routing entropy across layers without any explicit load-balancing loss. At the same time, the comparison with random routing shows that expert assignment is not arbitrary: content-aware routing still matters, but it need not necessarily be mediated by a separate learned projection. Our ImageNet-1K result further suggests that the same idea can remain effective in a standard vision classification setting. Taken together, these findings support the view that MoE routing may depend as much on representational organization as on the explicit form of the router itself. 2 Method ⬇ class LearnedGate ( BaseGate ): def __init__ ( self , model_dim , num_experts ): super (). __init__ ( model_dim , num_experts ) self . wg = nn . Linear ( model_dim , num_experts , bias = False ) def compute_logits ( self , x ): return F . linear ( x , self . wg . weight . float ()) ⬇ class SelfRouteGate ( BaseGate ): # Zero learned parameters: # Routing is encoded in the # last N dimensions. def compute_logits ( self , x ): return x [..., - self . num_experts :] Figure 1: Learned routing vs. Self-Routing. Self-Routing replaces the learned projection with a direct hidden-state readout while leaving the rest of the MoE top- k dispatch unchanged. 2.1 Background: Mixture-of-Experts Routing Mixture-of-Experts (MoE) layers replace a dense feed-forward block with a collection of N expert networks together with a routing mechanism that selects which experts process each token. Given a token hidden state \mathbf{h}\in\mathbb{R}^{H} , the router produces a score for each expert, and only the top- k experts are activated. This allows the model to increase parameter count while keeping the per-token compute closer to that of a sparse subset of experts rather than all experts ( 18 ; 7 ; 11 ) . Formally, let \{E_{i}\}_{i=1}^{N} denote the experts. A standard MoE layer first computes routing logits \mathbf{z}=g(\mathbf{h})\in\mathbb{R}^{N}, (1) where g is the router. The top- k experts according to \mathbf{z} are selected, and their normalized routing weights are obtained by applying a softmax over the selected experts only. If \mathcal{T}_{k}(\mathbf{z}) denotes the indices of the top- k logits, the MoE output is \mathrm{MoE}(\mathbf{h})=\sum_{i\in\mathcal{T}_{k}(\mathbf{z})}p_{i}(\mathbf{h})\,E_{i}(\mathbf{h}), (2) where p_{i}(\mathbf{h}) is the normalized routing weight for expert i . In practice, the experts are typically independent feed-forward networks with identical architecture. In most modern MoE models, the router is implemented as a learned linear projection \mathbf{z}=\mathbf{h}W_{r}, (3) where W_{r}\in\mathbb{R}^{H\times N} is learned jointly with the rest of the model. This design is simple and effective, and is the default choice in systems such as Switch Transformer and Mixtral ( 7 ; 11 ) . However, it also introduces dedicated routing parameters in every MoE layer. At scale, these router parameters are small relative to the full model, but they are not negligible, and they add an additional mechanism that must be learned and tuned. A central challenge in sparse MoE training is expert imbalance: without additional regularization, the router may overuse a small subset of experts while leaving others underutilized. Standard MoE training therefore often adds an auxiliary load-balancing objective ( 7 ) . Let f_{i} denote the fraction of tokens routed to expert i , and let p_{i} denote the mean routing probability assigned to that expert across a batch. A common balancing loss is \mathcal{L}_{\text{balance}}=N\sum_{i=1}^{N}f_{i}p_{i}. (4) This term encourages the router to distribute traffic more evenly across experts. In practice, the main training objective is the task loss plus, when used, a weighted load-balancing term. The standard learned router computes expert logits