Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts

Sagi Ahrac, Noya HochwaldMor Geva Blavatnik School of Computer Science and AI, Tel Aviv University {sagiahrac@mail, noyahochwald@mail, morgeva@tauex}.tau.ac.il · 2026

Read Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts by Sagi Ahrac, Noya HochwaldMor Geva Blavatnik School of Computer Science and AI, Tel Aviv University {sagiahrac@mail, noyahochwald@mail, morgeva@tauex}.tau.ac.il (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12476.txt.

Introduction

Ahrac, Hochwald and Geva uncover a geometric link between routers and experts in sparse expert models. For one token, router weights for the chosen expert and expert weights that process it share an input direction for their gradients, differing only in scale. Matched router-expert directions thus build up the same routed token history. In practice too, a one-billion-parameter model trained from scratch shows higher router scores predicting stronger expert activation. Auxiliary load-balancing losses, they show, disturb this coupling.

Catalog record

CREATORSagi Ahrac, Noya HochwaldMor Geva Blavatnik School of Computer Science and AI, Tel Aviv University {sagiahrac@mail, noyahochwald@mail, morgeva@tauex}.tau.ac.il
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41017
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12476

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts Sagi Ahrac, Noya HochwaldMor Geva Blavatnik School of Computer Science and AI, Tel Aviv University {sagiahrac@mail, noyahochwald@mail, morgeva@tauex}.tau.ac.il Abstract Sparse Mixture-of-Experts (SMoE) models enable scaling language models efficiently, but training them remains challenging, as routing can collapse onto few experts and auxiliary load-balancing losses can reduce specialization. Motivated by these hurdles, we study how routing decisions in SMoEs are formed mechanistically. First, we reveal a geometric coupling between routers and their corresponding experts. For a given token, the router weights for the selected expert and the expert weights processing it receive gradients along the same input direction, differing only in scalar coefficients. Thus, matched router–expert directions accumulate the same routed token history. This theoretical coupling also appears empirically in routing dynamics. In a 1 1 B SMoE trained from scratch, higher router scores predict stronger expert neuron activations, showing that routing decisions are mirrored inside the selected expert. Next, we analyze the effects of auxiliary load balancing on the router–expert geometric coupling, showing that such losses break this structure by spreading input-directed gradients across router weights, making distinct router directions nearly three times more similar to each other. Last, we demonstrate the centrality of geometric coupling for effective routing with a parameter-free online K-Means router, in which each expert maintains a running average of the hidden states routed to it and tokens are assigned based on cosine similarity. Compared with auxiliary-loss and loss-free balancing, this router achieves the lowest load imbalance with only a modest perplexity increase, indicating that geometric coupling captures a substantial part of what the router learns. Overall, our results explain how routers form assignment geometry that supports an effective division of labor. 1 Introduction Sparse Mixture-of-Experts (SMoE) has emerged as a leading architecture for scaling language model parameters without a proportional increase in inference latency [ 26 , 8 , 14 ] . Recent implementations, such as DeepSeek-V3 [ 6 ] and OLMoE [ 21 ] , match or outperform dense models while activating only a fraction of their total parameters. These efficiency gains stem from a division of labor in which a router , typically implemented as a small gating network, directs each input to a subset of independent expert networks [ 26 , 8 ] . Still, despite their wide adoption, training SMoEs with effective routing remains a challenge. Without intervention, routing concentrates on a shrinking subset of experts, leading to representation collapse [ 3 ] . Approaches to mitigate this apply auxiliary load-balancing losses [ 26 , 14 , 8 ] , which encourage balanced routing but often reduce expert specialization [ 30 , 6 ] . Motivated by these pathologies, we seek to understand the inner dynamics of routing in SMoEs, focusing on how routing decisions are derived mechanistically. We tackle this question from a geometric view. First, we analyze the gradients that shape both sides of the routing decision. Although routers and experts are often treated as separate modules [ 26 , 5 ] , SMoE gradients reveal a shared input-directed structure. For each assigned token, the router weights associated with the selected expert and the expert weights that process the token receive updates along the same input direction, differing only in scalar coefficients. Notably, this alignment is not a generic consequence of joint training: a shared computation graph derives both modules to receive gradients, but does not imply that their updates accumulate along shared input directions. In SMoE layers, however, the chain rule enforces this proportional form, inducing a geometric coupling where matched router–expert directions evolve as coupled accumulators of their shared routed token history. Next, we investigate if this theoretical coupling translates into empirical routing dynamics. To this end, we compare the router’s score for an expert and that expert’s neuron activations in response to the same token. In a 1 B SMoE trained from scratch for approximately 50 B tokens, we find that experts ranked higher by the router consistently exhibit stronger activations than experts not selected by the router. This shows that routing decisions are not merely an external assignment, but are mirrored in the computation of the selected expert, thus providing empirical evidence of the geometric coupling between the router and experts. Having established the router–expert geometric coupling, we revisit common routing instabilities and load-balancing side effects. Specifically, we study how the router’s geometry is influenced by the common auxiliary load-balancing loss employed in existing models [ 26 , 14 , 8 ] . This loss encourages balanced routing by penalizing uneven expert load. From a theoretical point of view, this optimization sends input-directed gradients to every router weight vector on every token, regardless of which experts were chosen. This is expected to unify different router directions over training, weakening the geometric coupling between the router and the experts. We evaluate this prediction by training two 1 B SMoEs that differ only in the balancing rule, finding that distinct router weight vectors become nearly three times more similar with the auxiliary loss than without it. These results show that auxiliary balancing regulates expert usage, yet breaks the geometric coupling, unifying expert-specific directions and eroding the specialization that coupling produces. Figure 1: Router–expert geometric coupling in SMoEs. The router scores a hidden state \mathbf{x} and selects a sparse set of top- K experts. For each selected expert, the matched router direction and expert input-side weights receive backpropagation updates proportional to the same hidden-state direction \mathbf{x} . Repeated updates make matched router–expert pairs accumulate a common routed-token history, which is read out at inference as higher router scores predicting stronger gate-neuron activations. Last, we demonstrate the centrality of geometric coupling in routing by evaluating a parameter-free centroid router that follows this natural coupling. Concretely, if router weights learn to summarize the hidden states assigned to each expert, then explicit summaries may recover much of their role. Following this idea, we implement the centroid router using online K-Means [ 19 ] . Each expert maintains a centroid of its assigned hidden states, and tokens are routed by cosine similarity to these centroids. On our 1 B SMoE setup, this router achieves the lowest load imbalance among all variants while maintaining comparable perplexity to the learned loss-free router. This indicates that geometric coupling is not only a byproduct of training, but a substantial component of what routers learn. In conclusion, our work makes the following contributions: 1. We reveal a geometric coupling between routers and experts in SMoEs, where matched router–expert pairs receive gradient updates along the same routed-token directions and evolve as coupled accumulators of their shared token history. 2. We show empirically that router scores are reflected in the selected expert’s internal computation, with higher-ranked experts exhibiting stronger activations for the same tokens. 3. We show that common auxiliary load-balancing losses act directly on router geometry, injecting input-directed gradients into every router weight vector regardless of selection and making distinct router directions nearly three times more similar. 4. Motivated by coupling, we instantiate a simple online K-Means router in which non-learnable EMA centroids replace learned router weights, and tokens are routed by cosine similarity with sign-updated biases. This router achieves the lowest load imbalance among our

Read the complete text