Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training

Charafeddine Mouzouni · 2026

Read Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training by Charafeddine Mouzouni (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04230.txt.

Introduction

Mouzouni tracks how expert routing shifts across training in two open mixture-of-experts models. A single congestion measure reveals three stages. First comes an early surge toward balanced load. Then experts specialize while balance holds steady. Finally balance gives way to quality as experts diverge. The pattern, hidden in finished models, shows early training favouring balance and late training favouring quality.

Catalog record

CREATORCharafeddine Mouzouni
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52109
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.04230

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training Charafeddine Mouzouni Abstract . We model Mixture-of-Experts (MoE) token routing as a congestion game with a single effective parameter—the congestion coefficient γ eff \gamma_{\mathrm{eff}} —that quantifies the balance-quality tradeoff. Tracking γ eff \gamma_{\mathrm{eff}} across training checkpoints of two open-source MoE models—OLMoE-1B-7B (20 checkpoints, with dense sampling in the surge region) and OpenMoE-8B (6 checkpoints)—reveals a three-phase trajectory: a surge phase where the router learns to balance load ( γ eff \gamma_{\mathrm{eff}} : 14 → 36 14\to 36 – 39 39 , peaking in the step 30K–40K region), a stabilization phase where experts specialize under steady balance ( B 0 B_{0} : 2.4 → 2.3 2.4\to 2.3 , steps 100K–400K), and a relaxation phase where the router trades balance for quality as experts differentiate ( γ eff \gamma_{\mathrm{eff}} : 27 → 9 27\to 9 , steps 400K–1.2M). This non-monotone trajectory—invisible to post-hoc analysis of converged models—reveals that early MoE training prioritizes balance while late training prioritizes quality. The theoretical framework is honest about its limits: the single-type equilibrium reduces to temperature-scaled softmax (held-out L 1 L^{1} : MFG = 0.199 =0.199 vs. softmax = 0.200 =0.200 ). The game is not a better predictor; it reveals what the temperature means and, critically, how that temperature evolves. Annealing checkpoints confirm that the three phases are pretraining-specific: γ eff \gamma_{\mathrm{eff}} is stable during fine-tuning. We complement the dynamics with an effective congestion decomposition ( γ eff = γ explicit + γ implicit \gamma_{\mathrm{eff}}=\gamma_{\mathrm{explicit}}+\gamma_{\mathrm{implicit}} ), a multi-type extension that improves load prediction via token clustering on all 16 layers (mean: 30 % 30\% ; robust to cluster count K = 2 , 4 , 8 K=2,4,8 ), and scope diagnostics ( K / M K/M , ε l \varepsilon_{l} ) that characterize where the per-layer model applies. All confidence intervals are from bootstrap resampling over 50 independent text batches. Code and data: https://github.com/Cmouzouni/three-phases-moe . 1. Introduction Mixture-of-Experts (MoE) architectures scale model capacity by routing each token to a subset of specialized expert networks [ 1 , 2 , 3 ] . The central engineering challenge is load balancing : without intervention, tokens concentrate on a few high-quality experts, leaving the rest idle. The standard remedy is the auxiliary balance loss [ 2 ] , which penalizes load concentration through a tunable coefficient \alpha . Variants include bias-based balancing [ 6 ] , capacity factors [ 3 ] , and expert-choice routing [ 5 ] . Each is effective in practice. None explains how the balance-quality tradeoff evolves during training. We observe that MoE routing is structurally a congestion game [ 15 ] . Tokens are players, experts are resources, expert quality determines individual payoffs, and load imbalance imposes congestion costs. When the token count is large ( N=2048 – 32768 in practice), the game admits a mean-field limit with a single effective parameter: the congestion coefficient \gamma , which quantifies the strength of the quality-balance tradeoff. What the theory reveals—and what it does not. We prove that the single-type mean-field game (MFG) equilibrium reduces to temperature-scaled softmax for well-balanced models (Theorem 2.4 ). Empirically, the two are indistinguishable: on OLMoE-1B-7B [ 9 ] , the MFG achieves held-out L^{1}=0.199 versus softmax L^{1}=0.200 . The game does not outperform softmax as a load predictor. It tells us why softmax arises (unique equilibrium of a potential game) and what the temperature means (the congestion coefficient). The value of the game-theoretic lens is not in static prediction. It is in dynamics. The three-phase trajectory. By fitting \gamma_{\mathrm{eff}} at each of 20 training checkpoints of OLMoE-1B-7B (50 texts per checkpoint, bootstrap confidence intervals), we discover that \gamma_{\mathrm{eff}} follows a characteristic non-monotone trajectory: Phase 1. Surge (steps 5K–50K). \gamma_{\mathrm{eff}} rises from 13.7 to a peak of 36 – 39 at steps 30K–40K. Routing entropy climbs from 0.923 to 0.974 . Phase 2. Stabilization (steps 100K–400K). The effective congestion plateaus at \gamma_{\mathrm{eff}}\approx 24 – 28 while experts specialize underneath: the quality spread B_{0} drops from 2.41 to 2.25 . The router has found its operating point for balance; expert learning proceeds within this constraint. Phase 3. Relaxation (steps 400K–1.2M). As expert roles solidify, the router loosens its balance enforcement. \gamma_{\mathrm{eff}} declines from 26.6\,[25.0,28.4] to 8.5\,[6.7,11.4] . The model trades balance for quality: experts have differentiated enough that the router can afford selectivity. This inverted-U trajectory is the paper’s central finding. It is invisible to any analysis of a converged model and reveals a fundamental tension: the early optimizer prioritizes balance, the late optimizer prioritizes quality. The transition between these regimes is governed by the anti-concentration threshold \gamma_{c}=MB_{0}/(M-1) . The pattern is not an artifact of layer-averaging: per-layer analysis shows 12 of 16 layers individually exhibit the surge pattern (early peak >1.5\times start) and 10 of 16 show relaxation (final <0.6\times mid-peak). Layers 12–15 never develop congestion structure ( \hat{\gamma}\to 0 throughout training), consistent with the mean-field assumption breaking down for late layers where token representations are most differentiated. Supporting contributions. Beyond the dynamics, three results complement the main finding: C1. Effective congestion decomposition (Section 3.2 ). The fitted \gamma_{\mathrm{eff}} decomposes as \gamma_{\mathrm{explicit}}+\gamma_{\mathrm{implicit}} , where \gamma_{\mathrm{explicit}}=\alpha M comes from the auxiliary loss and \gamma_{\mathrm{implicit}} captures balance internalized by training. At convergence, \gamma_{\mathrm{eff}}=8.5 on average while \gamma_{\mathrm{explicit}}=0.64 : the optimizer internalizes 13\times more effective congestion than the explicit loss provides. C2. Multi-type MFG (Section 4 ). A K -type extension models token heterogeneity: each type has its own quality vector while all types share the congestion signal. This goes beyond softmax-with-temperature by introducing population structure. The multi-type equilibrium improves load prediction on all 16 layers (mean improvement: 30%, early layers 36%, late layers 26%). The result is robust to cluster count: K=2 wins on 14/16 layers, K=4 on 15/16, K=8 on 14/16. C3. Scope diagnostics (Section 5 ). The top- K approximation bound shows the MFG error scales with 1-K/M . The continuation spread \varepsilon_{l} predicts per-layer fit quality ( r=0.63 , p=0.012 ). Together, these characterize where the per-layer model applies and where it breaks down. Outline. Section 2 develops the congestion game model. Section 3 defines effective congestion and presents the three-phase dynamics. Section 4 develops the multi-type extension. Section 5 collects the scope characterization theory. Section 6 presents the full empirical analysis. Section 7 discusses related work. Section 8 discusses implications and limitations. 2. The Congestion Game Model 2.1. Mixture-of-Experts routing A Mixture-of-Experts layer consists of M expert networks \{E_{1},\ldots,E_{M}\} and a gating (router) network. Given an input token x\in{\mathbb{R}}^{d} , the router computes scores s_{i}(x)=w_{i}^{\top}x+b_{i} for each expert i and selects the top- K experts by score. The output is (2.1) y=\sum_{i\in\text{Top-}K}g_{i}(x)\cdot E_{i}(x), where g_{i}(x)=\mathrm{softmax}(s(x))_{i} restricted to the selected experts. The dominant load-balancing mechanism is the auxiliary balance loss [ 2 ] : L_{\mathrm{aux}}=\alpha M\sum_{i=1}^{M}f_{i}P_{i} , where f_{i} is the fraction of tokens dispatched to expert i , P_{i} is the average router probability, M is the number of experts, and \alpha is a tunable coefficient. All balancing mechanisms share a common structure: they penalize load imbalance, trading expert quality for utilization. 2.2. The mean-field game formulation We map MoE routing to a mean-field game on the finite state space \{1,\ldots,M\} . Tokens are agents, experts

Read the complete text