Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing
Ryan Sun, Yixin Liu, Yonghui Wu, Lichao Sun · 2026
Read Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing by Ryan Sun, Yixin Liu, Yonghui Wu, Lichao Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11535.txt.
Introduction
Sun and co-authors propose Expert Threshold routing for language models. Token-choice routing sends each token to a fixed expert count and needs extra losses to balance loads. Each expert instead keeps a moving threshold drawn from the global token stream. A token joins an expert only when its score clears that expert's threshold. Loads balance on their own, with no auxiliary loss, and the rule stays fully causal. In pretraining, it cuts cross-entropy loss by 0.067 versus token-choice routing.
Catalog record
| CREATOR | Ryan Sun, Yixin Liu, Yonghui Wu, Lichao Sun |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63225 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11535
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing Ryan Sun, Yixin Liu, Yonghui Wu, Lichao Sun Abstract Token-choice Mixture-of-Experts (TC-MoE) routes each token to a fixed number of experts, limiting dynamic computation allocation and requiring auxiliary losses to maintain load balance. We propose Expert Threshold (ET) routing, where each expert maintains an exponential moving average (EMA) threshold estimated from the global token distribution. At both training and inference, each token is independently routed to an expert if its score exceeds the expert’s threshold, enabling dynamic computation allocation while achieving load balance without auxiliary losses. This fully causal mechanism eliminates dependence on other tokens in the batch, making it well-suited for autoregressive language modeling. In pretraining experiments scaling to 2.4B parameters on FineWeb-Edu, ET achieves 0.067 lower cross-entropy loss than TC-MoE, equivalent to reaching the same performance with 1.6× fewer tokens. † † affiliationnotice: Code available at GitHub repository . 1 Introduction Figure 1 : evaluation loss for Dense, TC, and ET. Compared to TC, ET achieves a 0.067 final loss gap (TC vs ET), or equivalently reaching same performance level with 1.6x few tokens. Token Choice (TC) Load Imbalance token input \mathrm{Top}_{G}\!\bigl(\{r_{t,i}\}_{\forall i}\bigr) \mathcal{M}_{1} \mathcal{M}_{2} \mathcal{M}_{3} \mathcal{M}_{4} Expert Choice (EC) Non Causal Seq top- k \mathcal{M}_{i} 1 sequence \mathrm{Top}_{k}\!\bigl(\{r_{t,i}\}_{\forall t\in\mathrm{seq}}\bigr) Batch top- k \mathcal{M}_{i} s_{1} s_{2} s_{3} 1 batch \mathrm{Top}_{k}\!\bigl(\{r_{t,i}\}_{\forall t\in\mathrm{batch}}\bigr) Expert Threshold (ET) Fully Causal EMA Threshold c_{i} Score r_{t,i} Token index t z_{t,i}=\mathbf{1}\{r_{t,i}>c_{i}\} routing pool size token sequence batch population Figure 2 : Illustration of TC, EC, and ET routing mechanisms and their routing pools. Left: TC routes each token independently to its top- G experts, causing load imbalance. Middle: EC has each expert select its top- k tokens from the batch, requiring access to all tokens including future ones (non-causal). Right: ET routes each token independently by comparing its score against the population’s top- (1/E) quantile estimated by an EMA-tracked threshold c_{i} , enabling fully causal routing over the population. Mixture of Experts (MoE) architectures ( 36 ; 21 ; 9 ) have emerged as a leading approach to scale language models efficiently, powering frontier models like DeepSeek-V3 ( 7 ) . By sparsely activating only a subset of expert networks per token, MoE decouples model capacity from computational cost, enabling massive parameter counts with tractable FLOPs. However, sparse routing introduces a fundamental tension: without intervention, routers tend to collapse onto a small subset of experts ( 36 ) . This harms model quality, as underutilized experts become redundant parameters that waste capacity. It also creates hardware bottlenecks under Expert Parallelism ( 21 ) , where skewed loads leave some devices idle and others overloaded. Thus, we need a routing mechanism that roughly maintains load balancing. Prior work falls into two categories. The prevalent token choice (TC) routing ( 9 ) fixes the number of experts each token selects. This sparsity constraint not only fails to address load imbalance, but further complicates the routing as it conflicts with load balancing, turning the routing into a combinatorial optimization problem. People resort to heuristics to approximate load balancing, such as auxiliary losses ( 21 ; 9 ) or PID controllers ( 43 ; 46 ) . In contrast, expert choice (EC) routing ( 56 ) relaxes the fixed computation budget per token and only enforces load balancing within a batch by selecting the top- k tokens for each expert, achieving perfect load balancing by construction while enabling dynamic computation allocation. However, EC routing fundamentally violates causality, making it unsuitable for autoregressive language models. Selecting top- k requires comparing against the entire batch that includes future positions. At training time this mechanism leaks information ( 46 ) ; at inference time future tokens simply do not exist. In this paper, we relax both per-token sparsity and per-batch load balancing, requiring only that load reaches a targeted activation rate in expectation. The resulting mechanism, Expert Threshold (ET) routing, routes each token by comparing its score to a quantile threshold tracked from each expert’s global score distribution. Because the same threshold is used at training and inference, ET routing is fully causal with no train-inference mismatch. Pretraining a 2.4B (0.56B active) language model on FineWeb-Edu, ET outperforms TC by 0.067 in cross-entropy loss while achieving near-perfect load balancing. We further show that EC’s performance improves with batch size, and that models trained with large-batch EC can perform causal inference using our threshold-based routing without retraining. 2 Preliminaries: Routing as Constrained Optimization An MoE layer replaces a dense feed-forward block with a router and GE experts. Consider a batch of N tokens with representations x_{t}\in\mathbb{R}^{d} . The router computes scores r_{t,i}=(W_{r}x_{t})_{i}, (1) collected into a matrix r\in\mathbb{R}^{N\times GE} . Based on r , a routing rule produces a binary assignment z\in\{0,1\}^{N\times GE} where z_{t,i}=1 indicates expert i is activated for token t and 0 otherwise. Each selected expert i computes an output y_{i,t}\in\mathbb{R}^{d} , weighted by a gate value p_{t,i}=\sigma(r_{t,i}) . The MoE output for token t is y_{t}=\sum_{i=1}^{GE}z_{t,i}\,p_{t,i}\,y_{i,t}. (2) The routing rule that determines z therefore controls both compute allocation and expert load balance. We formalize MoE routing as finding z that maximizes the total routing score subject to computational constraints, since higher scores indicate stronger token-expert affinity and, through the gate p_{t,i} , larger expert contributions to the output. Token Choice Routing The standard Token Choice routing goal is: \displaystyle\max_{z} \displaystyle\sum_{t=1}^{N}\sum_{i=1}^{GE}z_{t,i}r_{t,i} (3) \displaystyle\text{s.t.} \displaystyle\sum_{i=1}^{GE}z_{t,i}=G,\quad\forall t\quad\text{(Sparsity)} \displaystyle\sum_{t=1}^{N}z_{t,i}=k,\quad\forall i\quad\text{(Load Balancing)} \displaystyle z_{t,i}\in\{0,1\} Here the sparsity constraint ensures each token selects exactly G experts, and the Load Balancing constraint ensures each expert processes exactly k=N/E tokens. Solving ( 3 ) exactly requires combinatorial algorithms such as the O(N^{3}) Hungarian Matching algorithm. Most Token Choice (TC) methods therefore strictly enforce the sparsity constraint by setting z_{t,i}=1\iff i\in\text{Top}_{G}(r_{t,\cdot}) , while relying on auxiliary losses ( 21 ; 9 ) or loss-free load balancing strategies ( 46 ) to approximate the load balancing constraint. Expert Choice Routing While the load balancing constraint is essential to avoid routing collapse, the sparsity constraint has no practical benefit. Thus, Expert Choice (EC) ( 56 ) removes the sparsity constraint entirely and enforces only load balancing within batches. The primal problem becomes: \displaystyle\max_{z} \displaystyle\sum_{t=1}^{N}\sum_{i=1}^{GE}z_{t,i}r_{t,i} (4) \displaystyle\text{s.t.} \displaystyle\sum_{t=1}^{N}z_{t,i}=k,\quad\forall i \displaystyle z_{t,i}\in\{0,1\} with trivial closed-form solution z_{t,i}=\mathbf{1}\{t\in\text{Top}_{k}(r_{\cdot,i})\} , i.e. picking the top- k tokens in each batch. This design has two key benefits: (1) Perfect load balancing: each expert processes exactly k=N/E tokens by construction, eliminating the need for auxiliary losses or capacity clipping; (2) Dynamic computation: a token may be selected by zero, one, or multiple experts, enabling adaptive compute allocation based on token importance. However, the per sequence load balancing constraint in EC introduces a causality problem for autoregressive generation. The selection indicator z_{t,i} depends on all tokens’ scores \{r_{1,i},\ldots,r_{N,i}\} —including future tokens unavailable during inference. Extending EC to batch-level top- k ( 28 ) partially alleviates this but does not fully restore causality, as routing still depends on batch composition. 3 Expert Threshold In the preliminaries, we identified the constraints that token choice and expert choice routing impose, yet we question their necessity. To avoid routing collapse, asymptotic load balancing suffices. ET further relaxes the per-sequence or per-batch Load Balancing constraint to a stochastic