ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts
Heng Zhao, Zilei Shao, Guy Van den Broeck, Zhe Zeng · 2026
Read ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts by Heng Zhao, Zilei Shao, Guy Van den Broeck, Zhe Zeng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01509.txt.
Introduction
Zhao and co-authors make mixture-of-experts routing differentiable via probability. MoE models activate only a few experts per token. Top-k routing is discrete and blocks gradients, though. Gradient estimators for expert choice remain an open problem. ProbMoE models selection as a distribution over subsets. The subsets are constrained in cardinality by design. Routing becomes probabilistic inference in that discrete space. Exact-k routing samples k-expert subsets in the forward pass.
Backward gradients flow through each expert's exact marginal probability. That gives a tractable surrogate for the true gradient. A dynamic-k variant adapts expert counts per token. Training and inference share the same cardinality range there. Tests show strong performance against competitive baselines. Expert utilization and routing diversity both improve.
Catalog record
| CREATOR | Heng Zhao, Zilei Shao, Guy Van den Broeck, Zhe Zeng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01509
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts Heng Zhao, Zilei Shao, Guy Van den Broeck, Zhe Zeng Abstract Mixture-of-Experts (MoE) models scale by activating only a small subset of experts per token. However, training such models remains challenging because top- k k routing is discrete and non-differentiable, requiring gradient estimators for expert selection whose design remains a central open problem. We introduce ProbMoE, a probabilistic routing framework that models expert selection as a distribution over cardinality-constrained expert subsets and formulates routing as probabilistic inference in this discrete subset space. We first propose ProbMoE Exact- k k routing, which samples k k -expert subsets in the forward pass, and the backward pass uses gradients through each expert’s exact marginal probability as a tractable surrogate for the true gradient. ProbMoE naturally generalizes to a dynamic- k k routing setting, where both training and inference constrain the routing cardinality to the same predefined range, allowing adaptive expert allocation per token. Across benchmarks and model backbones, ProbMoE Exact- k k achieves strong performance compared to competitive baselines, with improved expert utilization and routing diversity; ProbMoE Dynamic- k k achieves comparable performance with fewer activated experts. Code is available at: https://github.com/HengHugoZhao/ProbMoE.git Figure 1 : Comparison of conventional Top- k training and ProbMoE training. Left: Conventional MoE applies a deterministic top- k operator to the softmax routing probabilities for expert selection, while propagating gradients only through these probabilities. Right: ProbMoE models expert routing as probabilistic inference over discrete expert subsets. ProbMoE samples an expert subset S from a cardinality-constrained distribution. Let z denote the binary mask of the sampled subset and let m denote the corresponding expert-selection marginals. Gradients are propagated through the straight-through mask g=\operatorname{stopgrad}(z-m)+m , which is combined with the softmax routing probabilities to form the final routing weights. This yields informative router gradients while preserving sparse expert execution. ProbMoE Dynamic- k allows the subset size to vary within a range, with ProbMoE Exact- k recovered as a special case. 1 Introduction Mixture-of-Experts (MoE) architectures have emerged as a central strategy for scaling large language models while keeping computational costs manageable ( 28 ; 10 ) . By selecting only a small subset of experts for each token, MoE models achieve sublinear growth in FLOPs and allow the total parameter count to substantially exceed the active compute budget ( 9 ; 21 ; 16 ) . However, training MoE models remains challenging because routing relies on a hard top- k operator that is discrete and non-differentiable, making it incompatible with standard gradient-based updates. In standard MoE architectures, the router computes expert scores by applying a softmax operation on expert logits and then selects the top- k experts per token. To bypass differentiating the top- k operator, common training procedures ignore its dependence on the router logits and propagate gradients only through the softmax probabilities ( 28 ; 19 ; 26 ) . As a result, the router receives limited learning signal about alternative expert subsets beyond the deterministically selected top- k experts, which can lead to highly concentrated routing distributions, poor expert utilization, and unstable training dynamics ( 20 ; 37 ; 6 ) . These challenges call for a principled routing mechanism. Instead of treating routing as a deterministic operator with heuristic gradient approximations, we propose to cast expert routing as a probabilistic inference problem: for each token, the selection scores of each expert induce a distribution over expert subsets, and selecting k experts corresponds to probabilistic inference under a cardinality constraint. From this perspective, training aims to optimize the expected loss under the expert subset distribution, which requires differentiating with respect to the parameters of a discrete, cardinality-constrained subset distribution. Although the space of expert subsets is combinatorial, our proposed probabilistic formulation of MoE routing admits tractable inference. To optimize this formulation, we adopt SIMPLE ( 2 ) , a gradient estimator for cardinality-constrained subset distributions. Our proposed framework ProbMoE uses SIMPLE to sample valid k -subsets of experts in the forward pass and differentiate through the induced expert-selection marginals in the backward pass, yielding routing gradients that capture how changes in the selection probability of each expert affect the expected loss. It enables stochastic exploration of expert subsets with informative router updates, while preserving sparse expert execution. Moreover, ProbMoE is agnostic to the underlying MoE architecture and can be flexibly integrated into the training of existing MoE models. Alongside improving fixed- k routing, another ongoing direction in MoE research is dynamic expert allocation, where the number of activated experts can vary across tokens. It is a compelling setting because it allows the model to adapt computation to token- or task-level complexity. Such flexibility can improve efficiency and better match the computational needs of language modeling than a fixed- k setting. However, dynamic- k routing poses a distinct challenge: modeling the choice of expert subset size k . Methods designed for exact- k routing assume a fixed subset size and therefore cannot be directly applied to variable cardinality selection, where the router must reason over subsets of different sizes. To this end, our proposed probabilistic framework naturally unifies exact- k and dynamic- k routing in a principled way, as shown in Figure 1 . ProbMoE Exact- k conditions the subset distribution on a fixed cardinality, whereas ProbMoE Dynamic- k conditions the same distribution on a range of feasible cardinalities. Unlike prior adaptive expert allocation methods that rely on thresholding, null experts, or other heuristic gating rules ( 35 ; 17 ; 18 ) , ProbMoE Dynamic- k considers a normalized distribution over feasible expert subsets and enables effective training with tractable marginal-based router gradients. Contributions We summarize our main contributions as follows: (1) we propose to cast MoE routing as probabilistic inference problems over cardinality-constrained expert subsets; (2) we adopt the gradient estimator SIMPLE in MoE routing to pair probabilistic subset sampling with marginal-based router gradients; (3) we introduce a range-constrained routing variant for dynamic- k MoE routing that jointly infers expert identities and routing cardinality for each token; (4) in empirical evaluations, we show that ProbMoE Exact- k improves expert utilization and routing diversity; (5) in dynamic k setting, experiment results show that ProbMoE Dynamic- k achieves competitive performance with fewer activated experts on average. 2 Related Work Routing Optimization and Differentiability. Standard Top- k routing is non-differentiable, motivating several strategies to stabilize training. One stream of work modifies gradient propagation without changing the discrete nature of the forward pass, employing techniques such as dense backpropagation ( 24 ; 34 ) or learned sparse gradient selection ( 21 ) . Another line of work relaxes the discrete constraint, using continuous activations to approximate routing decisions ( 32 ) . Although effective for improving optimization stability, these methods do not explicitly model the distribution over feasible expert subsets, and therefore provide limited mechanisms for exploring alternative expert combinations during training. Prior routers have also introduced stochastic elements, such as noise injection ( 28 ; 27 ) or randomized tie-breaking ( 19 ; 10 ) . However, they primarily treat routing as a deterministic operator augmented with heuristic randomization. To our knowledge, no prior work formulates MoE routing as probabilistic inference over discrete expert subsets, using stochastic subset selection as a principled mechanism for