Hierarchical Mixture-of-Experts with Two-Stage Optimization
Gleb Molodtsov, Alexander Miasnikov, Aleksandr Beznosikov · 2026
Read Hierarchical Mixture-of-Experts with Two-Stage Optimization by Gleb Molodtsov, Alexander Miasnikov, Aleksandr Beznosikov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08292.txt.
Introduction
Molodtsov and two colleagues redesign how sparse mixture of experts models route tokens to specialist networks. Such models scale capacity by sending each token to a few chosen experts, but routers face a trade off between load balance and specialisation. Strong balancing can suppress specialisation, while aggressive diversity often causes routing collapse, where experts stop differentiating.
Their Hi-MoE framework splits routing control into two coupled levels, inter group balancing for fair traffic across expert groups and intra group specialisation for complementary behaviour within groups. Analysis explains how the hierarchical goals reshape the router, promoting stable specialisation and curbing collapse. The method beats recent sparse routing and grouped mixture baselines across language and vision benchmarks, with scaling studies and ablations confirming robustness. In large scale pretraining on 58 billion tokens, Hi-MoE with 7 billion parameters cuts perplexity and improves expert balance over OLMoE of the same size.
Catalog record
| CREATOR | Gleb Molodtsov, Alexander Miasnikov, Aleksandr Beznosikov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08292
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hierarchical Mixture-of-Experts with Two-Stage Optimization Gleb Molodtsov, Alexander Miasnikov, Aleksandr Beznosikov Abstract . Sparse Mixture-of-Experts (MoE) models scale capacity by routing each token to a small subset of experts. However, their routers exhibit a fundamental trade-off: strong load balancing can suppress expert specialization, while aggressive diversity often causes routing collapse. We propose Hi-MoE , a grouped MoE framework that decomposes routing control into two coupled levels: (i) inter-group balancing that enforces fair traffic across expert groups, and (ii) intra-group specialization that promotes complementary expert behaviors while preventing within-group collapse. Our analysis provides a principled explanation of how our hierarchical objectives reshape the router, thereby promoting stable specialization and mitigating collapse. We observe consistent improvements over recent sparse-routing and grouped-MoE baselines across NLP and vision benchmarks, and confirm robustness via scaling studies (model size, expert count) and targeted ablations. In large-scale pre-training on 58B tokens, Hi-MoE -7B achieves a 5.6 % 5.6\% perplexity reduction and a 40 % 40\% improvement in expert balance over OLMoE -7B across diverse evaluation domains. 1. Introduction Mixture-of-Experts (MoE) architectures have revolutionized deep models by reaching trillion-parameter sizes while maintaining computational efficiency ( 29 ; 7 ; 34 ) . The core principle – routing inputs to a small subset of expert networks – has proven transformative for large language and multimodal models ( 22 ; 17 ; 41 ; 27 ) . However, uneven expert utilization has emerged as a critical challenge in MoE training ( 31 ) . Some experts become overactivated while others remain underused, leading to inefficient use of model capacity. This imbalance becomes particularly problematic when experts are distributed across devices. Even if expert usage appears balanced over time, individual batches may trigger uneven activation across devices. This creates load imbalance, degrades parallelism, and increases communication overhead. Figure 1. Diagram of the proposed Hi-MoE . Experts are organized into hierarchical groups, promoting complementary specialization within groups and balanced, device-level utilization across groups during routing. Recent approaches have addressed this challenge by grouping experts and enforcing balanced activation through fixed expert selection from each group ( 33 ) . This strategy ensures more uniform computational loads across devices. Yet it comes at a cost: constraining selection to a fixed number of experts per group limits the diversity of reachable expert combinations compared to free selection from all available experts. This reveals a fundamental trade-off in MoE training between balanced expert utilization and meaningful specialization. To address this challenge, we propose Hi-MoE , a hierarchical framework that combines explicit expert grouping with level-specific diversity and balancing mechanisms. While the idea of hierarchical MoE structures dates back to classic work on mixtures of experts ( 18 ) and has been revisited in modern settings ( 8 ; 28 ) , we adapt it to the context of grouped experts and large-scale sparse transformers. In particular, we merge recent grouped-expert designs with a two-level optimization hierarchy. At the intra-group level, we encourage complementary expert specialization, while at the inter-group level, we promote balanced utilization and prevent group-level collapse. Figure 1 illustrates the overall Hi-MoE architecture, where experts are partitioned into fixed groups and trained with distinct optimization strategies at each level. As a result, Hi-MoE makes expert grouping an explicit optimization mechanism. This hierarchical view yields a simple, modular component that can be dropped into modern MoE architectures to obtain consistent gains in quality, convergence speed, and hardware efficiency. 2. Related Work Mixture-of-Experts Foundations. The idea of combining multiple specialized predictors traces back to early work on mixtures and ensembles ( 5 ; 16 ; 37 ) , where the goal was to aggregate weak learners into a stronger model. Subsequent studies showed that individual components in a deep ensemble often capture complementary aspects of the data, and that ensemble effectiveness critically depends on maintaining diversity among its members ( 1 ) . However, standard deep ensembles are expensive: they require training and storing many full models, and inference scales linearly with the ensemble size. MoE addresses this bottleneck by sharing most parameters while activating only a small subset of expert networks per input. Formally, one MoE layer replaces the dense feed-forward block with a collection of N expert networks \{f_{i}\}_{i=1}^{N} , where each expert is a function f_{i}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} . A gating network g:\mathbb{R}^{d}\rightarrow\mathbb{R}^{N} maps an input representation \mathbf{x}\in\mathbb{R}^{d} to routing logits, which are converted to probabilities \bm{\pi}(\mathbf{x})=\mathrm{softmax}\bigl(g(\mathbf{x})\bigr)\in\mathbb{R}^{N}. An MoE layer then activates only the K most likely experts and combines their outputs. We write the sparse mixture as (1) \mathrm{SparseMoE}(\mathbf{x})\;=\;\sum_{i=1}^{N}\Bigl(\pi_{i}(\mathbf{x})\,\mathbb{I}\{i\in\mathrm{Top}\text{-}K(\bm{\pi}(\mathbf{x}))\}\Bigr)\,f_{i}(\mathbf{x}), where \mathrm{Top}\text{-}K(\bm{\pi}(\mathbf{x})) returns the indices of the K largest entries of \bm{\pi}(\mathbf{x}) , and \pi_{i}(\mathbf{x}) denotes the i -th component. In practice, a small subset of experts often attracts most tokens while many experts remain almost idle, degrading both specialization and effective capacity. This phenomenon, commonly referred to as routing collapse , has been observed to cause representation collapse and severe load skew across experts in large-scale sparse models ( 2 ) . From a systems perspective, such imbalances translate into stragglers and underutilized devices. From a modeling perspective, they undermine the very goal of MoE, namely to learn diverse expert behaviors. In what follows, we review modern routing and load-balancing strategies, and distill the insights that motivate our hierarchical treatment of specialization and balance in Hi-MoE . † † footnotetext: Sometimes the TopK operator in Equation ( 1 ) is applied before the softmax, but here we continue with this variant as it tends to perform slightly better. Loss-based load balancing. A standard way to counter routing collapse is to augment the training objective with an auxiliary load-balancing term \mathcal{L}_{\text{load}} in addition to the task loss \mathcal{L}_{\text{task}} . The goal of \mathcal{L}_{\text{load}} is to encourage the router to spread tokens and routing probability mass across experts, rather than letting a few experts dominate. GShard formulation ( 21 ) defines the auxiliary loss as (2) \mathcal{L}_{\text{load}}\;=\;\alpha\cdot N\cdot\sum_{i=1}^{N}h_{i}\cdot P_{i}, where h_{i} is the fraction of tokens dispatched to expert i in a batch \mathcal{B} of size |\mathcal{B}| , h_{i}=\frac{1}{|\mathcal{B}|}\sum_{\mathbf{x}\in\mathcal{B}}\left[\operatorname{TopK}\big(\bm{\pi}(\mathbf{x})\big)\right]_{i}, and P_{i} is the mean routing probability for expert i , P_{i}=\frac{1}{|\mathcal{B}|}\sum_{\mathbf{x}\in\mathcal{B}}\pi_{i}(\mathbf{x}). The coefficient \alpha controls the strength of this regularizer. Minimizing ( 2 ) drives both the token histogram \{h_{i}\} and the probability histogram \{P_{i}\} toward uniformity, simultaneously regularizing which experts are selected and how confidently they are chosen. Subsequent works refine this basic idea rather than replacing it. Techniques such as logit normalization, temperature scaling, and per-layer or even per-token adaptive coefficients for \mathcal{L}_{\text{load}} make the balancing signal more robust and easier to tune in deep stacks of MoE layers ( 7 ; 36 ; 6 ; 42 ; 38 ; 3 ; 30 ) . These advances reduce sensitivity to hyperparameters and improve training stability across layers, tasks, and batch regimes. However, all loss-based approaches share a structural challenge: the router is optimized under a compound objective \mathcal{L}_{\text{task}}+\mathcal{L}_{\text{load}} . As analyzed in 35 , the gradients from \mathcal{L}_{\text{load}} can conflict with task gradients, distorting the optimization trajectory of both the router and the experts. In practice, this interference can slow convergence or cap final quality, especially when strong balancing is required to prevent collapse. Loss-free and biased load balancing. A more radical alternative is loss-free balancing. These approaches typically modify the dispatch rule