Slicing and Dicing: Configuring Optimal Mixtures of Experts
Margaret Li, Sneha Kudugunta, Danielle Rothermel, Luke Zettlemoyer · 2026
Read Slicing and Dicing: Configuring Optimal Mixtures of Experts by Margaret Li, Sneha Kudugunta, Danielle Rothermel, Luke Zettlemoyer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11689.txt.
Introduction
Margaret Li and three colleagues run the first broad sweep of mixture-of-experts design choices, varying expert counts, sizes, shared experts, and load balancing across more than 2,000 pretraining runs up to 6.6 billion parameters. They find performance keeps rising with total expert parameters, optimal expert size tracks active parameters alone, and most other choices matter little beside expert count and granularity.
Catalog record
| CREATOR | Margaret Li, Sneha Kudugunta, Danielle Rothermel, Luke Zettlemoyer |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 136660 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11689
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Slicing and Dicing: Configuring Optimal Mixtures of Experts Margaret Li, Sneha Kudugunta, Danielle Rothermel, Luke Zettlemoyer Abstract Mixture-of-Experts (MoE) architectures have become standard in large language models, yet many of their core design choices — expert count, granularity, shared experts, load balancing, token dropping — have only been studied one or two at a time over narrow configuration ranges. It remains an open question whether these choices can be optimized independently, without considering interactions. We present the first systematic study of over 2,000 pretraining runs spanning models up to 6.6B total parameters, in which we exhaustively vary total experts, expert dimension, heterogeneous expert sizing within a single layer, shared expert size and load-balancing mechanisms. We find that at every active-parameter scale that we study, performance consistently improves with total MoE parameters even at extreme active expert parameter ratios like 128. Further, the optimal expert size is nearly invariant to total parameter count and depends only on active parameter count. Third, we see that other choices like shared experts, heterogeneous experts and load-balancing settings have small effects relative to expert count and granularity, although dropless routing yields a consistent gain. Overall, our results suggest a simpler recipe: focus on expert count and granularity, other choices have minimal effect on final quality. 1 Introduction Mixture of Experts (MoE) ( 34 ) models decouple computational overhead from model capacity by adding experts which are conditionally activated by a learned router. Recent MoE LLMs demonstrate improved efficiency when training at massive scales ( 13 ; 9 ; 11 ) . Yet, there is a lack of analysis to understand the effect of the central design choices controlling this sparse activation; we do not know exactly what makes existing recipes work so well. Several fundamental MoE design choices introduce substantial complexity beyond dense LLMs ( 31 ; 4 ) . To date, these innovations have been evaluated in isolation or in limited combinations. First, the granularity and topology of experts — including total expert count ( 5 ) , the use of shared FFN components alongside routed ones ( 32 ) , and the implementation of fine-grained experts ( 8 ) have significant implications for both model quality and throughput. Second, the choice of routing algorithm ( 5 ; 48 ; 21 ; 46 ; 8 ) greatly affects model quality. We present the first systematic study of all of these factors. Our main contributions are: Disentangling expert count from granularity: Unlike previous studies, we vary both expert granularity and total expert count separately to measure the independent impact of expert size and expert count under a fixed compute budget. We conduct over 2000 experiments ranging from 10 million to 6.6 billion total parameters, and find that performance improves monotonically with total MoE parameters even at an active expert parameter ratio of 128. Expert flexibility offers little: We test heterogeneous expert pools (mixed granularities within a layer) and shared “generalist” experts across many configurations. Neither improves over a well-configured homogeneous MoE, while generalists consistently hurt. Simplifying the routing design space: Across multiple expert configurations and scales, we see that MoE quality is robust to load-balancing hyperparameters within reasonable ranges, while dropless routing yields a small but consistent gain. We therefore make the practical recommendation to minimally sweep hyperparameters, but focus MoE performance optimization on expert count and granularity, setting expert size according to active parameter scale, then setting total expert count to match memory allowances. We also release our code and data at https://github.com/hadasah/slicing_and_dicing . 2 Mixture of Experts Figure 1 : A Mixture of Experts Layer . In an MoE Transformer layer, each token first passes through the same self attention mechanism. Then, router(s) activate the highest affinity experts. Finally, the outputs of any activated FFN modules are combined in a weighted sum. A standard homogeneous MoE layer includes n experts of identical granularity g , of which k are activated. We introduce more flexible configurations with heterogeneous MoE layers consisting of experts with different granularity, as shown above ( Expert Sets 1 and 2 ). An always-active generalist can also be added (labeled ‘G’). Mixture of Experts models contain conditionally activated submodules, selected by a routing mechanism. Recent MoE LLMs, which are based on the Transformer architecture, replace the feed-forward network (FFN) in some or all transformer layers. Instead of a single FFN, a router activates k out of n expert feed-forward networks. The chosen subset of k depends on the input, and is commonly selected per-timestep (i.e., per-token), per-sequence, or per-document. The selected subset of expert FFNs generates outputs which are combined via an average weighted by the per-expert router weights. 2.1 Expert Configurations Expert Size. In dense Transformer models, the intermediate FFN dimension is canonically 4 times the hidden dimension. MoEs were previously configured to have FFN experts of identical dimensions to their dense FFN counterparts. However, more recent work has explored fine-grained experts with decreased intermediate FFN dimension ( 8 ) . We define granularity: g=\frac{\text{intermediate dimension of expert FFN}}{\text{intermediate dimension of dense FFN}} A granularity of 1 indicates expert FFNs of the same size as the dense FFN, whereas a granularity of \frac{1}{4} indicates expert FFNs with an intermediate dimension equivalent to \frac{1}{4} that of the dense FFN. Active Expert Count. Increased active expert count incurs a higher FLOPs-per-token compute cost and memory usage, as each additional active expert entails additional computation. 20 studied the tradeoffs in higher expert activation count with increased compute budget, and chose to activate 2 experts, as more than 2 experts yielded relatively marginal gains, but activating only 1 expert reduced performance. 13 chose to activate 1 expert, but 47 found that with properly configured load balancing, 2 experts indeed achieved better performance, perhaps as a result of enabling gradients to directly update router rows in relation to each other. Active expert count is independent of expert size. However, to match FLOPs per step in performance comparisons, active expert count can be constrained to fix activated expert parameters: \text{\# of active experts}\cdot\text{expert granularity}=\text{1} In other words, to FLOP-match an MoE with expert granularity g to a dense model, we activate exactly k=\frac{1}{g} experts, thus maintaining an equal number of active parameters. Total Expert Count. Total expert count may also be varied independently of expert size and active expert count. A larger total expert count results in only a negligible increase in compute cost. However, each additional total expert incurs increased memory costs, depending on infrastructure. Activation Sparsity. To enable fair comparison, we define MoE activation sparsity s , the factor by which total FFN parameters exceed active FFN parameters s:=n\cdot g . This is sometimes referred to as expansion factor , but this term is overloaded and also used to refer to FFN dimension expansion. Heterogeneous Experts. The granularity g is typically shared for all experts; all experts have identical dimensions. However, this homogeneous sizing of experts is not a requirement of the architecture. For example, instead of a set of 16 experts with granularity \frac{1}{2} , an MoE layer might have 8 experts of granularity \frac{1}{2} , as well as 16 experts of \frac{1}{4} granularity. Intuitively, this additional flexibility may allow for additional improvement, as not all functions learned by the model require