The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts
Enric Boix-Adsera, Philippe Rigollet · 2025
Read The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts by Enric Boix-Adsera, Philippe Rigollet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06839.txt.
Introduction
Enric Boix-Adsera and Philippe Rigollet study why splitting a neural network into many small experts boosts its power, in a 2025 paper. Mixture-of-experts layers switch on only part of a model for each input, saving compute while growing total size. Designs differ in how many experts fire at once, a choice called granularity.
The authors prove an exponential gap in expressivity tied to that choice, favoring many fine-grained experts over a few large ones. Experiments back the theory and show the gap in practice. The result speaks to a live design split between leading model families. They argue granularity deserves attention as a core scaling lever.
Catalog record
| CREATOR | Enric Boix-Adsera, Philippe Rigollet |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 100 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06839
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts Enric Boix-Adsera, Philippe Rigollet Abstract Mixture-of-Experts (MoE) layers are increasingly central to frontier model architectures. By selectively activating parameters, they reduce computational cost while scaling total parameter count. This paper investigates the impact of the number of active experts, termed granularity , comparing architectures with many (e.g., 8 per layer in DeepSeek) to those with fewer (e.g., 1 per layer in Llama-4 models). We prove an exponential separation in network expressivity based on this design parameter, suggesting that models benefit from higher granularity. Experimental results corroborate our theoretical findings and illustrate this separation. 1 Introduction Mixture-of-experts (MoE) layers [ JJNH91 , ERS13 , SMM+17 ] are emerging as an increasingly important component in the design of frontier architectures [ LFX+24 , Met25 , JSR+24 , Mos24 , Qwe24 , Sno24 ] . The main advantage of MoE layers is that they enable scaling the total number of parameters in the network while keeping computational costs manageable. This is achieved by having only a small fraction of the layers’ parameters activate on a particular input. Thus, models can have a large number of total parameters (unlocking the benefits of scaling laws [ KMH+20 ] ), while simultaneously having a small number of active parameters (enabling low computational cost). DeepSeek-V3 convincingly demonstrates how MoE layers can dramatically reduce computational costs in large language models. Despite having 671B total parameters, DeepSeek-V3 activates only 37B parameters (approximately 5.5%) per token [ LFX+24 ] , resulting in computational requirements substantially lower than for comparable dense models. Similarly, Meta’s recently unveiled Llama-4 Maverick model, with 400B parameters [ Met25 ] , achieves 4.3% activation sparsity because 96% of its parameters lie in MoE layers. As modern architectures increasingly adopt MoE techniques, understanding optimal design choices for MoE layers becomes increasingly relevant for designing efficient models. Key decisions include determining the number of experts, the size of each expert, and the routing mechanism. With this background in mind, we seek to address the following question: How do specific MoE design choices influence model expressivity? In this paper, we focus on a central design choice – the granularity of the MoE layer, which is the number of experts that activate on a token [ KLA+24 ] . The granularity of an MoE should not be confused with its sparsity, which is the ratio of the number of active experts to total experts; see Table 1 . The sparsity of an MoE is another hyperparameter of significant interest, which deserves separate study. Indeed, the granularity and sparsity parameters can be decoupled (e.g. an MoE layer in which 4 out of 16 experts are active has the same sparsity as an MoE layer in which 1 out of 4 experts are active, but they have different granularities). Despite the centrality of the granularity parameter, no consensus exists on the optimal number of active experts to employ, with open-source frontier systems adopting widely varying configurations as shown in Table 1 . Architecture # active out of # total experts Granularity Sparsity DeepSeek-V3 [ LFX+24 ] 8 out of 256 8 3.1% Qwen1.5-MoE-A2.7B [ Qwe24 ] 4 out of 64 4 6.2% DBRX [ Mos24 ] 4 out of 16 4 25% Snowflake Arctic [ Sno24 ] 2 out of 128 2 1.6% Google GLaM [ DHD+22 ] 2 out of 64 2 3.1% Mixtral 8x7B and 8x22B [ JSR+24 ] 2 out of 8 2 25% Llama-4 Maverick [ Met25 ] 1 out of 128 1 0.8% Llama-4 Scout [ Met25 ] 1 out of 16 1 6.2% Table 1 : MoE hyperparameter choices in various frontier architectures. Granularity can be decoupled from sparsity of active parameters. For instance, MoE layers in Qwen1.5-MoE and Llama-4 Scout, have sparsity of 1/16 because 1/16 of the parameters are active on any token for both models. However, Qwen1.5-MoE has a granularity of 4, whereas Llama-4 Scout has a granularity of 1. The lack of consensus in Table 1 highlights the ambiguity surrounding optimal MoE layer design. Specifically, it remains unclear whether large granularity architectures, exemplified by DeepSeek-V3, or small granularity architectures, as seen in the Llama-4 suite, are preferable. While granularity, sparsity, and parameter count all influence model performance, this work aims to isolate the specific impact of granularity while controlling for other factors. Our contribution The main insight of this paper is that increasing the granularity of an MoE improves its expressivity exponentially, even while keeping the sparsity of the MoE unchanged. Thus, our result suggests that future frontier architectures should be designed with larger granularity, so as to reap the benefit of this extra expressivity with no significant change to the number of total and active parameters in the model. Our result agrees with the empirical scaling laws observed in [ KLA+24 ] , which show that higher granularity indeed leads to lower loss in trained MoE models. We discuss other considerations on increasing the granularity in Section 1.1 . In order to describe our result in more detail, let us recall the MoE architecture [ SMM+17 ] . A (m,k) -MoE layer f has granularity k and m experts. It consists of “expert” networks E_{1},\ldots,E_{m} , and a “gating” network G that outputs a k -sparse vector G(x)\in\mathbb{R}^{m} on input token x . The MoE layer outputs \displaystyle f(x)=\sum_{i=1}^{m}G(x)_{i}E_{i}(x)\,. We study the standard setting where the experts are two-layer fully-connected networks, and the gating network is a linear routing function activating on the top- k experts; see Section 2 for precise definitions. We prove our main result for constant, linear, and ReLU activation functions, when the input distribution \mu is either standard Gaussian or uniform over the unit ball. Our main theorem identifies the number of possible configurations of experts \binom{m}{k} as a key combinatorial quantity controlling the expressivity of an MoE layer. Below, we state the theorem informally and in a slightly weaker form than the full extent of our results, for the sake of exposition. Informal Theorem 1.1 (Expressivity benefits of higher granularity) . There are constants C,c>0 such that the following holds. Suppose that m\geq Ck and that \binom{m^{\prime}}{k^{\prime}}<c\binom{m}{k}^{0.99}\,. Then there is a (m,k) -MoE f that cannot be approximated by any (m^{\prime},k^{\prime}) -MoEs with the same number of active parameters. Namely, for any such (m^{\prime},k^{\prime}) -MoE f^{\prime} , it holds that \displaystyle\mathbb{E}_{x\sim\mu}\|f(x)-f^{\prime}(x)\|^{2}>c\mathbb{E}_{x\sim\mu}\|f(x)\|^{2}\,. If the granularity k is held constant and m grows, then the quantity \binom{m}{k} grows on the order of \Theta(m^{k}) , which scales exponentially with the granularity. This means that the effects of the granularity on expressivity are evident at even relatively small granularities such as those in Table 1 . For example, for DeepSeek-V3, the number of possible configurations of active experts is \binom{256}{8}\geq 4\times 10^{14} . Thus, Theorem 1.1 heuristically suggests 1 1 1 We write “heuristically suggests” because the constant c in the theorem is not explicitly computed. that an MoE layer with an equal number of active parameters, but granularity 1, would need on the order of 10^{14} experts to approximate an MoE layer of DeepSeek-V3. The result of this theorem appears intuitively evident in retrospect: higher granularity allows for more parameter-efficient models because the same parameters can be