Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
Seng Pei Liew, Kenta Shinzato, Yuyang Dong · 2026
Read Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints by Seng Pei Liew, Kenta Shinzato, Yuyang Dong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08215.txt.
Introduction
Seng Pei Liew, Kenta Shinzato, and Yuyang Dong seek principled design rules for mixture-of-experts language models. Their 2026 arXiv paper notes that modern designs rest on two factors: total parameters and active parameters. Total size sets the memory footprint, while active size sets inference cost. The authors find these two factors alone cannot describe an optimal architecture. Memory and inference limits shape real deployments, so missing factors can mislead builders. The paper aims at design rules that reflect how such models are actually run and stored.
Catalog record
| CREATOR | Seng Pei Liew, Kenta Shinzato, Yuyang Dong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08215
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints Seng Pei Liew, Kenta Shinzato, Yuyang Dong Abstract Modern Mixture-of-Experts (MoE) language models are designed based on total parameters (memory footprint) and active parameters (inference cost). However, we find these two factors alone are insufficient to describe an optimal architecture. Through a systematic study, we demonstrate that MoE performance is primarily determined by total parameters ( N t o t a l N_{total} ) and expert sparsity ( s := n e x p / n t o p k s:=n_{exp}/n_{topk} ). Moreover, n e x p n_{exp} and n t o p k n_{topk} do not "cancel out" within the sparsity ratio; instead, a larger total number of experts slightly penalizes performance by forcing a reduction in core model dimensions (depth and width) to meet memory constraints. This motivates a simple principle for MoE design which maximizes N t o t a l N_{total} while minimizing s s (maximizing n t o p k n_{topk} ) and n e x p n_{exp} under the given constraints. Our findings provide a robust framework for resolving architectural ambiguity and guiding MoE design. 1 Introduction Many publicly available Mixture-of-Experts (MoE) language models are characterized by total and active parameters, e.g., Qwen3-235B-A22B 30 and OLMoE-1B-7B 21 . This is likely due to the following practical deployment considerations: Total parameters determine the model’s memory footprint, while active parameters influence the computational cost during inference; both factors also fundamentally influence training latency. Despite their prevalence, these two parameters are insufficient to fully characterize an MoE architecture. To resolve this ambiguity, we define an MoE architecture using five variables: Number of layers or depth ( l ), hidden dimension or width ( d ), number of experts ( n_{exp} ), number of activated experts ( n_{topk} ), and the ratio of hidden dimension to expert hidden dimension (granularity, g\coloneq d/d_{exp} ). For a multi-head attention-based MoE model, the non-embedding parameter counts are: 1 1 1 For convenience, total and active parameters are always referred to as non-embedding parameters in this paper; a more precise description of configurations is given in Appendix A . \displaystyle N_{total} \displaystyle\approx ld^{2}\left(4+3n_{exp}/g\right) (1) \displaystyle N_{active} \displaystyle\approx ld^{2}\left(4+3n_{topk}/g\right) (2) Even if we constrain N_{total} and N_{active} , there are still multiple choices of width, depth, and MoE configuration that satisfy the above two constraints. This raises an important question: Given upper bounds on memory and inference, how should one select the variables to optimize performance? In this paper, we conduct a systematic study to identify the optimal configuration within a fixed budget of N_{total} and N_{active} ( N_{total}>N_{active} ). We show that as long as the width-to-depth ratio and granularity are within reasonable ranges, performance is mainly determined by N_{total} and expert sparsity s\coloneq n_{exp}/n_{topk} . Furthermore, we identify a specific n_{exp} penalty. For a fixed sparsity ratio, increasing n_{exp} requires a compensatory reduction in the model’s core dimensions ( l and d ) to stay within the total parameter budget. This reduction leads to a net decrease in the active compute capacity per token, which slightly penalizes performance. These findings allow us to derive a principle that determines the optimal MoE configuration that maximizes N_{total} while minimizing s (maximizing n_{topk} ) and n_{exp} under the given constraints. By fitting scaling laws across models and dataset sizes, we also test and validate our principle on two representative architectures. Related work. Various aspects of MoE models 25 ; 16 ; 7 have been studied in the literature. For a comprehensive survey of MoE models, please refer to 3 . We here only review the most relevant ones, which often relate the loss/performance of MoE models to the model configuration via scaling laws. 4 are among the first to propose a scaling law for MoE models, showing that the loss is a function of N_{total} and n_{exp} . 15 ; 18 ; 1 ; 20 ; 28 study aspects such as granularity, upcycling, sparsity, memory efficiency and dense-to-MoE efficiency ratio. While conclusions differ among studies, the consensus is that total parameters and/or active parameters are the most important factors affecting the performance of MoE models. There is however no work trying to disentangle the interplay of all the parameters appearing in Equations 1 and 2 given total parameters and active parameters, as far as we know. Setup. We use an architecture closely resembling that of Qwen3 30 , which in turn follows the architecture of 7 , scaling the feed-forward (FFN) layers of transformer-based language models into n_{exp} experts and activating only n_{topk} of them for each input token. Furthermore, the fine-grained MoE architecture is adopted with the FFN hidden dimension g times smaller than the attention hidden dimension 5 ; 15 . The models are trained using a standard language modeling objective on the FineWeb-Edu dataset 22 , with its performance evaluated on a held-out dataset. More details about training hyperparameters and model configurations can be found in Appendix A . 2 Ablation fixing Total and Active Parameters As discussed above, total and active parameters are vital at specifying MoE models. We first conduct ablation studies by fixing (approximately) these two parameters and varying other configurations to understand how different combinations of parameters change the loss. 2.1 Granularity We study the effect of granularity g by varying n_{exp},n_{topk} in conjuction with d_{exp} such that the total and active parameters are fixed. Higher g means finer granularity; i.e., smaller expert hidden dimension d_{exp} , but more experts. Table 1 shows the results of this ablation study on two different model sizes. We see that g of 4 to 8 generally minimize the loss, while increasing g leads to diminishing returns. Our results largely agree with 28 that g (in our notations) between 4 and 6 is a good choice. We use g=4 in the following experiments. l d g n_{exp} n_{topk} Loss diff (%) 8 384 2 64 4 0.7% 4 128 8 0.48% 8 256 16 0.00% 16 512 32 0.21% 18 1024 2 64 4 1.04% 4 128 8 0.00% 8 256 16 0.29% 16 512 32 0.58% Table 1: Ablation study on granularity g with fixed total and active parameters. See Appendix B for more results. 2.2 Width-to-depth Ratio Next, we wish to study the choice of width-to-depth ratio \gamma\coloneq d/l . We vary all relevant parameters in Equations 1 and 2 while keeping total and active parameters fixed approximately ( g is fixed to 4). We see in Table 2 that models with the lowest \gamma (e.g., 10) generally have higher loss, while those with mid-range values (e.g., 42) achieve the lowest loss in configurations explored. However, simply increasing \gamma to the maximum (136) does not guarantee the lowest loss, suggesting a sweet spot. We consequently choose to follow the standard setting of 14 by keeping \gamma between 32 and 64. l d d/l n_{nexp} n_{topk} Loss diff (%) 16 240 15 43 4 0.65% 8 336 42 43 4 0.00% 4 480 120 43 4 1.12% 16 160 10 103 16 2.68% 8 224