Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks

Taishi Nakamura, Satoki Ishikawa, Masaki Kawamura, Takumi Okamoto, Daisuke Nohara, Jun Suzuki, Rio Yokota · 2025

Read Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks by Taishi Nakamura, Satoki Ishikawa, Masaki Kawamura, Takumi Okamoto, Daisuke Nohara, Jun Suzuki, Rio Yokota (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18672.txt.

Introduction

Taishi Nakamura and co-authors ask how sparse a large language model should be in a 2025 paper. Models of the mixture of experts type hold many expert parts but use only a few for each input. That sparsity adds a design choice that older scaling rules, built for dense models, do not cover.

The team trained families of such models while varying total size, active size, and how many experts fire per input, all under fixed compute budgets. Training loss alone did not tell the whole story. Models with the same loss but more active compute did better on reasoning tasks. The work splits memory skills from reasoning skills to set clearer design rules.

Catalog record

CREATORTaishi Nakamura, Satoki Ishikawa, Masaki Kawamura, Takumi Okamoto, Daisuke Nohara, Jun Suzuki, Rio Yokota
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18672

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks Taishi Nakamura, Satoki Ishikawa, Masaki Kawamura, Takumi Okamoto, Daisuke Nohara, Jun Suzuki, Rio Yokota Abstract Empirical scaling laws have driven the evolution of large language models (LLMs), yet their coefficients shift whenever the model architecture or data pipeline changes. Mixture‑of‑Experts (MoE) models, now standard in state‑of‑the‑art systems, introduce a new sparsity dimension that current dense‑model frontiers overlook. We investigate how MoE sparsity influences two distinct capability regimes: memorization and reasoning. We train families of MoE Transformers that systematically vary total parameters, active parameters, and top‑ k k routing while holding the compute budget fixed. For every model we record pre‑training loss, downstream task loss, and task accuracy, allowing us to separate the train‑test generalization gap from the loss‑accuracy gap. Memorization benchmarks improve monotonically with total parameters, mirroring training loss. By contrast, reasoning performance saturates and can even regress despite continued gains in both total parameters and training loss. Altering top‑ k k alone has little effect when active parameters are constant, and classic hyperparameters such as learning rate and initialization modulate the generalization gap in the same direction as sparsity. Neither post‑training reinforcement learning (GRPO) nor extra test‑time compute rescues the reasoning deficit of overly sparse models. Our model checkpoints, code and logs are open‑source at https://github.com/rioyokotalab/optimal-sparsity . 1 Introduction The recent evolution of large language models (LLMs) has been driven by empirical scaling laws ( Hestness et al. 2017 ) that link training loss to model size, dataset size, and compute budget. Kaplan et al. showed that these laws hold across seven orders of magnitude, establishing them as a reliable extrapolation tool for dense Transformers ( Kaplan et al. 2020 ) . Subsequent work by Hoffmann et al. demonstrated that scaling curves can be inverted to choose the compute-optimal combination of parameters and tokens for a fixed budget ( Hoffmann et al. 2022 ) . Together, these results have made scaling analysis a cornerstone of model planning at both academic and industrial labs. Yet the coefficients of the scaling laws are not universal. Highly expressive models trained under different optimizers or architectures often follow the same loss trajectory but diverge substantially on downstream reasoning benchmarks ( Liu et al. 2023 ) . Brandfonbrener et al. extend the classic laws with loss-to-loss prediction, showing that the mapping between training and test distributions admits its own power law when the distributions differ substantially ( Brandfonbrener et al. 2025 ) . These observations imply that optimal budgets must be re-estimated whenever we modify the model or the data pipeline. A particularly compelling architectural modification is the Mixture-of-Experts (MoE) paradigm, offering high capacity at fixed FLOPs by routing each token through a sparse subset of experts ( Shazeer et al. 2017 ; Lepikhin et al. 2021 ; Fedus et al. 2021 ) . Modern flagship models, e.g., Gemini 2.5 Pro ( Gemini Team 2025 ) , DeepSeek-V3 ( DeepSeek-AI 2025b ) , and Qwen3 ( Qwen Team 2025 ) now rely on MoE as a de-facto standard for economical scaling. Abnar et al. derive a parameters-vs-FLOPs frontier and locate an optimal sparsity for a given compute budget ( Abnar et al. 2025 ) . These findings emphasize that the classical dense-model frontier is an incomplete picture, and one must account for architectural knobs such as MoE sparsity and top- k routing. Furthermore, loss-based scaling curves do not always predict the performance on downstream tasks. Jelassi et al. report that increasing MoE sparsity improves memorization benchmarks, but saturates for reasoning performance ( Jelassi et al. 2025 ) . However, the Mixture of Parrots paper ( Jelassi et al. 2025 ) only explores the number of active vs. total parameters, ignoring the effect of routing strategies beyond standard top-2 routing. They also do not consider the effect of reinforcement learning and test-time compute on their reasoning benchmarks. Evaluating reasoning performance immediately after pre-training overlooks both the benefits of post-training adaptation and the leverage of additional test-time compute. Post-training methods such as GRPO, which use reinforcement signals to encourage coherent chain-of-thought generation, sharpen a model’s reasoning on complex tasks ( OpenAI 2024b ; DeepSeek-AI 2025a ) . Beyond these refinements, models can further improve outputs at test time by adopting calibrated decoding strategies that mirror how humans pause to reconsider difficult problems. These test-time approaches not only boost routine benchmark performance but, when properly tuned, substantially enhance multi-step mathematical reasoning, demonstrating that adaptive computing at test time is a powerful complement to both model scale and post-training adaptation. In this paper, we aim to identify how the optimal sparsity of MoE changes between memorization (TriviaQA, HellaSwag) and reasoning (GSM8K, GSM-Plus) tasks. In this work, we use the term dense models to refer to standard Transformers with a single feed-forward network per layer. For MoE models, we define sparsity as \text{sparsity}=1-\tfrac{\text{Top-}k}{\text{Experts}} following the convention that sparsity measures the fraction of inactive parameters. We train families of MoEs varying not only the total vs. active parameters, but also the number of top- k experts. For each model, we measure the loss on the pre-training data, the task loss on the downstream benchmarks, and the accuracy on those benchmarks. This allows us to disentangle the generalization gap between the train vs. test loss, and the gap between loss vs. accuracy. For both memorization and reasoning benchmarks, the train loss decreases monotonically with the total parameters. The task loss and accuracy follow the same monotonic trend as the train loss for memorization benchmarks. In contrast, for reasoning benchmarks, the task loss and accuracy diverge from the monotonic trend as the total parameters increase and training loss decreases. We found that changing the k in top- k routing itself has a negligible effect if the number of active parameters is kept constant. We also consider classic generalization-gap controls by sweeping the learning rate and initialization, and show that their effects align strikingly with the generalization-gap caused by sparsity. This confirms that the gap between the performance on memorization vs. reasoning tasks can be induced not only by sparsity of the MoE, but also classical hyperparameters like learning rate and initialization. We further investigate whether applying GRPO or additional test-time compute could recover the poor reasoning ability of sparser models. Our results show that the gap between memorization and reasoning performance caused by increased sparsity remains unchanged even after GRPO and increased test-time compute. This means that finding the optimal sparsity of the MoE during pre-training is crucial for training a reasoning model under a fixed compute budget. We release model checkpoints, code and logs are open-source at https://github.com/rioyokotalab/optimal-sparsity . 2 Background and Related Work 2.1 Mixture of Experts MoE Architecture. Mixture-of-Experts (MoE) networks were introduced by ( Jacobs et al. 1991 ; Jordan & Jacobs 1994 ) and later brought to large-scale neural language modeling by Shazeer et al. 2017 . Within the Transformer architecture ( Vaswani et al. 2017 ) , MoE layers have proven especially effective, scaling to hundreds of billions of parameters while maintaining manageable training costs ( Lepikhin et al. 2021 ; Fedus et al. 2021 ; Du et al. 2022 ; Zoph et al. 2022 ) . Consequently, modern state-of-the-art language models, including Gemini 2.5 Pro (

Read the complete text