Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations

Ajay Jaiswal, Jianyu Wang, Yixiao Li, Pingzhi Li, Tianlong Chen, Zhangyang Wang, Chong Wang, Ruoming Pang, Xianzhi Du · 2025

Read Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations by Ajay Jaiswal, Jianyu Wang, Yixiao Li, Pingzhi Li, Tianlong Chen, Zhangyang Wang, Chong Wang, Ruoming Pang, Xianzhi Du (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05586.txt.

Introduction

Ajay Jaiswal and eight co-authors study how to slim down mixture-of-experts language models by dropping their weakest expert modules. These models switch on only a few experts per input, which lets them grow large, but they carry heavy memory costs and many experts seem to add little. The team asks how to find the experts that can go.

They built a test suite of old and new ways to score expert importance, and compared recipes for choosing which experts to drop. Against the usual one-shot cut, they found that pruning step by step, with scores recomputed each round, works better. Extra fine-tuning during the process also helped the model recover. One skill suffered most from dropping experts: following instructions. The authors show it can be restored with a few worked examples and further fine-tuning.

Catalog record

CREATORAjay Jaiswal, Jianyu Wang, Yixiao Li, Pingzhi Li, Tianlong Chen, Zhangyang Wang, Chong Wang, Ruoming Pang, Xianzhi Du
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-05586

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations Ajay Jaiswal, Jianyu Wang, Yixiao Li, Pingzhi Li, Tianlong Chen, Zhangyang Wang, Chong Wang, Ruoming Pang, Xianzhi Du Abstract Sparsely activated Mixture-of-Experts (SMoE) has shown promise in scaling up the learning capacity of neural networks. However, vanilla SMoEs have issues such as expert redundancy and heavy memory requirements, making them inefficient and non-scalable, especially for resource-constrained scenarios. Expert-level sparsification of SMoEs involves pruning the least important experts to address these limitations. In this work, we aim to address three questions: 1 What is the best recipe to identify the least knowledgeable subset of experts that can be dropped with minimal impact on performance? 2 How should we perform expert dropping (one-shot or iterative), and what correction measures can we undertake to minimize its drastic impact on SMoE subnetwork capabilities? 3 What capabilities of full-SMoEs are severely impacted by the removal of the least dominant experts, and how can we recover them? Firstly, we propose MoE Experts Compression Suite (MC-Suite) , which is a collection of some previously explored and multiple novel recipes to provide a comprehensive benchmark for estimating expert importance from diverse perspectives, as well as unveil numerous valuable insights for SMoE experts. Secondly, unlike prior works with a one-shot expert pruning approach, we explore the benefits of iterative pruning with the re-estimation of the MC-Suite criterion. Moreover, we introduce the benefits of task-agnostic fine-tuning as a correction mechanism during iterative expert dropping, which we term MoE Lottery Subnetworks . Lastly, we present an experimentally validated conjecture that, during expert dropping, SMoEs’ instruction-following capabilities are predominantly hurt, which can be restored to a robust level subject to external augmentation of instruction-following capabilities using k-shot examples and supervised fine-tuning. † † affiliationnotice: Equal contribution 1 Introduction Sparsely activated Mixture-of-Experts (SMoEs) are a promising architecture design that facilitates an amalgamation of the collective intelligence of multiple experts and are distinguished by their ability to dynamically allocate computational resources based on the input. Mixture-of-Experts, initially introduced in ( Shazeer et al., 2017a ) , has undergone extensive exploration and advancement, and is now adopted in industry-scale LLMs ( e.g. , Mixtral-8 \times 7B, Grok-1, DBRX, etc. ), achieving stellar performance across various NLP and CV task leaderboards. Despite the sparse nature of MoEs promising enhanced efficiency and scalability, they have crucial limitations: 1 SMoEs trade space for FLOPs, which require high memory usage due to the duplication of the network layers into multiple copies as experts; 2 SMoEs tend to have poor utilization of their capacity and existence of redundancy ( Mittal et al., 2022 ; Chen et al., 2023 ) due to representation collapse. In parallel to well-studied techniques that address memory and compute bottlenecks using weight sparsity ( Jaiswal et al., 2023c ; Lee et al., 2019 ; Frankle & Carbin, 2019 ; Yin et al., 2023b ; Liu et al., 2023a ) and quantization ( Liu et al., 2023b ; Kim et al., 2023 ; Dettmers et al., 2023 ; Frantar et al., 2022 ; Lin et al., 2023 ) , SMoEs architecture design facilitates a unique opportunity for expert-level sparsification that aims to compact the SMoE model by retaining fewer but more knowledgeable experts. For instance, Figure 2 illustrates that the existence of some experts is critically important (dominant) and dropping them could lead to an abrupt performance drop, while some experts are notably redundant with negligible impact when removed. Recently, a few works have proposed expert importance estimation techniques such as token reconstruction loss ( Lu et al., 2024 ) and heavy-hitters counting ( Muzio et al., 2024 ) , illustrating the potential of expert dropping. However, a comprehensive benchmarking of possible task-agnostic recipes to select the best recipe is still missing. At this point, one key question arises: What is the best recipe to identify less knowledgeable experts that can be dropped without sacrificing the vital knowledge and capabilities of the SMoE? Figure 1: MoE Experts Compression Suite (MC-Suite): A comprehensive basket of criterions ( c ) to investigate dominant experts across different SMoE blocks from weight, expert behavior, intermediate activations, and gradient behavior perspective . Criterion with indicate it has been previously explored either in exactly the same formulation or with slight variation. Based on the score of a criterion ( score {}_{c}^{e} ) estimated within a MoE layer, an expert (e) is identified and removed. Figure 2: Wikitext Perplexity of Mixtral 8 \times 7B pretrained checkpoint when removing a single expert e from layer l . In this work, we present MoE Experts Compression Suite (MC-Suite) , a comprehensive collection of potential recipes for expert importance estimation which studies “clues” from four broad and diverse perspectives: a expert & router weight dynamics, b expert inference behavior dynamics, c intermediate activation properties, and d expert gradient properties. In addition to expert importance, MC-Suite unveils numerous valuable insights across experts: dominant experts tend to have lower stable-rank ( i.e., pretraining knowledge is well compressed ( Jaiswal et al., 2024 ) ) which is favorable for additional compression using low-rank factorization; intermediate activation and gradients corresponding to dominant experts tend to have higher entropy indicating better information quantity and conducive finetuning abilities for downstream adaptation ( Zhang et al., 2024 ; Zhao et al., 2024 ) ; among many others as outlined in Section 3 . It is important to note that dropping experts involves deleting its entry in the router gating function, which leaves the MoE subnetwork in a sub-optimal state ( i.e., increased skewness in load distribution across retained experts, abrupt drop in performance with high dropping ratio). Most existing prior works ( Lu et al., 2024 ; He et al., 2024 ; Muzio et al., 2024 ) adopt one-shot criterion estimation for expert removal that alleviates impact incurred due to sparsification in the form of load imbalance and abrupt performance drop. In this work, we systematically illustrate that extending one-shot pruning to iterative pruning with re-estimation of importance criterion in k -rounds 1 1 1 Our experiments in Section 3.3 illustrate that subnetworks identified from one-shot vs. iterative pruning are significantly different. We conclude that iterative pruning helps in improving subnetwork quality while additional finetuning helps in retaining the capabilities of subnetwork to avoid abrupt performance degradation. , leads to identifying a better subset of experts for dropping. Moreover, motivated by lottery ticket hypothesis ( Frankle & Carbin, 2018 ) , we propose MoE Lottery Subnetwork which involves task-agnostic budget finetuning 2 2 2 Our experiments in Appendix B confirms that a limited number of training iterations are sufficient to address the sub-optimal state of MoE subnetwork produced after expert-level sparsification. using next-token prediction objective to address the intermediate sub-optimal state induced due to expert-level sparsification. More specifically, the MoE lottery subnetwork is derived using an iterative estimation-prune-finetune procedure, and our experiments illustrate that the task-agnostic finetune submodule can help in load distribution across remaining experts along with improving the performance. To unveil the true merits of expert-level sparsification, in this work we ask an interesting question: Given the existence of redundancy across experts, during expert-level sparsification, what capabilities of full-MoE are severely impacted? We hypothesize that during expert-level sparsification

Read the complete text