Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density
Zhendong Mi, Yixiao Chen, Pu Zhao, Xiaodong Yu, Hao Wang, Yanzhi Wang, Shaoyi Huang · 2026
Read Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density by Zhendong Mi, Yixiao Chen, Pu Zhao, Xiaodong Yu, Hao Wang, Yanzhi Wang, Shaoyi Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09316.txt.
Introduction
Large language models built as a mixture of experts gain power by routing work among many specialist networks, but storing all those experts makes them heavy to run. A 2026 paper led by Zhendong Mi, with six co-authors, offers a compression method that trims this burden without treating every expert alike. The core insight is that experts differ greatly in how often they are called on and how much distinct information they carry.
The method, named RFID-MoE, scores each expert group by combining routing frequency with a measure of information content. It then gives higher rank to the groups that matter most under a fixed size budget. Instead of throwing away the leftover compression error, it rebuilds that residue through a lean sparse projection. Tests on representative mixture-of-experts models show it beating leading rivals, with clear gains in prediction quality and zero-shot accuracy.
Catalog record
| CREATOR | Zhendong Mi, Yixiao Chen, Pu Zhao, Xiaodong Yu, Hao Wang, Yanzhi Wang, Shaoyi Huang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-09316
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density Zhendong Mi, Yixiao Chen, Pu Zhao, Xiaodong Yu, Hao Wang, Yanzhi Wang, Shaoyi Huang Abstract Mixture-of-Experts (MoE) based Large Language Models (LLMs) have achieved superior performance, yet the massive memory overhead caused by storing multiple expert networks severely hinders their practical deployment. Singular Value Decomposition (SVD)-based compression has emerged as a promising post-training technique; however, most existing methods apply uniform rank allocation or rely solely on static weight properties. This overlooks the substantial heterogeneity in expert utilization observed in MoE models, where frequent routing patterns and intrinsic information density vary significantly across experts. In this work, we propose RFID-MoE, an effective framework for MoE compression by exploiting heterogeneous Routing Frequency and Information Density. We first introduce a fused metric that combines expert activation frequency with effective rank to measure expert importance, adaptively allocating higher ranks to critical expert groups under a fixed budget. Moreover, instead of discarding compression residuals, we reconstruct them via a parameter-efficient sparse projection mechanism to recover lost information with minimal parameter overhead. Extensive experiments on representative MoE LLMs (e.g., Qwen3, DeepSeekMoE) across multiple compression ratios demonstrate that RFID-MoE consistently outperforms state-of-the-art methods like MoBE and 𝐃 2 \mathbf{D}^{2} -MoE. Notably, RFID-MoE achieves a perplexity of 16.92 on PTB with the Qwen3-30B model at a 60% compression ratio, reducing perplexity by over 8.0 compared to baselines, and improves zero-shot accuracy on HellaSwag by approximately 8%. 1 Introduction The landscape of Natural Language Processing (NLP) has been reshaped by Transformer-based LLMs with strong performance across tasks ( 42 ) . To balance massive capacity demands with computational constraints, recent architectures adopt Mixture-of-Experts (MoE) designs using specialized expert networks with sparse activation ( 41 ) , enabling models like DeepSeek ( 24 ) and Qwen3 ( 46 ) to scale effectively ( 13 ) . However, MoE models face severe deployment challenges from substantial memory overhead ( 40 ; 49 ; 14 ) . For instance, Qwen3-235B-A22B-2507 exceeds the 320GB capacity of an 8 \times 40GB A100 cluster, causing out-of-memory failures ( 39 ) . Existing MoE compression can be categorized into three categories: expert pruning, expert merging, and decomposition. Pruning removes experts but often loss specialized knowledge, degrading performance on domain-specific tasks ( 44 ; 27 ; 10 ) . Expert merging consolidates experts assuming functional redundancy, but experts often exhibit distinct specializations ( 21 ; 4 ; 26 ) . Decomposition methods, particularly SVD-based approaches ( 23 ; 19 ; 5 ) , compress expert weights without reducing expert count, better preserving capacity. However, existing SVD-based methods face several limitations. Challenge 1: Employing uniform rank without considering imbalanced expert utilization may lead to suboptimal compression performance. Expert utilization in MoE models exhibits significant non-uniformity across practical inference workloads, with some experts frequently activated while others remain largely underutilized. Despite this heterogeneity, existing compression methods predominantly assign uniform ranks to all experts or expert groups ( 5 ; 22 ; 12 ) , leading to suboptimal performance. Even approaches that consider differences between experts typically rely exclusively on static weight properties ( 45 ; 19 ) , neglecting dynamic activation patterns during inference. While incorporating routing frequency as a criterion appears intuitive, it presents a fundamental challenge: activation distributions in large MoEs are often severely skewed, with some experts never activated, therefore with 0 rank allocated. Thus, compression that allocate ranks based purely on routing frequency degenerate into expert pruning, risking severe performance degradation on specialized tasks where infrequently-activated experts encode critical domain-specific knowledge ( 5 ; 10 ) . Challenge 2: The compression residual is non-negligible. In SVD-based compression, the fundamental objective is to minimize reconstruction error between the compressed and original matrices, thereby preserving model performance. Standard SVD-based methods retain the dominant singular components while directly discarding the residual term ( 20 ; 5 ) , implicitly assuming negligibility of the approximation error. However, this assumption deserves closer examination. In particular, three important questions remain underexplored: (i) whether these residuals are genuinely negligible in magnitude and impact; (ii) what statistical properties characterize their distribution, and (iii) whether they can be reconstructed in a parameter-efficient manner instead of being discard entirely. To address the challenges, we propose RFID-MoE , an effective SVD-based expert compression method tailored for Mixture-of-Experts (MoE) models that exploits the heterogeneous R outing F requency and I nformation D ensity across expert groups. Specifically, we develop adaptive rank allocation strategy that joint considers expert routing frequency and information density. By allocating larger ranks to frequently routed and information-rich experts while assigning fewer ranks to less critical ones, our method better preserves task-relevant performance under a fixed compression budget. This approach avoids the suboptimal uniform rank assignment or purely static weight-based rank allocation used in prior decomposition-based methods. Additionally, rather than discarding the compression residual, we reconstruct it using a parameter-efficient mechanism. This design enables RFID-MoE to retain complementary information that cannot be captured by low-rank approximation alone, further enhancing performance with only marginal additional parameters. Our contributions are summarized as follows: • We propose RFID-MoE, an effective rank allocation compression method for MoEs that adaptively distributes compression rank budgets across experts to better preserve information with superior performance. • We propose using the effective rank to analyze the information density of each expert group. Building on this, we develop a novel rank allocation metric that combines effective rank with normalized routing frequency to better preserve information for critical expert groups during compression. • We analyze residual distribution patterns and propose a parameter-efficient reconstruction method to recover information lost in low-rank approximation. We conduct extensive experiments on representative MoE-based LLMs including DeepSeekMoE-16B-Base, Qwen3-30B-A3B-2507, and Ling-mini-2.0. RFID-MoE achieves state-of-the-art performance across diverse tasks and compression ratios, e.g., surpassing the best baseline by 8.0 PPL on PTB and 8% on HellaSwag at 60% compression ratio. 2 Preliminaries and Motivation 2.1 Mixture-of-Experts Architecture In Transformer-based models, the standard dense Feed-Forward Network (FFN) can be replaced with a MoE layer comprising a routing module and n independent expert networks. Following recent open-source models ( 38 ) , individual experts are typically implemented as FFNs with SwiGLU activation. For an input token representation x\in\mathbb{R}^{d} , the computation performed by the i -th expert E^{i}(x) consists of three linear transformations: an up-projection, a gate-projection, and a down-projection. The forward pass is formulated as: \small E^{i}(x)=W_{down}^{i}\cdot\left(W_{up}^{i}x\odot\text{SiLU}(W_{gate}^{i}x)\right) (1) where \odot denotes the element-wise product. W_{up}^{i},W_{gate}^{i}\in\mathbb{R}^{p\times d} and W_{down}^{i}\in\mathbb{R}^{d\times p} are the three learnable weight matrices, where p is the intermediate dimension. The router module G employs a learnable weight matrix W_{g}\in\mathbb{R}^{n\times d} to compute routing probabilities and select the top- K experts with the highest affinity scores: \small G(x)=\text{TopK}(\text{Softmax}(W_{g}x)) (2) The final output of the MoE layer is computed by aggregating the outputs of the selected experts, weighted by their corresponding routing probabilities. This computation is performed independently for each token: \small y=\sum_{i=1}^{K}G^{i}(x)E^{i}(x) (3) where K denotes the number of activated experts per token. Figure 1 : Routing frequency of experts for different layers of Qwen3-30B-A3B-2507 from layer 24 to 48. 2.2 SVD-based compression with basis sharing for MoE Weight matrices often exhibit significant redundancy ( 43 ; 5 ; 31 ) , enabling joint SVD decomposition across multiple