Delta Decompression for MoE-based LLMs Compression
Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, Yike Guo · 2025
Read Delta Decompression for MoE-based LLMs Compression by Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, Yike Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17298.txt.
Introduction
Hao Gu and seven co-authors shrink mixture-of-experts language models by splitting what experts share from what makes each one distinct. In such models, many expert networks sit side by side, and only a few fire per input. The design is strong but eats storage and memory. Their 2025 method merges each expert's weights into a shared base, weighted by how much each part matters. The leftover differences, called deltas, are then compressed hard, since they turn out to have a simple, low-rank shape.
Rebuilding a model means adding its deltas back to the base. Smart compression of the differences keeps quality while cutting size. The authors name the scheme D2-MoE. It targets the real bottleneck for serving these models: fitting them on hardware at sane cost.
Catalog record
| CREATOR | Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, Yike Guo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-17298
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Delta Decompression for MoE-based LLMs Compression Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, Yike Guo Abstract Mixture-of-Experts (MoE) architectures in large language models (LLMs) achieve exceptional performance, but face prohibitive storage and memory requirements. To address these challenges, we present D 2 D^{2} -MoE, a new delta decompression compressor for reducing the parameters of MoE LLMs. Based on observations of expert diversity, we decompose their weights into a shared base weight and unique delta weights. Specifically, our method first merges each expert’s weight into the base weight using the Fisher information matrix to capture shared components. Then, we compress delta weights through Singular Value Decomposition (SVD) by exploiting their low-rank properties. Finally, we introduce a semi-dynamical structured pruning strategy for the base weights, combining static and dynamic redundancy analysis to achieve further parameter reduction while maintaining input adaptivity. In this way, our D 2 D^{2} -MoE successfully compact MoE LLMs to high compression ratios without additional training. Extensive experiments highlight the superiority of our approach, with over 13% performance gains than other compressors on Mixtral|Phi-3.5|DeepSeek|Qwen2 MoE LLMs at 40 ∼ \sim 60% compression rates. Codes are available in https://github.com/lliai/D2MoE. † † affiliationnotice: Equal contribution 1 Introduction Recent advances in Large Language Models (LLMs) increasingly favor Mixture of Experts (MoE) ( Cai et al., 2024 ) architectures for their ability to scale model capacity through specialized expert networks while maintaining computational efficiency via sparse activation. The success of MoE is evident in recent LLMs like DeepSeek-V3 ( DeepSeek-AI et al., 2024 ) and MiniMax-01 ( MiniMax et al., 2025 ) , which demonstrate unprecedented capabilities in language understanding and generation tasks. Despite their compelling advantages, MoE LLMs face critical challenges in practical deployment scenarios ( Tang et al., 2024 ; Zhong et al., 2024 ; Hwang et al., 2024 ) . Their substantial parameter footprint, coupled with considerable memory overhead from storing multiple expert weights ( Song et al., 2023 ) , creates significant barriers to resource-constrained environments. Table 1 : Comparison of our method with other MoE compressors. Diversity means retaining individual per-expert information. Maximum ratio denotes the maximum parameter compression ratio. Method Strategy Structured Train-free Diversity Max-Ratio NAEE ( 2024b ) Prune ✓ ✗ ✗ 50% MoE-Compress ( 2024 ) Prune ✗ ✓ ✗ 50% MoE-Pruner ( 2024 ) Prune ✗ ✗ ✗ 50% MoE- I^{2} ( 2024 ) Prune ✓ ✗ ✗ 55% MC-SMoE ( 2023a ) Merge ✗ ✗ ✗ 75% HC-SMoE ( 2024 ) Merge ✓ ✓ ✗ 50% EEP ( 2024a ) Merge ✓ ✓ ✗ 75% D^{2} -MoE (Ours) Delta ✓ ✓ ✓ 80% To address these challenges, MoE compression methods have recently gained significant attention. As illustrated in Table 1 , current approaches broadly categorized into expert pruning and expert merging methods. (1) Expert pruning approaches, represented by MoE-Pruner ( Xie et al., 2024 ) , NAEE ( Lu et al., 2024a ) , and MoE-I 2 ( Yang et al., 2024 ) , implement inter-expert pruning and intra-expert weight sparsification. While these approaches achieve significant parameter reduction, they often result in substantial performance degradation due to the irreversible loss of expert knowledge. The direct removal of expert weights compromises the model’s specialized capabilities, frequently necessitating additional fine-tuning to partially recover performance. (2) Expert merging methods , on the other hand, aim to consolidate multiple experts into fewer, more compact representations. Methods like EEP ( Liu et al., 2024a ) , MC-SMoE ( Li et al., 2023a ) , and HC-SMoE ( Chen et al., 2024 ) develop various weighting schemes for weighted summation of different experts’ weights. While these approaches preserve more information than direct pruning, it introduces new challenges. The merging process assumes significant overlap in expert functionalities, but in practice, experts often possess distinct, complementary specializations. This leads to a fundamental dilemma: experts with similar weights can be effectively merged, but those with dissimilar yet important weights resist efficient compression, resulting in either suboptimal compression ratios or performance degradation. These challenges present the question: How can we design new frameworks beyond pruning and merging methods in effectively balancing compression and preserving expert diversity? Figure 1 : Centered Kernel Alignment (CKA) similarity of experts weights of Mixtral-8x7B, Phi-3.5-MoE, DeepSeekMoE-16B-Base. "Diversity is not about how we differ. Diversity is about embracing one another’s uniqueness." — Ola Joseph As the quote goes, recent fine-tuning methods ( Ping et al., 2024 ) quantize delta weights between fine-tuned and original models to effectively capture both similarities and variations. Inspired by these successes, we investigate whether it is possible to recycle the difference (delta) weights that are always discarded during expert merging to maintain performance without introducing excessive computational or memory overhead. Specifically, we brainstorm the idea to efficiently reallocate these abandoned delta weights (differences between merged expert weights than original weights) to preserve the diversity and specialization of experts. To explore this, we conduct two key experiments to analyze the properties of expert weights in MoE LLMs: (1) We evaluate expert similarity using centered kernel alignment (CKA) metrics. As shown in Figure 1 , the similarity between different expert weights consistently falls within the 0.3 to 0.5 range. This indicates a moderate overlap in their feature spaces, suggesting that while some aspects of their weights can be merged, preserving expert diversity remains crucial. (2) We examine distributions of single values energy retention (detailed in Appendix B.2 ) for different expert weight decompositions, as illustrated in Figure 2 . The larger singular values energy retentions in the delta weights show that most of the matrix’s information is concentrated in a small number of singular vectors, indicating a strong low-rank structure. This shows that these delta weights can be efficiently approximated using low-rank decomposition methods without excessive degradation of information. These findings underscore that reutilizing delta weights to expert merging is a promising way for MoE compression that balances efficiency, diversity, and performance. Building on these insights, we develop D^{2} -MoE, a novel compression framework to address the growing challenges of parameter redundancy, memory overhead, and storage inefficiency in MoE LLMs while preserving model performance and scalability. Rather than directly removing or merging experts, our approach strategically decomposes expert weights into a shared base weight, which captures the commonalities across all experts, and a delta weight, which encodes the expert-specific variations. This decomposition not only reduces redundancy but also facilitates efficient compression of delta weights by exploiting their inherent low-rank structure. To ensure that the shared base weight accurately represents the most critical information across experts, D^{2} -MoE incorporates a Fisher-weighted averaging mechanism. This approach computes the shared base weight by weighting each expert’s contribution based on its Fisher importance, which quantifies the sensitivity of the model’s parameters to the input data. By prioritizing the contributions of the most important experts, Fisher-weighted averaging balances the trade-off between redundancy reduction and representational fidelity. To further compress the delta weights, D^{2} -MoE employs a truncation-aware SVD method that integrates activation statistics into the decomposition process. This method adjusts the singular value truncation threshold based on the input activation patterns, ensuring that essential information is preserved while compressing delta weights. Finally, D^{2} -MoE proposes semi-dynamical structured pruning on the shared base weight, combining