Domain Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
Zican Dong, Han Peng, Peiyu Liu, Wayne Xin Zhao, Dong Wu, Feng Xiao, Zhifeng Wang · 2025
Read Domain Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations by Zican Dong, Han Peng, Peiyu Liu, Wayne Xin Zhao, Dong Wu, Feng Xiao, Zhifeng Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06792.txt.
Description
Zican Dong and co-authors show how to shrink very large AI models for one field of use. The work dates to 2025. Such models hold many expert parts but use only a few at a time. Storing all parts still takes huge memory, which blocks use on small setups.
The team found that a few sample prompts in a field light up the same small set of parts each time. Their method keeps just those parts and drops the rest. It scores parts by gate scores and output size, and by how much each word shifts. Tests kept skill high while speed rose and memory use fell by half.
Catalog record
| CREATOR | Zican Dong, Han Peng, Peiyu Liu, Wayne Xin Zhao, Dong Wu, Feng Xiao, Zhifeng Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06792