Domain Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations

Zican Dong, Han Peng, Peiyu Liu, Wayne Xin Zhao, Dong Wu, Feng Xiao, Zhifeng Wang · 2025

Read Domain Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations by Zican Dong, Han Peng, Peiyu Liu, Wayne Xin Zhao, Dong Wu, Feng Xiao, Zhifeng Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06792.txt.

Description

Zican Dong and co-authors show how to shrink very large AI models for one field of use. The work dates to 2025. Such models hold many expert parts but use only a few at a time. Storing all parts still takes huge memory, which blocks use on small setups.

The team found that a few sample prompts in a field light up the same small set of parts each time. Their method keeps just those parts and drops the rest. It scores parts by gate scores and output size, and by how much each word shifts. Tests kept skill high while speed rose and memory use fell by half.

Catalog record

CREATORZican Dong, Han Peng, Peiyu Liu, Wayne Xin Zhao, Dong Wu, Feng Xiao, Zhifeng Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06792

Browse the full library · About the Tower of Babel