SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
Shengkun Tang, Zekun Wang, Bo Zheng, Liangyu Wang, Rui Men, Siqi Zhang, Xiulong Yuan, Zihan Qiu, Zhiqiang Shen, Dayiheng Liu · 2026
Read SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training by Shengkun Tang, Zekun Wang, Bo Zheng, Liangyu Wang, Rui Men, Siqi Zhang, Xiulong Yuan, Zihan Qiu, Zhiqiang Shen, Dayiheng Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08738.txt.
Description
Shengkun Tang and many coauthors study how to compress giant mixture of experts language models during pretraining, using pruning and teaching from a larger model. Three questions guide the work: whether pruning beats fresh training, how expert cuts play out after long continued training, and which teaching strategy works best. Their large scale tests answer each in turn.
Pruned starts beat scratch training at equal budget, rival expert cuts converge to similar ends, and merging experts with partial preservation helps most tasks. Teaching combined with normal training beats teaching alone, and gradual pruning beats one shot cuts. Their compressed model shrinks an eighty billion class system dramatically while staying competitive.
Catalog record
| CREATOR | Shengkun Tang, Zekun Wang, Bo Zheng, Liangyu Wang, Rui Men, Siqi Zhang, Xiulong Yuan, Zihan Qiu, Zhiqiang Shen, Dayiheng Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08738