Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training
Rui Pan, Shivanshu Shekhar, Boyao Wang, Shizhe Diao, Jipeng Zhang, Xingyuan Pan, Renjie Pi, Tong Zhang · 2025
Read Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training by Rui Pan, Shivanshu Shekhar, Boyao Wang, Shizhe Diao, Jipeng Zhang, Xingyuan Pan, Renjie Pi, Tong Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03460.txt.
Description
Rui Pan and co-authors present Adapt-Pruner, a 2025 method for making small language models. Small models suit phones and other edge devices. Training them from scratch costs a lot. Cutting down a large model is cheaper, but it often hurts quality. The team studied methods that mix cutting, called pruning, with further training.
Their method prunes each layer by a fitting amount, rather than one amount for all. Pruning a small share of neurons at a time, then training again, worked best. On a large test model, Adapt-Pruner beat three rival pruning methods on common sense tasks. It also restored a small model's scores with far less training data than starting fresh. In one case it produced a new model that beat a well known rival of the same size.
Catalog record
| CREATOR | Rui Pan, Shivanshu Shekhar, Boyao Wang, Shizhe Diao, Jipeng Zhang, Xingyuan Pan, Renjie Pi, Tong Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-03460