Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training

Rui Pan, Shivanshu Shekhar, Boyao Wang, Shizhe Diao, Jipeng Zhang, Xingyuan Pan, Renjie Pi, Tong Zhang · 2025

Read Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training by Rui Pan, Shivanshu Shekhar, Boyao Wang, Shizhe Diao, Jipeng Zhang, Xingyuan Pan, Renjie Pi, Tong Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03460.txt.

Description

Rui Pan and co-authors present Adapt-Pruner, a 2025 method for making small language models. Small models suit phones and other edge devices. Training them from scratch costs a lot. Cutting down a large model is cheaper, but it often hurts quality. The team studied methods that mix cutting, called pruning, with further training.

Their method prunes each layer by a fitting amount, rather than one amount for all. Pruning a small share of neurons at a time, then training again, worked best. On a large test model, Adapt-Pruner beat three rival pruning methods on common sense tasks. It also restored a small model's scores with far less training data than starting fresh. In one case it produced a new model that beat a well known rival of the same size.

Catalog record

CREATORRui Pan, Shivanshu Shekhar, Boyao Wang, Shizhe Diao, Jipeng Zhang, Xingyuan Pan, Renjie Pi, Tong Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-03460

Browse the full library · About the Tower of Babel