SlimGPT: Layer-wise Structured Pruning for Large Language Models
Gui Ling, Ziyang Wang, Yuliang Yan, Qingwen Liu · 2024
Read SlimGPT: Layer-wise Structured Pruning for Large Language Models by Gui Ling, Ziyang Wang, Yuliang Yan, Qingwen Liu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18110.txt.
Introduction
Gui Ling, Ziyang Wang, Yuliang Yan, and Qingwen Liu present SlimGPT in a 2024 paper, a low-cost way to prune large language models layer by layer. Big models are hard to run in the field for their vast size. Structured pruning cuts whole parts away, to strike a balance of skill and speed, but the lost skill is hard to win back on a tight budget.
SlimGPT rests on the Optimal Brain Surgeon frame. A batched greedy step prunes fast and near the best point. Grouped math sharpens the error call per attention head, and a shifting group size speeds the feedforward cuts.
Catalog record
| CREATOR | Gui Ling, Ziyang Wang, Yuliang Yan, Qingwen Liu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-18110
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SlimGPT: Layer-wise Structured Pruning for Large Language Models Gui Ling, Ziyang Wang, Yuliang Yan, Qingwen Liu Abstract Large language models (LLMs) have garnered significant attention for their remarkable capabilities across various domains, whose vast parameter scales present challenges for practical deployment. Structured pruning is an effective method to balance model performance with efficiency, but performance restoration under computational resource constraints is a principal challenge in pruning LLMs. Therefore, we present a low-cost and fast structured pruning method for LLMs named SlimGPT based on the Optimal Brain Surgeon framework. We propose Batched Greedy Pruning for rapid and near-optimal pruning, which enhances the accuracy of head-wise pruning error estimation through grouped Cholesky decomposition and improves the pruning efficiency of FFN via Dynamic Group Size, thereby achieving approximate local optimal pruning results within one hour. Besides, we explore the limitations of layer-wise pruning from the perspective of error accumulation and propose Incremental Pruning Ratio, a non-uniform pruning strategy to reduce performance degradation. Experimental results on the LLaMA benchmark show that SlimGPT outperforms other methods and achieves state-of-the-art results. 1 Introduction Large Language Models (LLMs) [ 1 , 2 , 3 ] have made significant strides in various natural language processing tasks, leading to the emergence of novel applications such as AI agents [ 4 ] . One of the factors contributing to the exceptional capabilities of LLMs is their massive parameter scales. However, these extensive parameters also introduce increased inference costs and deployment challenges, hindering the widespread application and adoption of LLMs. Accelerating inference for LLMs has become a focal point of current research. Model compression [ 5 ] , as one of the strategies for inference acceleration, including techniques like pruning and quantization [ 6 , 7 ] , has been extensively researched. Nevertheless, earlier model compression techniques, particularly model pruning, typically rely on heavy post-training to recover the model’s capabilities, which typically involves retraining with the entire training dataset. Given the constraints of current computational resources, the above approaches are not feasible for LLMs. In the domain of LLM pruning, recent studies have largely focused on unstructured (or semi-structured) pruning [ 8 ] , a method that shrinks models by selectively zeroing out weights considered non-critical. Despite its advancements, unstructured pruning falls short in substantially reducing parameter count, which is crucial for accelerating LLM inference as it is often bottlenecked on memory bandwidth and communication [ 9 ] . To accelerate inference speed, unstructured pruning models are often paired with specialized frameworks or hardware solutions. Conversely, structured pruning [ 10 , 11 ] effectively decreases the model’s parameter count by systematically eliminating columns or rows from weight matrices, enabling significant improvements in inference speed, and reduce deployment cost on conventional hardware. Yet, structured pruning often entails more pronounced compromises in model performance, which poses a greater challenge. Recently, researchers have applied the classic Optimal Brain Surgeon (OBS) framework to the compression of LLMs. This approach includes parameter compensation which can mitigate the loss incurred during compression and reduce the dependence on post-training. The OBS framework is currently applied in the areas of unstructured pruning [ 12 ] and quantization [ 13 ] for LLMs. However, there exist some challenges in its application to structured pruning: • The OBS is a fine-grained compression framework that compresses one parameter at each iteration, whereas structured pruning has a minimum granularity of either a column or head. Directly applying the OBS framework will result in high numerical errors, impairing model performance. • The OBS is essentially a layer-wise compression method. It focuses on each individual layer, thus failing to allocate pruning ratios for each layer rationally using global information (such as global gradients). This is crucial for LLM structured pruning, which relies on a non-uniform strategy to reduce the impact on performance. To address these issues, we propose a new structured pruning method for LLMs. We introduce Batched Greedy Pruning to achieve low-cost and rapid pruning for LLMs. Specifically, for attention heads, we propose grouped Cholesky decomposition to select nearly optimal heads for pruning in each iteration, thereby maintaining an approximately locally optimal pruning result. For Feed-Forward Networks (FFNs), we achieve near-optimal and efficient pruning results through Dynamic Group Size. Furthermore, since the OBS is essentially a layer-wise compression framework, we investigate the error accumulation phenomenon in layer-wise pruning and propose pruning by Incremental Pruning Ratio, a straightforward non-uniform strategy to control the pruning rate of each layer, further mitigating performance loss under a given overall pruning ratio. Contribution. In this paper, we propose SlimGPT, a layer-wise pruning approach that extends the classical OBS framework to structured pruning for LLMs. The characteristics of SlimGPT can be summarized as follows: (i) Task-agnostic pruning scheme. Only a random sample of data from generic pre-training corpora is needed as a calibration set, and we can obtain a compressed model with most performance preserved; (ii) Low-cost, low-resource, and time-efficient compression scheme. The model can be compressed using just a single GPU, a few hundred of calibration data, and about one hour; (iii) A universal pruning method for Transformer-based models. It has good transferability and, theoretically, is applicable to all large models based on the conventional Transformer architecture. We employ LLaMA models for pruning and conduct evaluations on wikitext2 and Commonsense Reasoning tasks. The results indicate that SlimGPT substantially retains the performance of the pruned models, surpassing state-of-the-art methods. 2 Related Work Compression methods with regularization . Before the era of LLMs, using the scaling factors from Batch Normalization layers as indicators of channel importance made pruning based on regularization a very popular method [ 14 , 15 ] . Notably, Louizos et al. [ 16 ] implemented the non-differentiable L0 penalty in a differentiable form, a technique frequently used for pruning in large models. Compresso [ 17 ] combines L0 regularization with LoRA training [ 18 ] , effectively preserving model performance at a low cost. In a similar vein, Sheared LLaMA [ 19 ] employs augmented L0 regularization on inserted masks for structured pruning, using extensive data to restore performance and deliver compact yet powerful pruned models. Global gradient-based compression methods . NVIDIA’s works [ 20 , 21 ] involve a Taylor expansion of the global loss. By eliminating higher-order terms, it is revealed that the impact of a weight on the loss can be assessed using the magnitude of the weight combined with gradient information. Based on this, LLM-Pruner [ 11 ] employs a first-order importance estimation to gauge the importance of weights. LORAPrune [ 22 ] measures the importance of weights based on the gradients of the LORA parameters rather than the model’s parameters, achieving commendable results. Outliers-dependent compression methods . Dettmers et al. [ 23 ] identifies an attribute unique to LLMs, where a small subset of activation values in the data features have magnitudes significantly larger than the others. And removing corresponding weights impacts model performance substantially. Building upon this, Wanda [ 24 ] proposes a simple yet effective unstructured pruning method, using the product of a weight’s L1 norm and the L2 norm of eigenvalues to gauge its importance, achieving impressive pruning results. OWL [ 25 ] determines layer-wise sparsity ratios based on Layerwise Outlier Distribution (LOD), obtaining substantial performance gains at high