Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models

Chuan Sun, Han Yu, Lizhen Cui, Xiaoxiao Li · 2025

Read Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models by Chuan Sun, Han Yu, Lizhen Cui, Xiaoxiao Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01731.txt.

Introduction

Chuan Sun and co-authors present a smarter way to shrink large language models in a 2025 paper. Cutting parts of a model makes it cheaper to run, but cutting each layer by the same share wastes the fact that some layers matter far more than others. Their method grades each layer by what it adds to the whole model.

Layers that add more keep more of their parts, while layers that add less are cut more deeply. A sliding window shortcut makes the grading fast enough to use in practice. Tests on several known model families show clear gains over a leading fixed method. At high cut rates, error fell by about one fifth on two large models.

Catalog record

CREATORChuan Sun, Han Yu, Lizhen Cui, Xiaoxiao Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01731

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models Chuan Sun, Han Yu, Lizhen Cui, Xiaoxiao Li Abstract Pruning large language models (LLMs) is a promising solution for reducing model sizes and computational complexity while preserving performance. Traditional layer-wise pruning methods often adopt uniform sparsity at all Transformer layers, which leads to suboptimal performance due to the varying significance of transformer layers not being accounted for. To this end, we propose the S hapley V alue-based N on- U niform P runing ( SV-NUP ) method for LLMs. This method quantifies the contribution of each transformer layer to the overall model performance, enabling the assignment of tailored pruning budgets to different layers to retain critical parameters. To further improve efficiency, we design the S liding W indow-based S hapley V alue (SWSV) approximation method. It substantially reduces computational overhead compared to exact SV calculation methods. Extensive experiments on various LLMs including LLaMA-v1/v2/v3, and OPT demonstrate the effectiveness of SV-NUP . The results reveal that non-uniform pruning significantly enhances the performance of pruned models. Notably, SV-NUP achieves a reduction in perplexity (PPL) of 18.01% and 19.55% on LLaMA-7B and LLaMA-13B, respectively, compared to SparseGPT at 70% sparsity. 1 Introduction Large language models (LLMs) have emerged as a transformative technology, demonstrating remarkable capabilities in natural language understanding and generation tasks [ 1 ] . LLMs have showcased significant adaptability through fine-tuning, enabling their deployment in highly specialized applications. These advantages underscore the critical role of LLMs in solving real-world challenges [ 2 ] . Despite these capabilities, the deployment of LLMs has been hindered by their immense computational demands. Modern LLMs often consist of billions or even trillions of parameters [ 3 ] , as seen in models like GPT-3 (175 billion parameters) and PaLM (540 billion parameters). The immense scale incurs significant memory, storage and power costs, making it challenging to run these models on resource-constrained devices. To address these issues, researchers have increasingly turned to model compression techniques, particularly pruning, to reduce the size and computational requirements of LLMs, while retaining their performance [ 4 ] . Pruning [ 5 ; 6 ; 7 ] is one of the most prominent model compression techniques. It aims to remove redundant or less important parameters from neural networks, thereby reducing their sizes and computational complexity. Recent advances in pruning have progressed from unstructured sparsity (i.e., individual weight removal) to structured sparsity (i.e., eliminating entire neurons, heads or layers), thereby enabling hardware-efficient implementation [ 4 ; 8 ; 9 ; 10 ] . For LLMs, several state-of-the-art methods have been proposed, including magnitude pruning, lottery ticket hypothesis approaches, and structured pruning based on attention mechanisms. These methods demonstrate the potential of pruning to enable LLM deployment on resource-constrained devices without causing substantial performance degradation. Figure 1: Overview and effectiveness of SV-NUP : (a) Distribution of weight magnitude across Transformer layers; (b) Conceptual comparison of uniform pruning vs. SV-NUP ; (c) PPL↓ (the smaller the better) comparison under different pruning strategies. The red number is the improvement percentage. Unlike uniform pruning, which ignores layer-wise importance and may degrade LLM performance, SV-NUP leverages Shapley value to estimate the contribution of each Transformer layer and allocates pruning ratios accordingly to better preserve model performance. Existing pruning methods often use a layer-wise strategy that applies uniform sparsity across all layers, ignoring their varying importance. While simple to implement, this approach overlooks the inherent differences in the contributions of different layers to the overall performance of the model. Thus, it can only find the local optimal pruning solution, but not the global optimal solution. Empirical evidence also suggests that certain layers are more critical than others, and uniformly pruning across all layers may lead to the removal of essential parameters, ultimately impairing the pruned model’s performance [ 11 ; 12 ] . To address this limitation, we leverage the concept of Shapley value from cooperative game theory and propose a novel S hapley V alue-based N on- U niform P runing ( SV-NUP ) method for LLMs, which was originally designed to fairly assess the contributions of multiple players in a game. In this study, we treat each Transformer layer in an LLM as a “player". Based on this idea, we evaluate the contribution of each Transformer layer to the overall performance of an LLM. The results are further used as a basis to allocate customized sparsity ratios to each Transformer layer, prioritizing the preservation of parameters in more important ones. This approach not only improves the performance of the pruned LLMs, but also introduces a theoretical foundation for sparsity allocation, moving beyond heuristic methods. We further propose a S liding W indow-based S hapley V alue approximation method (SWSV) to reduce the computational cost. The overview of SV-NUP is shown in Figure 1 . We conduct extensive experiments to evaluate the performance of SV-NUP for pruning LLMs across a spectrum of LLMs, including LLaMA-v1/v2/v3, and OPT. SV-NUP achieves a reduction in perplexity (PPL) of 18.01% and 19.55% on LLaMA-7B and LLaMA-13B, respectively, compared to SparseGPT at 70% sparsity. The results validate our hypothesis that non-uniform sparsity allocation can significantly enhance the performance of pruned models, paving the way for more efficient and practical LLM deployment on resource-constrained devices. 2 Related Work LLM pruning often requires retraining to recover performance, which presents substantial challenges. While existing LLM-specific pruning methods primarily adopt uniform pruning strategies [ 9 ; 13 ; 4 ; 14 ] , non-uniform layerwise sparsity has been extensively studied in vision models [ 15 ; 16 ; 11 ; 17 ; 18 ; 19 ; 20 ; 21 ; 22 ] . However, global pruning is often inefficient and computationally expensive for LLMs. Although prior work has analyzed the importance of LLM components [ 23 ; 24 ; 25 ; 26 ; 27 ; 28 ; 29 ] , none have explored Shapley value (SV)-based non-uniform pruning. SV-NUP fills this gap. For a more detailed literature review, refer to Appendix A . 3 Motivation In this section, we conduct empirical studies on three LLMs (TinyLLama, LLaMA-7B, and Mistral-7B) to motivate the need for allocating different pruning ratios across Transformer layers. We analyze the differences using the Magnitude method [ 30 ] and summarize each Transformer layer’s structure with the mean and standard deviation of innerlayer metrics. As shown in Figure 2 , two key observations can be drawn from the results. (a) TinyLLaMA with Mean (b) LLaMA-7B with Mean (c) Mistral-7B with Mean (d) TinyLLaMA with Std (e) LLaMA-7B with Std (f) Mistral-7B with Std Figure 2: Comparison of the mean and standard deviation (Std) across different Transformer layers under Magnitude. TinyLLaMA consists of 22 Transformer layers, LLama-7B and Mistral-7B consist of the same 32 Transformer layers, where each Transformer layer consists of 7 inner-layers. 3.1 Observations Obervation 1 Different Transformer layers in LLMs exhibit significant variations in the mean and std of magnitude. As the Transformer index increases, the mean magnitude decreases across all LLMs (Figures 2(a) , 2(b) and 2(c) ). Notably, the changes in LLaMA-7B and Mistral-7B are more pronounced compared to TinyLLaMa. The earlier (shallower) Transformer layers tend to have higher mean magnitudes. From the std of magnitude in Figures 2(d) ,

Read the complete text