DLP: Dynamic Layerwise Pruning in Large Language Models

Yuli Chen, Bo Cheng, Jiale Han, Yingying Zhang, Yingting Li, Shuhao Zhang · 2025

Read DLP: Dynamic Layerwise Pruning in Large Language Models by Yuli Chen, Bo Cheng, Jiale Han, Yingying Zhang, Yingting Li, Shuhao Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23807.txt.

Introduction

Yuli Chen and co-authors prune language models layer by layer at rates set by each layer's measured worth, rather than one flat rate. Their 2025 method answers a known failure: even pruning hurts badly once most weights are cut. Layers do not contribute alike, yet preset non-even rates can also miss.

The method blends weight data with live activation signals to score each layer on the fly. Pruning then follows those scores. At seventy percent sparsity, the method cut a standard model's confusion score well below rival tools and raised average task accuracy. It also pairs with other compression tricks and with light fine-tuning. Code is shared in public for follow-up work.

Catalog record

CREATORYuli Chen, Bo Cheng, Jiale Han, Yingying Zhang, Yingting Li, Shuhao Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23807

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DLP: Dynamic Layerwise Pruning in Large Language Models Yuli Chen, Bo Cheng, Jiale Han, Yingying Zhang, Yingting Li, Shuhao Zhang Abstract Pruning has recently been widely adopted to reduce the parameter scale and improve the inference efficiency of Large Language Models (LLMs). Mainstream pruning techniques often rely on uniform layerwise pruning strategies, which can lead to severe performance degradation at high sparsity levels. Recognizing the varying contributions of different layers in LLMs, recent studies have shifted their focus toward non-uniform layerwise pruning. However, these approaches often rely on pre-defined values, which can result in suboptimal performance. To overcome these limitations, we propose a novel method called Dynamic Layerwise Pruning (DLP). This approach adaptively determines the relative importance of each layer by integrating model weights with input activation information, assigning pruning rates accordingly. Experimental results show that DLP effectively preserves model performance at high sparsity levels across multiple LLMs. Specifically, at 70% sparsity, DLP reduces the perplexity of LLaMA2-7B by 7.79 and improves the average accuracy by 2.7% compared to state-of-the-art methods. Moreover, DLP is compatible with various existing LLM compression techniques and can be seamlessly integrated into Parameter-Efficient Fine-Tuning (PEFT). We release the code 1 1 1 The code is available at: https://github.com/ironartisan/DLP . to facilitate future research. 1 Introduction Pruning ( Jaiswal et al., 2023 ; Ma et al., 2023 ; Sun et al., 2024 ; Muralidharan et al., 2024 ; Cai et al., 2024 ; Men et al., 2024 ) has garnered significant attention in both academia and industry due to its ability to substantially reduce the parameter count of Large Language Models (LLMs) ( OpenAI, 2023 ; Touvron et al., 2023a ; Touvron et al., 2023b ; Dubey et al., 2024 ) . The core concept of pruning is to optimize resource utilization by eliminating redundant or less important parameters. SparseGPT ( Frantar & Alistarh, 2023 ) implements a layer-by-layer and row-by-row greedy pruning strategy, ensuring that local optimizations have minimal impact on global performance. Recent studies ( Xiao et al., 2023 ; Lee et al., 2023 ; Lin et al., 2024 ) highlight the pivotal role of outliers in LLMs. Although outliers constitute a small fraction of the model, they exert a disproportionately large influence on predictive accuracy. Building on the emergence of outlier features in LLMs ( Puccetti et al., 2022 ; Lee et al., 2023 ; Lin et al., 2024 ) , Wanda ( Sun et al., 2024 ) introduces a novel approach to evaluate weight importance by integrating the absolute weight values with the norm of the corresponding input activations. Although previous works ( Frantar & Alistarh, 2023 ; Zhang et al., 2024 ; Sun et al., 2024 ) have achieved satisfactory performance, they fail to account for the varying importance of different layers within the model, instead assigning a uniform sparsity rate to all layers. This limitation leads to a significant performance drop under high sparsity conditions. Inspired by the presence of outliers, Outlier Weighed Layerwise Sparsity (OWL) ( Yin et al., 2024 ) introduces a novel pruning paradigm that leverages the criticality of layers with a higher proportion of outliers. Based on the principle that layers with a higher proportion of outliers are more critical, OWL assigns different sparsity rates to each layer of LLMs. In comparison to uniform layerwise pruning ( Zhu & Gupta, 2018 ) , OWL demonstrates superior performance in preserving model accuracy. However, OWL still has certain limitations in practical applications. Its reliance on predefined criteria for outlier selection not only limits its adaptability to the dynamic needs of the model but also hinders the achievement of optimal performance. To address the above issue, we compute the unimportance of each layer from an inverse perspective, which is then transformed into the relative importance between layers. Based on the principle that layers with higher importance should have lower sparsity, we allocate layerwise sparsity rates. Some previous works ( He et al., 2019 ; Zhang et al., 2023 ) use the median to identify redundant elements in a model, assuming that central elements can be replaced by other elements from the same layer. We demonstrate the effectiveness of the median in LLMs through three empirical studies. Additionally, we inherently place more emphasis on outliers. Due to the median’s insensitivity to outliers ( Huber et al., 2001 ) , it provides a more accurate reflection of the central tendency of a layer when the weights contain outliers. In this paper, we propose a novel Dynamic Layerwise Pruning (DLP) method. DLP adaptively determines the importance of each layer by combining model weights with input activation information, offering greater flexibility in sparsity allocation. Our goal is to determine the layerwise importance of LLMs, which we first derive by identifying the layerwise unimportance and then applying an inversion operation to obtain relative importance. Specifically, we begin by calculating the unimportance of each Transformer block based on the median of model weights and input activation values in the same layer. We then evaluate the relative unimportance across layers, which leads to the determination of the model’s relative importance. Finally, pruning rates are assigned to each layer according to the principle that layers with higher importance should have lower sparsity. The pipeline of DLP is illustrated in Figure 1 . We conduct comprehensive experimental evaluations across multiple mainstream LLMs with varying parameter sizes (ranging from 7B to 30B) and architectures (e.g., LLaMA ( Touvron et al., 2023a ) , Vicuna ( Chiang et al., 2023 ) , Mistral ( Jiang et al., 2023 ) ). The experimental results show that our method consistently outperforms the state-of-the-art LLM pruning techniques, particularly at high sparsity levels. For instance, at 70% sparsity, DLP reduces the perplexity of LLaMA2-7B by 7.79 and improves the average accuracy by 2.7%. When evaluated on the DeepSparse ( Kurtic et al., 2023 ) inference engine, DLP achieves 2.8x-3.7x end-to-end acceleration on CPU with 70% - 90% sparsity. Furthermore, we find that a brief period of fine-tuning can restore the LLM to a reasonable range after high sparsity pruning. As a general method, our approach can be applied to unstructured pruning, as well as N:M sparsity ( Sun et al., 2021 ) and structured pruning ( Ma et al., 2023 ) , consistently outperforming layerwise methods. Our method is orthogonal to quantization ( Dettmers et al., 2022 ; Xiao et al., 2023 ; Balanca et al., 2024 ) , and it can also be extended to Singular Value Decomposition (SVD) and Parameter-Efficient Fine-Tuning (PEFT) ( Liu et al., 2022a ) , achieving substantial performance improvements. Overall, the contributions of our work are as follows: • We propose a novel method for measuring layerwise importance that does not rely on empirical values or model type. This method comprehensively considers both intra-layer and inter-layer elements to automatically determine the relative importance of each layer. • We propose an effective method for unstructured pruning. Extensive experimental results consistently show that, at high sparsity levels, DLP not only outperforms state-of-the-art pruning techniques for LLMs but also achieves significant end-to-end speedups on CPU. • Our method can not only be integrated with LLM compression techniques but also extended to PEFT. Figure 1: Illustration of Uniform Layerwise Pruning and Dynamic

Read the complete text