Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization
Guanchen Li, Yixing Xu, Zeping Li, Ji Liu, Xuanwu Yin, Dong Li, Emad Barsoum · 2025
Read Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization by Guanchen Li, Yixing Xu, Zeping Li, Ji Liu, Xuanwu Yin, Dong Li, Emad Barsoum (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09657.txt.
Introduction
Guanchen Li and co-authors present a structural pruning method for large language models in their 2025 arXiv paper. Pruning cuts model size for faster inference, but global pruning methods often judge parts in isolation and miss how structures depend on each other. Local layer-by-layer pruning, in turn, ignores the model's whole shape.
Their framework builds a supernet by applying local pruning at many sparsity levels to every layer, then searches for the best global sparsity pattern under a target budget. An iterative prune-and-search loop with finer and finer steps speeds convergence. On a seventy-billion-parameter model, the method removed half the parameters while keeping ninety-seven percent of performance.
Catalog record
| CREATOR | Guanchen Li, Yixing Xu, Zeping Li, Ji Liu, Xuanwu Yin, Dong Li, Emad Barsoum |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-09657
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization Guanchen Li, Yixing Xu, Zeping Li, Ji Liu, Xuanwu Yin, Dong Li, Emad Barsoum Abstract Structural pruning enhances hardware-agnostic inference efficiency for large language models (LLMs) yet often fails to maintain comparable performance. Local pruning performs efficient layer-by-layer compression but ignores global topology. Although global pruning aims to identify an optimal sparse model, intuitive methods typically adopt a two-stage paradigm that first evaluates substructure saliency and then applies global pruning, which ignores inter-structure dependencies and fails to achieve end-to-end optimization. To address these limitations, we propose Týr-the-Pruner , an efficient end-to-end search-based global structural pruning framework. This framework constructs a supernet by repeatedly applying local pruning across a range of sparsity ratios to each layer in an LLM, with the core goal of determining the optimal sparsity distribution under a target overall sparsity ratio. Concretely, we introduce an effective local pruning and an expectation error accumulation approach to improve supernet construction. Furthermore, we employ an iterative prune-and-search strategy with coarse-to-fine sparsity granularity to ensure efficient search convergence. Experimental results show that Týr-the-Pruner achieves state-of-the-art structural pruning, retaining 97% of the dense model’s performance while removing a challenging 50% of Llama-3.1-70B’s parameters. Code will be available at https://github.com/AMD-AGI/Tyr-the-Pruner . 1 Introduction Large language models (LLMs) have significantly advanced natural language processing, achieving exceptional performance in tasks such as text understanding, generation, and reasoning [ 53 , 7 , 3 ] . However, the computational and storage resources required for model deployment incur high costs and environmental impacts, limiting their accessibility in resource-constrained scenarios. Model compression techniques, such as quantization [ 21 , 10 ] , pruning [ 9 , 23 , 39 ] , and low-rank decomposition [ 45 ] , are essential for reducing LLM size and computational demands. This paper focuses on structural pruning, which enhances inference efficiency in a hardware-agnostic manner. Existing structural pruning methods for LLMs are typically classified into local and global techniques. Local pruning methods [ 17 , 26 ] , which prune layers individually, enable efficient compression of hundred-billion-scale LLMs on a single GPU via offload approaches. However, they overlook global dependencies in model topology and restrict the sparsity to be uniform across layers. Global pruning methods [ 23 , 18 , 1 ] alleviate local constraints, facilitating sparsity allocation and the potential for optimal pruning. However, many existing methods estimate the saliency of local substructures and prune them accordingly via global ranking, ignoring inter-structure dependencies and hindering end-to-end optimization. Such methods may also suffer from the inefficiency of backpropagation-based saliency estimation and overfitting when calibration data is limited. Therefore, a question arises: How to achieve efficient global structural pruning with end-to-end optimization? To address this challenge, we propose Týr-the-Pruner , an efficient search-based global pruning framework with end-to-end optimization. Our framework constructs a supernet by applying local pruning to each layer, producing pruned copies with different sparsity ratios. The objective is to identify an optimal subnet that satisfies the target overall sparsity ratio within the supernet by determining the optimal sparsity distribution across layers. We use evolutionary search [ 22 ] to solve this optimization problem. To construct reliable supernets and perform effective and efficient search, we make the following contributions: • To improve supernet construction , we propose an effective local pruning approach for attention heads and feed-forward networks (FFN), using Taylor expansion-based first- and second-order optimization information to identify redundant structures and adjust remaining weights. Pruning and weight adjustments are applied progressively and finely to preserve accuracy. Additionally, we introduce an expectation error accumulation approach to address the challenge of unclear error propagation caused by the multiple pruned copies within the supernet. This approach ensures balanced mutual awareness across sparse structures during supernet construction. • To enhance the efficacy and efficiency of subnet search , we employ a tailored distillation-inspired metric as the optimization objective to guide the search process, aiming to preserve the subnet’s generative capability. In general, Týr-the-Pruner is formed as an iterative prune-and-search framework that refines sparsity allocation for each layer with reduced search space and fast convergence. Each iteration prunes and constructs a supernet across a specific range of sparsity ratios, coupled with a sparsity-shift-driven evolutionary search, where random sparsity shifts between layers generate parent candidates, and the best-performing ones are filtered as offspring. The sparsity interval is refined after each iteration. By making these contributions, Týr-the-Pruner achieves end-to-end global pruning with strong efficacy and efficiency. Notably, the proposed framework only requires 4M tokens for calibration and search. Experimental results demonstrate that Týr-the-Pruner surpasses state-of-the-art pruning methods. For example, Týr-the-Pruner outperforms the SOTA method FLAP, achieving 3.45 lower perplexity in language comprehension and 10.26% higher average downstream accuracy when pruning 37.5% of the parameters of Llama-3.1-8B. Moreover, it maintains 97% performance with 50% pruning on Llama-3.1-70B, a sparsity ratio that is considered aggressive for existing methods. 2 Method Figure 1: An overview for Týr-the-Pruner . Large language models (a) will be effectively locally pruned across multiple sparsity ratios and constructed into a supernet (b). An iterative prune-and-search strategy will be used to select the optimal sparse structure for each layer while maintaining a target overall sparsity ratio: pruning and sparsity-shift-driven evolutionary search are implemented iteratively with a coarse-to-fine sparsity interval granularity (c). Ultimately, the post-pruned LLM with the optimal sparsity distribution (d) is obtained. This section presents Týr-the-Pruner, a novel structural pruning framework for large language models (cf. Section 2.1 for preliminaries), as illustrated in Figure 1 . This framework (1) constructs a supernet by applying local pruning across various sparsity ratios to each model layer, aiming to (2) search the optimal sparsity distribution under a target overall sparsity ratio. Specifically, we propose an effective local pruning approach (cf. Section 2.2 ) and an expectation error accumulation approach (cf. Section 2.3 ) to enhance supernet construction. An iterative prune-and-search strategy with coarse-to-fine sparsity granularity (cf. Section 2.4 ) ensures efficient search convergence. 2.1 Preliminaries Large language models typically use the Transformer decoder architecture [ 42 ] , as shown in Figure 1 (a). Each Transformer layer consists of two key components: the multi-head self-attention (MHA) and the feed-forward network (FFN), followed by a residual connection and layer normalization. Given the input \mathbf{X}_{\ell-1} to the \ell -th layer, the output \mathbf{X}_{\ell} can be expressed as: \displaystyle\mathbf{X} \displaystyle=\text{LayerNorm}\left(\mathbf{X}_{\ell-1}+\operatorname{MHA}(\mathbf{X}_{\ell-1})\right), (1) \displaystyle\mathbf{X}_{\ell} \displaystyle=\text{LayerNorm}\left(\mathbf{X}+\mathrm{FFN}(\mathbf{X})\right). The MHA mechanism captures dependencies across different positions in the input sequence with multiple attention heads, each with its query ( \mathbf{W}_{q} ), key ( \mathbf{W}_{k} ), value ( \mathbf{W}_{v} ), and out ( \mathbf{W}_{o} ) linear transformations. Modern LLMs typically employ a SwiGLU-based FFN [ 34 ] , consisting of gate ( \mathbf{W}_{gate} ), up ( \mathbf{W}_{up} ), and down ( \mathbf{W}_{down} ) linear transformations, with activation after the gate. This structure aids in extracting non-linear representations. Structural pruning for LLMs can be applied across four key dimensions: (1) attention heads, (2) FFN intermediate neurons, (3) embedding dimension size, and (4) model depth. It can be isotropic (uniform sparsity across layers) or non-isotropic (layer-specific sparsity). This paper focuses on pruning attention heads and FFN intermediate neurons with non-uniform sparsity: pruning functionally independent heads and neurons allows for controllable accuracy loss, while layer-specific sparsity further enhances pruning by tailoring