FASP: Fast and Accurate Structured Pruning of Large Language Models

Hanyu Hu *, Pengxiang Zhao, Ping Li, Yi Zheng, Zhefeng Wang, Xiaoming Yuan · 2025

Read FASP: Fast and Accurate Structured Pruning of Large Language Models by Hanyu Hu *, Pengxiang Zhao, Ping Li, Yi Zheng, Zhefeng Wang, Xiaoming Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09412.txt.

Introduction

Six researchers present a fast pruning method that shrinks large language models in a paper from 2025. Structured pruning removes whole rows and columns, cutting memory and compute for small devices. Their method, FASP, links neighboring layers so cuts in one layer remove matching parts in the next without extra loss. A cheap scoring rule, adapted from an earlier method, picks what to cut.

After pruning, a restoration step adjusts the surviving weights to recover quality. On two model families, FASP beat rival methods on language scores and task accuracy. It pruned one small model in seventeen seconds and a thirty-billion-parameter model in fifteen minutes on one graphics card. The authors stress practical speed.

Catalog record

CREATORHanyu Hu *, Pengxiang Zhao, Ping Li, Yi Zheng, Zhefeng Wang, Xiaoming Yuan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-09412

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FASP: Fast and Accurate Structured Pruning of Large Language Models Hanyu Hu *, Pengxiang Zhao, Ping Li, Yi Zheng, Zhefeng Wang, Xiaoming Yuan Abstract The rapid increase in the size of large language models (LLMs) has significantly escalated their computational and memory demands, posing challenges for efficient deployment, especially on resource-constrained devices. Structured pruning has emerged as an effective model compression method that can reduce these demands while preserving performance. In this paper, we introduce FASP ( F ast and A ccurate S tructured P runing), a novel structured pruning framework for LLMs that emphasizes both speed and accuracy. FASP employs a distinctive pruning structure that interlinks sequential layers, allowing for the removal of columns in one layer while simultaneously eliminating corresponding rows in the preceding layer without incurring additional performance loss. The pruning metric, inspired by Wanda, is computationally efficient and effectively selects components to prune. Additionally, we propose a restoration mechanism that enhances model fidelity by adjusting the remaining weights post-pruning. We evaluate FASP on the OPT and LLaMA model families, demonstrating superior performance in terms of perplexity and accuracy on downstream tasks compared to state-of-the-art methods. Our approach achieves significant speed-ups, pruning models such as OPT-125M in 17 seconds and LLaMA-30B in 15 minutes on a single NVIDIA RTX 4090 GPU, making it a highly practical solution for optimizing LLMs. 1 Introduction Large language models (LLMs) have profoundly transformed various aspects of daily life in recent times, showcasing remarkable capabilities across diverse applications, including machine translation, conversational agents, text generation, as well as image and video synthesis ( OpenAI, 2023 ; Meta AI, 2023 ; Gemini Team et al., 2023 ; Arefeen et al., 2024 ; Li et al., 2024 ) . However, the increasing size of these models imposes substantial demands on computational and GPU memory resources, creating significant barriers to the deployment of advanced LLMs. This challenge becomes even more pronounced when considering the deployment of LLM technologies on mobile devices, such as smartphones, particularly in light of the rising trend driven by the release of advanced AI capabilities like Apple Intelligence ( Apple Inc, 2024 ) . In light of these challenges, the compression of LLMs has become a critical requirement to mitigate the resource demands associated with their deployment, thereby enhancing their accessibility and operational efficiency across diverse platforms. Pruning techniques represent one of the most prevalent methods for compressing LLMs ( Frantar & Alistarh, 2023 ; Sun et al., 2023 ; Ma et al., 2023 ; Shen et al., 2024 ; Fang et al., 2024 ) , effectively reducing model size and computational demands while maintaining performance integrity. Among the various pruning strategies, structured pruning stands out for its systematic approach of removing entire components, such as neurons or channels, which directly enhances computational efficiency and is compatible across diverse hardware platforms. In contrast, unstructured pruning targets individual weights, necessitating support for sparse data structures and corresponding computational methods. Similarly, semi-structured pruning often relies on specific hardware architectures, such as the 2:4 semi-structure utilized in NVIDIA’s Ampere architecture ( Mishra et al., 2021 ) . Additionally, the inference acceleration achieved through semi-structured pruning is less effective than that obtained with structured pruning at the same level of sparsity ( Ashkboos et al., 2024 ) . Therefore, this work will concentrate on structured pruning. Several existing structured pruning methods have shown varying effectiveness in optimizing LLMs. For example, LLM-Pruner ( Ma et al., 2023 ) employs structured pruning to selectively remove non-critical coupled structures based on gradient information, and it requires a fine-tuning step to recover the performance of the pruned models, which may take several hours. SliceGPT ( Ashkboos et al., 2024 ) replaces each weight matrix with a smaller dense matrix, thereby reducing the embedding dimension of the network. However, this approach requires the application of principal component analysis for each linear layer, which can be time-consuming. Additionally, it introduces two auxiliary transformation matrices in each decoder block, which mitigates the overall effectiveness of the pruning. FLAP ( An et al., 2024 ) formulates structured importance metrics and adaptively searches for a globally compressed model. Although it is relatively efficient, it often compromises performance. Shen et al. (2024) introduces a neural architecture search-based approach (denoted as NASLLM hereafter) to identify subnets of the original LLMs and implements a reformation algorithm to rectify the inherited weights. However, due to its searching nature, this method can be time-consuming. Figure 1: Illustration of the proposed pruning structure on the OPT model. In this approach, columns of W_{\text{$fc_{2}$}} are removed along with the corresponding rows of W_{\text{$fc_{1}$}} without impacting performance, thanks to the inherited position mapping in matrix multiplication. The same principle applies to columns of W_{\text{out}} and rows of W_{V} , as well as the rows of W_{\text{Q}} and W_{K} . Given the limitations of current structured pruning methods, particularly concerning time efficiency and the performance of pruned models, we propose a novel approach, FASP, that emphasizes both speed and accuracy. First, we introduce an innovative pruning structure that connects two sequential linear layers. This design enables the pruning of a subset of columns in the later layer, allowing for the corresponding rows in the preceding layer to be removed without adversely impacting performance. Inspired by Wanda ( Sun et al., 2023 ) , we develop a pruning metric for selecting columns to prune, which is both simple and efficient. Finally, we implement an effective restoration mechanism to recover the performance of the pruned model by leveraging inherited weights, thereby enhancing overall model fidelity. We conduct comprehensive experiments on the OPT ( Zhang et al., 2022 ) and LLaMA ( Touvron et al., 2023 ) model families, comparing the proposed FASP against state-of-the-art techniques. Our results demonstrate that FASP achieves superior performance in terms of perplexity and accuracy on downstream tasks across these models. Specifically, we evaluate the pruning time for models of varying sizes, finding that FASP requires approximately 17 seconds to prune the OPT-125M model and about 20 minutes for the LLaMA-30B model on a single NVIDIA RTX 4090 GPU, highlighting substantial improvements in pruning speed. These findings emphasize the advantages of our method in practical applications, showcasing its potential to optimize LLMs efficiently while maintaining high performance. 2 Background and Related Work Pruning techniques in neural networks can be broadly categorized into unstructured and structured pruning. Unstructured and semi-structured pruning remove individual weights or partial substructures of a model to reduce its size. These methods, such as SparseGPT ( Frantar & Alistarh, 2023 ) , Wanda ( Sun et al., 2023 ) , FPSTAPruner ( Zhao et al., 2024 ) and MaksLLM ( Fang et al., 2024 ) , often achieve high sparsity levels without severely degrading model performance. However, they come with practical limitations. Unstructured pruning may provide limited computational speedup and memory savings due to its irregular sparsity structure and the necessity of storing indices. Additionally, the inference of semi-structured sparsified LLMs typically requires specialized hardware, such as NVIDIA’s Ampere architecture GPUs ( Mishra et al., 2021 ) , which restricts their applicability in real-world scenarios. In contrast, structured pruning entirely removes substructures from rows or columns in the weight matrices to attention heads, layers, or blocks. This results

Read the complete text