GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models

Ziyang Wang, Jiangfeng Xiao, Chuan Xiao, Ruoxiang Li, Rui Mao, Jianbin Qin · 2026

Read GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models by Ziyang Wang, Jiangfeng Xiao, Chuan Xiao, Ruoxiang Li, Rui Mao, Jianbin Qin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19398.txt.

Introduction

Wang and colleagues present GRASPrune, a pruning tool that shrinks large language models after pretraining under a global budget, cutting their heavy serving costs. GRASPrune prunes feed-forward channels and KV head groups. It learns light gate scores with a projected straight-through estimator. That estimator enforces a hard mask meeting the budget at every step, while backbone weights stay frozen. After the mask is fixed, scaling factors on kept units are tuned. They are folded into the pruned weights, yielding a smaller checkpoint with no extra weights.

Catalog record

CREATORZiyang Wang, Jiangfeng Xiao, Chuan Xiao, Ruoxiang Li, Rui Mao, Jianbin Qin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56433
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19398

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models Ziyang Wang, Jiangfeng Xiao, Chuan Xiao, Ruoxiang Li, Rui Mao, Jianbin Qin Abstract Large language models (LLMs) are expensive to serve because model parameters, attention computation, and KV caches impose substantial memory and latency costs. We present GRASPrune, a structured pruning framework applied after pretraining that jointly prunes FFN channels and KV head groups under a single global budget. Instead of learning importance scores without constraints and applying the budget only after training, GRASPrune learns lightweight gate scores with a projected straight-through estimator that enforces a hard mask satisfying the budget at every step while keeping the backbone weights frozen. After the mask is fixed, we calibrate scaling factors on the retained units to mitigate scale mismatch caused by pruning, and fold these factors into the pruned weights to obtain a smaller dense checkpoint with no extra parameters at inference. On LLaMA-2-7B, GRASPrune removes 50% of parameters and achieves 12.18 perplexity on WikiText-2 while maintaining competitive average zero-shot accuracy on five benchmarks, using four epochs on 512 unlabeled calibration sequences on a single NVIDIA A100 80GB GPU without any full model fine-tuning. 1 1 1 https://github.com/ZiY-Wang/GRASPrune † † footnotetext: Corresponding author: [email protected] 1 Introduction Figure 1: WikiText-2 perplexity of LLaMA-2-7B under different parameter retention ratios. Large language models (LLMs) achieve strong performance in reasoning and generation ( 39 ; 5 ; 31 ) , but serving them remains expensive. In practice, both large parameter counts and the KV cache that grows with sequence length contribute substantially to memory use and latency ( 20 ) . This motivates structural compression methods that reduce checkpoint size and cache-related memory. Beyond compression, runtime system techniques can also reduce serving cost without modifying model parameters by alleviating bottlenecks in attention and KV caching. Examples include sparse attention kernels ( 19 ; 42 ) and methods that manage or compress the KV cache ( 49 ; 23 ; 43 ) . These techniques target runtime attention and memory traffic and are largely complementary to structural compression. In this work, we study structured pruning after pretraining. By removing channels or head groups, pruning produces a smaller dense checkpoint that can be deployed with standard serving stacks. As shown in Figure 1 , model quality degrades nonlinearly as the pruning budget becomes tighter, which suggests that uniform sparsity is insufficient and that the budget must be allocated in a principled way that accounts for unit cost. Under deployment constraints, structured pruning becomes a discrete selection problem over heterogeneous structure types. Existing pipelines often fall short in three ways. First, FFN channels and KV head groups are usually pruned with separate criteria, even though they compete for the same deployment budget and representational capacity ( 22 ; 15 ) . Second, many methods prescribe keep ratios for each layer or use depth dependent schedules ( 36 ; 26 ) , which hard code how the budget is distributed across layers and structures instead of learning a single global allocation. Third, because the final selection is discrete and unit costs differ, many pipelines estimate importance scores first and impose the budget only afterward, often followed by reconstruction or additional fine-tuning ( 1 ; 29 ) . This weakens the connection between the final sparse architecture and the training objective, and it increases engineering overhead. Although a few recent works take a more global view ( 21 ; 12 ) , they often ignore cost differences between FFN and KV structures or rely on expensive search and reconstruction. Our key observation is that the main issue is not the lack of another saliency metric, but the mismatch between how scores are learned and how the final mask is selected. Existing score first and select later pipelines learn importance under an unconstrained surrogate and impose the deployment budget only at the end. We instead enforce budget feasibility inside the optimization loop, so gate scores are learned under the same constraint that defines the final deployable mask. Based on this view, we propose GRASPrune, a structured pruning framework with a single resource budget. The backbone weights remain frozen, and we optimize only a small set of scalar gates on a small unlabeled calibration set. GRASPrune jointly selects FFN channels and KV head groups under one global budget and compiles the resulting mask into a smaller dense checkpoint that can be served with the standard stack. To match prior pruning baselines, we use parameter count as the budget proxy, and we also report KV cache and throughput results to connect pruning to serving performance. On LLaMA-2-7B, our pruned model reaches 12.18 perplexity on WikiText-2 at a 50% parameter ratio and 16.65 at 40%. Learning this allocation is also efficient: four epochs of gate learning plus a brief calibration stage take roughly 6 minutes on a single A100 80GB GPU. Our contributions are as follows: • A single global pruning formulation over FFN channels and KV head groups. We formulate structured pruning as one global selection problem over FFN channels and KV head groups with different unit costs, and optimize a single mask under one budget constraint. • Projected STE training with budget feasibility enforced at every step. We argue that scores should be learned while the budget is active during training. If learned scores are post-hoc re-ranked by a cost-normalized criterion, the resulting allocation can shift toward cheaper units and degrade performance in our setting. Motivated by this, we use a projected STE loop that applies a hard mask satisfying the budget at each step while optimizing only scalar gates. • A smaller dense checkpoint through calibration on the retained units. After selecting the retained units, we calibrate per unit scales only on this support and fold them into the sliced weights, producing a smaller dense checkpoint without adding extra parameters during inference. Figure 2: Overall pipeline of GRASPrune. 2 Related Work 2.1 Pruning for LLMs We focus on structured pruning, which removes channels, heads, or layers. Existing LLM-oriented methods can be roughly grouped into three families by how they allocate sparsity. The first family uses fixed per-layer sparsity schedules with local saliency inside each layer: given a target global sparsity, they pre-assign layer-wise keep ratios and then prune the least important structures according to magnitude- or activation-based scores, sometimes followed by light calibration or fine-tuning ( 34 ; 45 ; 17 ) . LLM-Pruner is a representative example that ranks structures with Taylor-style gradients under an OWL-style per-layer schedule ( 26 ) . A second family keeps the layer-wise view but lets data or error decide the ratios, deriving layer-wise sparsity from activations or reconstruction loss instead of prescribing it. SlimGPT is a prominent instance of this design: it extends Optimal Brain Surgeon to structured pruning, using batched greedy updates together with an incremental pruning schedule that is explicitly tuned to layer-wise error, so that more sensitive layers are pruned less aggressively ( 24 ) . A third family adopts a genuinely global view, coordinating sparsity across layers and structure types under a single constraint. ZipLM aggregates multi-granularity saliency into a global ordering and selects structures directly under a target sparsity ( 21 ) . DISP-LLM further relaxes the coupling along

Read the complete text