High-Layer Attention Pruning with Rescaling
Songtao Liu, Peng Liu · 2025
Read High-Layer Attention Pruning with Rescaling by Songtao Liu, Peng Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01900.txt.
Introduction
Songtao Liu and Peng Liu present a pruning method for large language models in a 2025 paper. Pruning cuts parts of a model to make it faster and cheaper to run. Older methods remove attention heads, the parts that weigh context, by one fixed rule across all layers.
Their method, called HARP, focuses its cuts on the higher layers of the model. Removing heads can shrink the size of the signals that pass through, so the method adds a rescaling step to restore the right scale. Tests on four well-known models, across 27 data sets, beat older pruning methods. The gain was largest on tasks where the model must generate text, rather than just pick an answer.
Catalog record
| CREATOR | Songtao Liu, Peng Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01900
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
High-Layer Attention Pruning with Rescaling Songtao Liu, Peng Liu Abstract Pruning is a highly effective approach for compressing large language models (LLMs), significantly reducing inference latency. However, conventional training-free structured pruning methods often employ a heuristic metric that indiscriminately removes some attention heads across all pruning layers, without considering their positions within the network architecture. In this work, we propose a novel pruning algorithm that strategically prunes attention heads in the model’s higher layers. Since the removal of attention heads can alter the magnitude of token representations, we introduce an adaptive rescaling parameter that calibrates the representation scale post-pruning to counteract this effect. We conduct comprehensive experiments on a wide range of LLMs, including LLaMA3.1-8B, Mistral-7B-v0.3, Qwen2-7B, and Gemma2-9B. Our evaluation includes both generation and discriminative tasks across 27 datasets. The results consistently demonstrate that our method outperforms existing structured pruning methods. This improvement is particularly notable in generation tasks, where our approach significantly outperforms existing baselines. 1 Introduction Large language models (LLMs) [ 78 , 67 , 42 , 88 , 26 ] , pre-trained on extensive text data, have achieved surprising performance in downstream tasks such as information retrieval [ 4 ] , code generation [ 28 ] , and mathematical reasoning [ 80 , 90 , 37 ] . These LLMs, however, contain a huge number of parameters, resulting in substantially slower inference speed compared to their smaller counterparts. Many efforts have leveraged pruning algorithms [ 22 , 40 , 83 , 5 , 87 , 39 , 97 , 21 , 92 , 93 , 100 ] to reduce redundant parameters and lower inference latency without significantly compromising the task performance. Among these, training-free structured pruning methods [ 61 , 2 , 74 ] estimate the importance of parameters using predefined pruning metrics, allowing them to remove less important components without retraining. While most pruned parameters typically come from feedforward networks, these methods also remove some low-importance attention heads. This is especially beneficial for long-context generation tasks, where the attention mechanism becomes the main computational bottleneck due to its quadratic time complexity with respect to sequence length. However, current pruning algorithms generally prune attention heads across all targeted layers, either by uniformly removing the same heads from each layer [ 61 ] or by adaptively pruning more heads in certain layers and fewer in others [ 2 ] . These approaches do not consider which attention head positions are actually more redundant and thus better suited for pruning. We argue that the pruning metrics used in these methods are not accurate enough to reliably identify the most redundant attention heads. In this work, we aim to address the following question: Can we develop a structured pruning algorithm that effectively removes attention heads? We approach this problem by analyzing the relative importance of different layers in LLMs. Recent studies [ 62 , 101 , 27 , 31 , 72 ] demonstrate that pruning blocks or entire layers in the high layers of LLMs often yields better performance than pruning those in the low layers. Building on these findings, we propose a training-free structured pruning algorithm, called HARP ( H igh-layer A ttention R escaled P runing), that removes the attention heads, specifically the query and key parameters, in the higher layers. After pruning, the attention mechanism in these layers can be skipped entirely. However, bypassing the attention computation may alter the output magnitude, as the value representations are no longer combined through weighted averaging based on attention scores over the full sequence. To mitigate this effect, we introduce a rescaling parameter in the attention residual block to calibrate the token representation magnitude after pruning. Figure 1: Left : Performance comparison of our method against state-of-the-art baselines on LLaMA3.1-8B, demonstrating superior generation capabilities. Right : Ablation study on the rescaling parameter across benchmarks, showing that applying rescaling consistently leads to significant performance improvements over the variant without rescaling. We evaluate the effectiveness of our approach by applying it to LLaMA3.1-8B, Mistral-7B-v0.3, Qwen2-7B, and Gemma2-9B. Extensive experiments on 27 representative downstream tasks show that our method outperforms popular training-free structured pruning algorithms, including Magnitude Pruning [ 30 ] , LLM-Pruner [ 61 ] , FLAP [ 2 ] , and Wanda [ 74 ] . As illustrated on the left side of Figure 1 , our method, HARP, significantly outperforms these baselines on generation tasks under the same attention head pruning ratio. More notably, the results on the right side of Figure 1 show that applying rescaling significantly improves performance compared to the unscaled variant, demonstrating the importance of rescaling for properly adjusting the representation magnitude. 2 High-Layer Attention Pruning In this section, we first demonstrate the importance of pruning attention heads in LLMs as part of structured pruning algorithms to reduce latency in downstream long-context tasks (Section 2.1 ). Next, we provide a theoretical and empirical explanation for why attention heads in higher layers are less important (Section 2.2 ). Finally, we present the details of our pruning algorithm (Section 2.3 ). 2.1 Pruning Attention Heads for Long-Context Modeling Most existing structured pruning algorithms for LLMs primarily focus on pruning parameters in the Feed-Forward Network (FFN), as they contain the majority of the model’s parameters. By pruning the FFN, we can significantly reduce the latency. However, pruning attention heads is also crucial. In long-context tasks such as retrieval-augmented generation (RAG), long-document summarization, and many-shot in-context learning (ICL), the attention mechanism often becomes the primary computational bottleneck due to its complexity \mathcal{O}\left(N^{2}\right) with respect to the sequence length N . We provide a detailed analysis of the time complexity associated with each component of LLMs. Proposition 1 . Let N be the sequence length, d the hidden size, and d_{\mathrm{ff}} the intermediate dimension in the FFN. In a standard transformer block, the linear projections in the attention module, including query, key, value, and output projections, have a time complexity of \mathcal{O}\left(Nd^{2}\right) . The self-attention mechanism, which includes computing attention scores and aggregating values, has a time complexity of \mathcal{O}\left(N^{2}d\right) . The FFN, consisting of two linear transformations with an intermediate dimension d_{\mathrm{ff}} , has a time complexity of \mathcal{O}\left(Ndd_{\mathrm{ff}}\right) . When N<d_{\mathrm{ff}} , the FFN’s \mathcal{O}\left(Ndd_{\mathrm{ff}}\right) complexity becomes the main time bottleneck. Conversely, when N>d_{\mathrm{ff}} , the \mathcal{O}\left(Nd^{2}\right) complexity of the self-attention module is dominant. In long-context tasks, prompt lengths can reach up to 2 million tokens [ 44 ] , making the operations in the self-attention module extremely time-expensive. By pruning attention heads [ 8 ] , we can achieve significant improvements in computational efficiency for long-context tasks while only pruning a small fraction of the model’s parameters. 2.2 High-Layer Attention Heads Are Less Important While many structured pruning algorithms focus on pruning attention heads, they often overlook the attention head position within the network. In this section, we show that attention heads in higher layers are typically less important than those in lower layers. Theoretical Analysis. We begin with the following theoretical analysis. Definition 1 . Define the average pairwise cosine similarity among tokens in a matrix \mathbf{H}\in\mathbb{R}^{N\times d} as \operatorname{Sim}(\mathbf{H}):=\frac{2}{N(N-1)}\sum_{1\leq i<j\leq N}\frac{\mathbf{h}_{i}^{\top}\mathbf{h}_{j}}{\|\mathbf{h}_{i}\|\|\mathbf{h}_{j}\|}. (1) This measures the global similarity of token representations. Theorem 1 . As \operatorname{Sim}(\mathbf{H}^{(\ell)})\to 1 for a high-layer \ell in a Transformer, the attention