SHRP: Specialized Head Routing and Pruning for Efficient Encoder Compression

Zeli Su *, Ziyin Zhang *, Wenzheng Zhang *, Zhou Liu, Guixian Xu †, Wentao Zhang † · 2025

Read SHRP: Specialized Head Routing and Pruning for Efficient Encoder Compression by Zeli Su *, Ziyin Zhang *, Wenzheng Zhang *, Zhou Liu, Guixian Xu †, Wentao Zhang † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20635.txt.

Introduction

Zeli Su and co-authors prune the attention heads of Transformer encoders for production search and ranking systems. Encoders deployed at web scale suffer latency and memory costs, much of it from attention redundancy. Their approach, SHRP, treats every attention head as an independent expert module followed by a shared refining network.

A single usage-driven mechanism routes tokens to heads during training and then prunes deterministically for deployment. On the GLUE benchmark with a BERT-base encoder, the compressed model keeps most of the original accuracy with nearly half the parameters gone. In an extreme setting with eleven of twelve layers pruned, accuracy remains respectable while throughput multiplies and computation falls to a small fraction. Latency-sensitive services are the target.

Catalog record

CREATORZeli Su *, Ziyin Zhang *, Wenzheng Zhang *, Zhou Liu, Guixian Xu †, Wentao Zhang †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20635

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SHRP: Specialized Head Routing and Pruning for Efficient Encoder Compression DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Thanks: * Equal contribution. Thanks: † Corresponding authors. CCS: Computing methodologies Artificial intelligence Natural language processing CCS: Computing methodologies Machine learning Supervised learning CCS: Computer systems organization Real-time computing Zeli Su *, Ziyin Zhang *, Wenzheng Zhang *, Zhou Liu, Guixian Xu †, Wentao Zhang † Abstract . Transformer encoders are widely deployed in large-scale web services for natural language understanding tasks such as text classification, semantic retrieval, and content ranking. However, their high inference latency and memory consumption pose significant challenges for real-time serving and scalability. These limitations stem largely from architectural redundancy, particularly in the attention module. The inherent parameter redundancy of the attention mechanism, coupled with the fact that its attention heads operate with a degree of independence, makes it particularly amenable to structured model compression. In this paper, we propose SHRP (Specialized Head Routing and Pruning) , a novel structured pruning framework that automatically identifies and removes redundant attention heads while preserving most of the model’s accuracy and compatibility. SHRP introduces Expert Attention , a modular design that treats each attention head as an independent expert, followed by a lightweight shared expander feed-forward network that refines their outputs. The framework employs a unified Top-1 usage-driven mechanism to jointly perform dynamic routing during training and deterministic pruning at deployment. Experimental results on the GLUE benchmark using a BERT-base encoder show that SHRP achieves 93% of the original model accuracy while reducing parameters by 48% . Under an extreme compression scenario where 11/12 of the layers are pruned , the model still maintains 84% accuracy and delivers a 4.2 × \times throughput gain while reducing computation to as low as 11.5% of the original FLOPs, demonstrating its practical utility for large-scale and latency-sensitive web deployments. 1. Introduction Transformer-based architectures—spanning both encoder-only and decoder-only designs—form the backbone of modern NLP. Encoder models such as BERT ( 3 ) and XLM-R ( 1 ) are widely used in classification, retrieval, and tagging pipelines ( 21 ; 20 ) , while decoder-only models such as LLaMA ( 17 ) , DeepSeek ( 2 ) , and Qwen ( 23 ) have driven rapid progress in generative applications. With their ability to capture long-range dependencies and scale to billions of parameters, Transformers have become indispensable across a wide range of tasks. However, despite their representational power, encoder deployments remain inefficient at production scale. Inference over millions of user queries and documents—common in large-scale web services for natural language understanding tasks such as text classification, semantic retrieval, and content ranking—pushes latency, memory, and energy consumption to their limits. Even on modern GPUs, encoder models are substantially overparameterized: prior work has shown that many attention heads contribute little to final predictions ( 14 ; 19 ) , and that a large fraction of model parameters and FLOPs can be removed without degrading performance ( 7 ; 15 ) . This indicates a fundamental gap between model capacity and actual usage—a form of architectural redundancy that motivates compression. Redundancy arises at multiple levels: first, at the head level , where many attention heads are inactive or underutilized; second, at the feed-forward level , where the FFN sublayer accounts for the majority of both parameter count and computation cost. Yet most existing methods fail to fully exploit this redundancy. Coarse-grained approaches , such as layer dropping or head pruning ( 14 ; 19 ) , remove large components, but tend to suffer from sharp accuracy loss under aggressive pruning. Fine-grained methods , including movement pruning and structured sparsity ( 15 ; 25 ) , achieve higher compression but often rely on heuristic scoring, custom masking, or hardware-aware patterns that limit real-world deployment. Critically, most attention-head pruning methods operate only within the attention sublayer. The FFNs—despite dominating overall computation—are typically left untouched. To preserve dimensionality, many approaches apply zero-padding or weight replication, which preserves FLOPs and fails to reduce runtime. These limitations motivate us to rethink structured compression from first principles. This paper proposes SHRP (Specialized Head Routing and Pruning) , a new pruning framework that compresses encoder models by modularizing attention heads as independent experts and coupling them with a shared lightweight Expander-FFN. Rather than relying on handcrafted heuristics, SHRP draws inspiration from Mixture-of-Experts (MoE) models ( 16 ; 9 ) , but repurposes the mechanism for compression instead of capacity expansion. By introducing lightweight routing during training, SHRP enables interpretable head usage patterns that guide pruning—without introducing any dynamic behavior at inference. The overview of the SHRP framework shown in Figure 1 . The process begins with Attention Transformation , where standard multi-head attention is replaced by Expert Attention —a modular structure where each head is routed independently through a shared lightweight Expander FFN . The framework then proceeds through four phases: layer-wise conversion with balanced routing, expert specialization, head-usage analysis, and final Top-1 pruning. Key challenges. Recasting heads as experts raises several new challenges: (i) how to maintain expert diversity during early training to avoid collapse; (ii) how to induce interpretable and stable head usage for pruning; (iii) how to integrate FFN compression without disrupting residual connections; and (iv) how to eliminate the routing overhead, which can account for up to 50% of runtime once the model is slimmed down. Our solution. SHRP introduces three innovations. First, we replace standard multi-head attention with Expert Attention , where each head is modularized and routed independently through a shared Expander FFN that performs a direct d_{\text{head}}\to d projection. This allows both attention and FFN sublayers to be pruned jointly. Second, we adopt a progressive two-stage training pipeline : a load-balancing objective encourages uniform usage in early layers, followed by a specialization phase that yields clear pruning signals. Third, we employ Top-1 routing and pruning , where usage statistics are tracked during training and underused experts are pruned post hoc. The router is then removed, yielding a static encoder with no dynamic computation. While the SHRP framework can in principle generalize to Transformer variants, we focus on encoder-only architectures due to their practical dominance in deployment and their suitability for structured pruning (detail discussing in Appendix A ). Contributions. In summary, this paper makes the following contributions: • We introduce SHRP ( Specialized Head Routing and Pruning ), a unified pruning framework that compresses encoder models by treating attention heads as independent experts routed through a shared Expander-FFN, enabling joint pruning of attention and FFN components. • We propose a two-stage progressive training pipeline that induces interpretable head specialization, allowing pruning without auxiliary scoring functions or architectural constraints. • We demonstrate that Top-1 routing and pruning achieves the best trade-off between accuracy and compression, and removes all routing overhead at inference time, producing compact and deterministic encoders. On the GLUE benchmark, SHRP achieves up to 88.5% parameter reduction , 4.2 \times throughput improvement , and reduces compute to as low as 11.5% of original FLOPs , while retaining over 84% of baseline accuracy even under aggressive compression. These results demonstrate that modular design, combined with MoE-inspired training and principled routing, enables pruning to move

Read the complete text