S-HPLB : Efficient LLM Attention Serving via Sparsity-Aware Head Parallelism Load Balance
Di Liu, Yifei Liu, Chen Chen ∗ , Zhibin Yu † , Xiaoyi Fan ‡ , Quan Chen, Minyi Guo · 2026
Read S-HPLB : Efficient LLM Attention Serving via Sparsity-Aware Head Parallelism Load Balance by Di Liu, Yifei Liu, Chen Chen ∗ , Zhibin Yu † , Xiaoyi Fan ‡ , Quan Chen, Minyi Guo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10353.txt.
Introduction
Di Liu and co-authors present S-HPLB, a serving strategy that balances attention work across GPUs by respecting how sparse each attention head can safely be. Heads differ in stable ways, so head-adaptive sparsity budgets save compute without hurting quality. Uneven budgets, though, leave some GPUs idle while others lag. S-HPLB plans head-parallel placement to shrink those bubbles. On long-context benchmarks it more than halves average attention latency, indeed improving it nearly threefold, with no loss in inference quality.
Catalog record
| CREATOR | Di Liu, Yifei Liu, Chen Chen ∗ , Zhibin Yu † , Xiaoyi Fan ‡ , Quan Chen, Minyi Guo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32120 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10353
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
S-HPLB : Efficient LLM Attention Serving via Sparsity-Aware Head Parallelism Load Balance Thanks: ∗ Chen Chen is the corresponding author. Di Liu, Yifei Liu, Chen Chen ∗ , Zhibin Yu † , Xiaoyi Fan ‡ , Quan Chen, Minyi Guo Abstract . With the increasing volumes of Large Language Models (LLMs) and the expanding context lengths, attention computation has become a key performance bottleneck in LLM serving. For fast attention computation, recent practices often parallelize the attention heads on multiple GPUs, and also widely adopt attention sparsification to reduce the computation amount—which selectively computes a subset of attention pairs under a preset sparsity budget. In this paper, we notice that attention heads of an LLM model often exhibit heterogeneous-yet-stable sparsity elasticities, which motivates us to enforce head-adaptive sparsity budgets to attain better efficiency while preserving high inference quality. Yet, from the system aspect, with heterogeneous sparsity levels, attention computation time on different heads would be inconsistent, yielding cross-GPU resource bubbles under head-parallel deployment. To further minimize such bubbles, we propose a novel attention deployment strategy called Sparsity-aware Head-Parallel Load Balance (S-HPLB). Experiments on long-context benchmark show that, S-HPLB can achieve a 2.88 × 2.88\times improvement in average attention computation latency without quality degradation. 1. Introduction Recent advances in large language models (LLMs) have enabled remarkable capabilities in natural language understanding and generation. Meanwhile, applications such as document understanding ( 24 ) , long-form question answering ( 14 ) , and code synthesis ( 11 ; 2 ) feed models with increasingly longer context inputs. As both model size and context length grow, the computational demand of LLM inference—especially the attention operation—escalates dramatically. For instance, when processing an input with a 128K context length using the Llama-3-8B model, attention operation accounts for over 70% of the total time-to-first-token (TTFT) ( 16 ) . Therefore, reducing the cost of attention operation is crucial for efficient LLM serving. For efficient LLM attention serving, from the system perspective, recent practices increasingly adopt distributed attention deployment ( 33 ; 23 ; 32 ; 13 ) . On the one hand, to avoid cross-module resource interference, the attention module in each Transformer layer can be decoupled from the other modules (e.g., FFN), which is called attention-FFN disaggregation (AFD) ( 33 ; 23 ; 22 ) ; this renders it possible to optimize the attention efficiency independently. On the other hand, to particularly speed up the attention module with expanded resources, leading LLM serving systems ( 32 ; 13 ) often evenly distribute the attention heads of an attention module on different GPUs, a deployment mechanism we call Head-parallelism (or HP, which is a special form of the classical Tensor Parallelism). Meanwhile, from the algorithm perspective, sparse attention is a popular optimization technique in the literature ( 31 ; 28 ; 21 ; 10 ) . Specifically, the attention operation has inherent sparsity, meaning that each query vector essentially interacts with only a subset of the key vectors; sparse attention can thus alleviate the attention cost by bypassing the attention computation of insignificant query-key pairs. Moreover, in determining the number of tokens to select in attention computation, existing sparse-attention methods typically rely on a fixed token budget (i.e., adopting the top- k policy where k is the selected token number) ( 28 ; 10 ) , which is uniformly applied to all the attention heads. However, in attaining high attention efficiency while still preserving the accuracy quality, we find that enforcing a uniform sparsity budget across different heads is far from optimal. In fact, our testbed measurements ( Figure 4 ) show that the attention heads of a Transformer layer usually exhibit diverse sparsity levels (i.e., with diverse relationship shapes between the overall attention weight and the sparsity budget). In that case, enforcing a uniform token budget would lead to redundant computations on some (high-sparsity) heads, yet at the same time incur large attention loss on other (low-sparsity) heads. Recently, some works ( 29 ; 15 ) have proposed the top- p method to implicitly handle such sparsity heterogeneity, which adaptively determines the token budget for each head—such that the total attention weight of its computed query-key pairs reaches a preset threshold p . Nonetheless, such a top- p method requires costly analysis of the attention map, and is often inaccurate in determining the attention participants ( 15 ) . Besides, when attention heads have different sparsity budgets, the attention computation time on different GPUs would be inconsistent, suffering resource wastage due to the synchronization barriers. Motivated by the above discussions, in this paper we present Sparsity-aware Head Parallelism Load Balance ( S-HPLB ), a system-algorithm co-designed mechanism to improve the end-to-end attention serving efficiency while maintaining high accuracy. From the algorithm aspect, we first observe that, despite the cross-head sparsity heterogeneity, on each head the sparsity pattern is however stable across different use cases that have diverse input lengths and tasks; this makes it possible to conduct per-head sparsity modeling in an offline manner. Besides, given an expected sparsification ratio (i.e., the k budget) and the profiled per-head sparsity pattern, we propose to better off the low-sparsity heads with cross-head budget shifting, so as to enhance the accuracy performance without inflating the overall computation amount. Moreover, from the system aspect, with the algorithm-side sparsity budgets, we model the head deployment problem as a classical multiway partitioning problem, and propose a greedy heuristic to minimize the resource wastage caused by cross-GPU load inconsistency. We evaluate both the accuracy and efficiency of S-HPLB on three leading open source LLMs, running the widely adopted RULER benchmark ( 9 ) on a server equipped with eight A100 GPUs. Experimental results show that, compared to state-of-the-art sparse-attention algorithms, S-HPLB improves the model accuracy and attention latency by up to 5.34% and 2.88 \times , respectively. Further deep-dive studies demonstrate that, S-HPLB operates consistently on the Pareto frontier of the latency-accuracy skyline. Meanwhile, the head parallel load balancer itself reduces latency by 1.26 \times . 2. Background and Motivation Figure 1 . Illustration of the attention head parallelism. 2.1. Attention in LLM Inference A critical operation of LLM inference is self-attention. Given an input of n tokens, self-attention first transforms them into three matrices: query Q , key K , and value V . Using the query and key, the attention weight A is derived and the output O is computed as the weighted sum over the value V : (1) A=\text{Softmax}(\frac{Q\times K^{T}}{\sqrt{d}}),~O=A\times V In mainstream attention architectures such as Multi-Head Attention (MHA) and Group-Query Attention (GQA) ( 1 ) , the attention operation is typically composed of multiple modules, each known as an attention head (e.g., 32 heads for the Llama-3.1-8B model). Each head independently performs self-attention as in Eq. ( 1 ) and captures diverse features from different subspaces. The results from all heads are then aggregated to yield the output. Moreover, recent LLMs have demonstrated exceptional performance in processing long contexts , supporting a wide range of applications like document understanding and repository-level code debugging ( 2 ; 11 ) . While crucial for capturing relationships within sequences, efficiently serving attention in long-context scenarios remains challenging. In the prefill phase, processing all input tokens at once incurs quadratic computational