Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval

Wenhao Li, Yuxin Zhang, Gen Luo, Haiyuan Wan, Ziyang Gong, Fei Chao, Rongrong Ji · 2025

Read Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval by Wenhao Li, Yuxin Zhang, Gen Luo, Haiyuan Wan, Ziyang Gong, Fei Chao, Rongrong Ji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19740.txt.

Introduction

Wenhao Li and six co-authors trim the memory of language models in twenty twenty-five. The key-value cache is a heavy burden during generation. Selecting only the critical cache entries at decoding time keeps quality while cutting cost. Existing selection uses random linear hashing to find important tokens.

That hashing is inefficient, they argue, because queries and keys sit in two narrow cones. Their method, Spotlight Attention, uses nonlinear hashing instead. It reshapes the embedding distribution for better coding efficiency and robustness. A light training framework uses a ranking-based loss from the Bradley-Terry model. It trains on a modest GPU in eight hours. Experiments show much better retrieval precision and shorter times. Readers get faster generation with a smarter cache.

Catalog record

CREATORWenhao Li, Yuxin Zhang, Gen Luo, Haiyuan Wan, Ziyang Gong, Fei Chao, Rongrong Ji
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-19740

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval Wenhao Li, Yuxin Zhang, Gen Luo, Haiyuan Wan, Ziyang Gong, Fei Chao, Rongrong Ji Abstract Reducing the key-value (KV) cache burden in Large Language Models (LLMs) significantly accelerates inference. Dynamically selecting critical KV caches during decoding helps maintain performance. Existing methods use random linear hashing to identify important tokens, but this approach is inefficient due to the orthogonal distribution of queries and keys within two narrow cones in LLMs. We introduce Spotlight Attention, a novel method that employs non-linear hashing functions to optimize the embedding distribution of queries and keys, enhancing coding efficiency and robustness. We also developed a lightweight, stable training framework using a Bradley-Terry ranking-based loss, enabling optimization of the non-linear hashing module on GPUs with 16GB memory in 8 hours. Experimental results show that Spotlight Attention drastically improves retrieval precision while shortening the length of the hash code at least 5 × \times compared to traditional linear hashing. Finally, we exploit the computational advantages of bitwise operations by implementing specialized CUDA kernels, achieving hashing retrieval for 512K tokens in under 100 μ \mu s on a single A100 GPU, with end-to-end throughput up to 3 × \times higher than vanilla decoding. All the training and evaluation stuff can be found at Anonymous/Spotlight . 1 Introduction Large Language Models (LLMs) are propelling groundbreaking advancements in various natural language tasks, significantly enhancing applications such as content creation and chat assistance. Generally, the inference process of LLMs can be divided into (1) the pre-filling phase calculates the key-value (KV) cache for input tokens in the prompt prior to autoregressive generation, and (2) the decoding phase auto-regressively generates tokens, producing one token per forward pass based on the KV cache. Among them, the decoding phase serves as the primary inference bottleneck due to the frequent exchanges between on-board and on-chip memory for model parameters and KV cache, which limits GPU scalability ( agrawal2023sarathiefficientllminference ) more so than the pre-filling phase that processes input prompts in parallel. For example, deploying LLaMA2-7B ( touvron2023llama2openfoundation ) on an A100 GPU for a single request achieves nearly 100% GPU utilization during the pre-filling phase but drops to below 10% on average during decoding, which largely restrains the inference efficiency of LLMs. To alleviate this inference bottleneck, extensive research has focused on heuristically eliminating the KV cache burden based on attention scores ( zhang2023ho ; xiao2024efficient ; oren2024transformersmultistaternns ) . While effective for short sequences, such irreversible removal of KV cache can significantly degrade performance on long-sequence tasks, especially in Needle-in-a-Haystack scenarios ( needle ) . To explain, tokens initially considered unimportant and removed might later attain higher attention scores during the prolonged decoding phase, which is crucial for output quality ( tang2024quest ; li2025surveylargelanguagemodel ) . To overcome this limitation, recent works have turned to retaining all KV cache tokens and dynamically selecting important tokens for computation during decoding ( tang2024quest ; chen2024magicpig ) , which is the focus of this paper. Figure 1 : Overview. ( Left ) Architecture. Comparison of our Spotlight Attention versus normal attention. Spotlight Attention adds an additional hash code-based retrieval mechanism for each layer. ( Middle ) Performance. Spotlight attention achieves the most accurate retrieval and generates the closest response compared to the original model on QA datasets. ( Right ) Visualization. For arbitrarily complex attention patterns, our method estimates the top-k sequences well, with an average correctness rate of more than half for different models. Despite the convincing performance of such on-the-fly KV cache selection, how to effectively pick up those important tokens remains challenging. As a pioneering effort, Quest ( tang2024quest ) selects tokens by matching queries and keys in a block-wise manner. While effective, such coarse-grained selection hardly guarantees precise localization of important tokens. MagicPIG ( chen2024magicpig ) advances Quest by implementing token-level cache retrieval, specifically utilizing Local Sensitive Hashing (LSH) to encode queries and keys into hash codes and pinpointing the best matches as the selected tokens. However, as depicted in Figure 2 , the efficacy of such linear hashing heavily depends on the hash code length, e.g. , a hash code length of 1,024 bits per query/key is necessitated to achieve promising token retrieval. Considering the already substantial size of the KV cache, storing these lengthy hash codes markedly impairs deployment efficiency. Moreover, employing a linear projection with a large output dimension for key processing incurs significant computational overhead. Delving deeper, prior work ( chen2024magicpig ) has discovered that the queries and keys within LLMs typically form nearly orthogonal cones within the embedding space, as depicted in Figure 2 . Given the truth that linear hashing function partitions the embedding space using random hyperplanes, such orthogonal distribution of queries and keys barely lead to satisfying encoding quality, which can even result in a collapse of the hashing outcomes, i.e. , identical codes for all queries and keys, as shown in Figure 2 . Therefore, extremely long hash codes are necessary to mine meaningful information and accurately match essential tokens. MagicPIG attempted to mitigate this issue by normalizing the keys before retrieval. However, this approach remains suboptimal as it overlooks the query distribution and introduces bias to the retrieval process. To address the aforementioned limitation, we propose Spotlight Attention, a novel method that replaces random hyperplanes with curved surfaces for space partitioning via a non-linear MLP hashing function. As depicted in Figure 2 , this non-linearity can better fit skewed distributions, thereby improving code quality. In particular, we utilize the Bradley-Terry ranking objective ( 19ff28b9-64f9-3656-ba40-08326a05748e ) to optimize the non-linear MLP layer, wherein the learning target involves minimizing the difference between the estimated top- k indices and the ground truth top- k indices obtained via the vanilla attention scores. This learning process is exceptionally efficient, with the LLM backbone remaining frozen and requiring only a minimal amount of calibration data. As a result, the optimized non-linear hashing function can match the performance of linear hashing while using 5 \times shorter hash codes, achieving higher efficiency than MagicPIG. We further implemented CUDA kernels for the hash code processing, including bit-packing and bitwise NXOR GEMM operators, achieving significant latency reductions in practice. For example, our method achieves up to 3 \times increase in Qwen2.5-7B ( qwen2025qwen25technicalreport ) inference throughput for both 32K and 128K sequences, with only \sim 2% performance degradation on the LLaMA3 ( grattafiori2024llama3herdmodels ) series and no loss on Qwen2.5 ( qwen2025qwen25technicalreport ) series. Our contributions are threefold: • We propose Spotlight Attention for accelerating LLM inference, which employs non-linear hashing function to encode and match queries and key values within LLMs, thereby efficiently selecting critical KV cache for model inference. • We develop a lightweight and robust training framework based on the Bradley-Terry ranking objective, which effectively optimizes the non-linear hashing function using only a small amount of calibration data. • Extensive experiments demonstrate that Spotlight Attention can drastically reduce LLM inference latency while maintaining the strongest performance retention in comparison with state-of-the-art methods. (a) (b) (c) (d) Figure 2 : Motivation. (a) The empirical evaluation shows that upgrading the hashing function from linear to MLP can bring a huge improvement, (b) this is because query and key are usually distributed

Read the complete text