Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Acceleration Conference: Great Lakes Symposium on VLSI 2025; June 30-July 2, 2025; New Orleans, LA, USA Great Lakes Symposium on VLSI 2025 (GLSVLSI ’25), June 30-July 2, 2025, New Orleans, LA, USA DOI: 10.1145/3716368.3735145 ISB
Peilin Chen, Xiaoxuan Yang · 2025
Read Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Acceleration Conference: Great Lakes Symposium on VLSI 2025; June 30-July 2, 2025; New Orleans, LA, USA Great Lakes Symposium on VLSI 2025 (GLSVLSI ’25), June 30-July 2, 2025, New Orleans, LA, USA DOI: 10.1145/3716368.3735145 ISB by Peilin Chen, Xiaoxuan Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17787.txt.
Introduction
Peilin Chen and Xiaoxuan Yang present, in a 2025 paper, a hardware and software design that shrinks the memory a large language model needs while it writes. These models store key and value data to avoid redoing work, but that cache can grow far larger than the model itself on long inputs. Their design, Titanus, squeezes the cache as the model runs.
A method they call cascade pruning-quantization trims and compresses the cache in steps, and only the non-zero parts are moved between chip and memory. A two-stage search tunes the design, and a pipeline layout cuts the wait for the first token. In tests, Titanus used far less energy and ran much faster than a top graphics chip and a rival accelerator.
Catalog record
| CREATOR | Peilin Chen, Xiaoxuan Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17787
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Acceleration Conference: Great Lakes Symposium on VLSI 2025; June 30-July 2, 2025; New Orleans, LA, USA Great Lakes Symposium on VLSI 2025 (GLSVLSI ’25), June 30-July 2, 2025, New Orleans, LA, USA DOI: 10.1145/3716368.3735145 ISBN: 979-8-4007-1496-2/2025/06 CCS: Hardware Application-specific VLSI designs Peilin Chen, Xiaoxuan Yang Abstract . Large language models (LLMs) have gained great success in various domains. Existing systems cache Key and Value within the attention block to avoid redundant computations. However, the size of key-value cache (KV cache) is unpredictable and can even be tens of times larger than the weights in the long context length scenario. In this work, we propose Titanus , a software-hardware co-design to efficiently compress the KV cache on-the-fly. We first propose the cascade pruning-quantization (CPQ) method to reduce the KV cache movement. The hierarchical quantization extension strategy is introduced to tackle the non-independent per-channel quantization issue. To further reduce KV cache movement, we transfer only the non-zero KV cache between the accelerator and off-chip memory. Moreover, we customize a two-stage design space exploration framework for the CPQ method. A novel pipeline and parallelism dataflow is designed to reduce the first token generation time. Experiments show that Titanus achieves 159.9 × 159.9\times ( 49.6 × 49.6\times ) and 34.8 × 34.8\times ( 29.2 × 29.2\times ) energy efficiency (throughput) compared to Nvidia A100 GPU and FlightLLM respectively. The code for Titanus is available at this link . † † cc-license: by 1. Introduction Transformer-based large language models (LLMs) have achieved state-of-the-art (SOTA) results in a wide range of natural language processing tasks ( Yang et al. 2024 ; Zhang et al. 2024 ) . The attention mechanism, which combines the computation of query, key, and value, is an important building block for the LLMs. To improve the capabilities of LLMs, the model size has expanded from billions to trillions of parameters. Such large-scale LLMs consume significant amounts of energy to support their inference ( Luo et al. 2024 ) . In order to reduce the parameter size, prior work ( Lin et al. 2024 ) quantizes LLM weights into low-bit. Moreover, prior works ( Xiao et al. 2023 ; Dettmers et al. 2022 ; Shao et al. 2023 ; Ashkboos et al. 2023 ) quantize both weights and activations to achieve significant improvements in latency and memory reduction. However, the key-value cache (KV cache), which stores the attention keys and values to avoid redundant computations, is becoming the emerging memory bottleneck for LLM inference. For example, in the OPT-175B model ( Zhang et al. 2022 ) , with a batch size of 16, an input context length of 512, and an output context length of 1024, KV cache size can reach up to 1.2TB , which is 3.8\times the model’s weights ( Sheng et al. 2023 ) . Most hardware platforms like GPUs and the SOTA transformer accelerator FlightLLM ( Zeng et al. 2024 ) store all weights and KV cache on high bandwidth memory (HBM). Due to the autoregressive nature in the decode stage, these platforms repeatedly reload all the LLM weights and KV cache to generate the next token. A large and inevitable portion of energy is consumed in the data movement process. More importantly, the KV cache cannot be pre-quantized like weights because the key and value are dynamically generated during inference. Therefore, we are inspired to design a specialized accelerator that supports KV cache pruning and quantization on-the-fly to reduce both data movement and computation overhead. On the algorithm side, ThinK ( Xu et al. 2024 ) focuses on key cache pruning and does not prune the most recent tokens and newly generated key cache. This approach cannot guarantee a high pruning ratio when the prompt is short and the output context length is large. KIVI ( Liu et al. 2024 ) and KVQuant ( Hooper et al. 2024 ) quantize KV cache into 3-bit and even 2-bit with a slight accuracy drop, using group-wise per-channel quantization (PCQ) for key cache and group-wise per-token quantization (PTQ) for value cache. However, we observe that the computation and storage overhead of PTQ is significantly larger than PCQ due to the long input/output context length. Moreover, we find that KV cache across different layers, and even Key and Value within each layer, exhibit varying sensitivities to quantization bit-width. This motivates us to adopt a hardware-friendly and layer-sensitive mixed bit-width quantization for the KV cache, aiming to achieve significant compression with minimal hardware overhead. On the hardware side, prior works A^{3} ( Ham et al. 2020 ) , ELSA ( Ham et al. 2021 ) , DOTA ( Qu et al. 2022 ) , and FlightLLM ( Zeng et al. 2024 ) propose various techniques to address the quadratic computation and memory complexity challenge of self-attention mechanism, including approximate self-attention algorithm, low-rank linear transformation, and utilizing block-wise and N:M sparsity patterns. Previous hardware designs ignore the KV cache optimization and overlook the inefficiency of repeatedly transferring all weights and KV cache between the accelerator and memory during the decode stage. To tackle these challenges, we propose Titanus , which supports KV cache pruning and quantization on-the-fly and adopts chiplet-based computing-in-memory (CIM) ( Yang et al. 2022b ) design to eliminate static weights movement. Specifically, the contributions of our work are as follows. (1) We design the specialized hardware Titanus to support KV cache pruning and quantization on-the-fly to reduce the transferred KV cache size. (2) We propose the cascade pruning-quantization compression method to reduce KV cache movement. Moreover, hierarchical quantization extension strategy is designed to address the non-independent per-channel quantization issue. (3) To further reduce KV cache movement, we transfer only the non-zero KV cache between the accelerator and off-chip memory, omitting the zeros from pruning and quantization. (4) We explore optimal pruning and quantization configurations for Key and Value in different LLM layers through a two-stage design space exploration (DSE). (5) We introduce the intra-core pipeline and inter-core parallelism dataflow to reduce the first token generation time. 2. Preliminary and Motivation 2.1. Computing-in-Memory The computing-in-memory architecture has a significant energy efficiency advantage against the conventional platforms (CPU, GPU, and FPGA) ( Song et al. 2017 ; Verma et al. 2019 ; Yang et al. 2020 ) . Prior CIM designs ( Tu et al. 2022a ; Chih et al. 2021 ; Tu et al. 2022b ) have limited chip area and cannot store all weights of LLMs. The large-size data movement still exists in their CIM designs, which makes it challenging to achieve the best performance brought by CIM ( Chen et al. 2025 ) . In this work, to fully utilize the advantage of CIM architecture, we design the chiplet-based CIM accelerator to avoid repeatedly weight reloading by keeping all static weights on-chip. We do not store the dynamic weight KV cache on-chip due to the unpredictable size of the KV cache. To tackle the KV cache movement challenge, Titanus compresses the KV cache significantly on-the-fly to reduce dynamic weight movement while maintaining the accuracy of LLMs. 2.2. Group-wise PCQ and PTQ Prior algorithm works ( Liu et al. 2024 ; Hooper et al. 2024 ) quantize KV cache in both