PAM: Processing Across Memory Hierarchy for Efficient KV-centric LLM Serving System DOI: XXXXXXX.XXXXXXX Conference: The 58th IEEE/ACM International Symposium on Microarchitecture; October 18–22, 2025; Seoul, Korea ISBN: 978-X-XXXX-XXXX-X/XX/XX

Lian Liu, Shixin Zhao, Yutian Zhou, Yintao He, Mengdi Wang, Yinhe Han, Ying Wang · 2026

Read PAM: Processing Across Memory Hierarchy for Efficient KV-centric LLM Serving System DOI: XXXXXXX.XXXXXXX Conference: The 58th IEEE/ACM International Symposium on Microarchitecture; October 18–22, 2025; Seoul, Korea ISBN: 978-X-XXXX-XXXX-X/XX/XX by Lian Liu, Shixin Zhao, Yutian Zhou, Yintao He, Mengdi Wang, Yinhe Han, Ying Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-11521.txt.

Introduction

Serving large language models is bottlenecked by key-value operations, not just compute. Lian Liu and six co-authors propose PAM, a key-value-centric serving system that coordinates heterogeneous processing-in-memory devices across a memory levels. Single-level memory designs cannot meet bandwidth and capacity demands jointly.

PAM distributes key-value tokens across the levels by exploiting context locality, computes focus in parallel across devices with a new recipe, and balances workloads through mapping, migration, and online scheduling interfaces. Addressing bandwidth and capacity jointly, PAM greatly enhances speed and scalability for large-scale serving, paving the way for cost-effective high-results systems in the era of long contexts.

Catalog record

CREATORLian Liu, Shixin Zhao, Yutian Zhou, Yintao He, Mengdi Wang, Yinhe Han, Ying Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-11521

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PAM: Processing Across Memory Hierarchy for Efficient KV-centric LLM Serving System DOI: XXXXXXX.XXXXXXX Conference: The 58th IEEE/ACM International Symposium on Microarchitecture; October 18–22, 2025; Seoul, Korea ISBN: 978-X-XXXX-XXXX-X/XX/XX Lian Liu, Shixin Zhao, Yutian Zhou, Yintao He, Mengdi Wang, Yinhe Han, Ying Wang Abstract . The widespread adoption of Large Language Models (LLMs) has exponentially increased the demand for efficient serving systems. With growing requests and context lengths, key-value (KV)-related operations, including attention computation and KV cache storage, have emerged as critical bottlenecks. They require massive memory bandwidth and capacity. Unfortunately, existing LLM serving systems, optimized for compute-bound workloads, fail to handle these memory-intensive operations effectively. Even with Processing-In-Memory (PIM) technology, current single-level memory designs cannot simultaneously satisfy the bandwidth and capacity requirements, particularly in cost-sensitive cloud environments. To address these challenges, we propose Processing Across Memory (PAM), a KV-centric LLM serving system that coordinates heterogeneous PIM-enabled memory devices within a hierarchical architecture. PAM introduces a novel computing paradigm to balance high memory bandwidth with scalable capacity. First, PAM exploits the inherent context locality in KV access patterns to intelligently distribute KV tokens across the memory hierarchy with near-data computing capability. Second, to further exploit context locality, it introduces the PAMattention algorithm, enabling fine-grained parallel attention computation across heterogeneous PIM devices. Finally, PAM incorporates an intra-device KV mapping, inter-device KV migration interface, and an inter-device online KV scheduling algorithm to dynamically balance computational workloads and reduce data transfer overhead. PAM adopts a full-stack design that orchestrates diverse PIM devices as a unified, hierarchical system to meet both bandwidth and capacity demands. By addressing both bandwidth and capacity demands simultaneously, PAM significantly enhances the efficiency and scalability of LLM serving systems, paving the way for cost-effective, high-performance solutions in the era of large-scale AI. 1. Introduction Large Language Models (LLMs) ( Achiam et al. 2023 ; Touvron et al. 2023a ; Touvron et al. 2023b ) have emerged as a transformative force in generative AI, revolutionizing applications such as chatbots ( Chiang et al. 2023 ; Chiang et al. 2024 ; Guo et al. 2025 ) , search engines ( Kabakuş and Dogru 2024 ; Xiong et al. 2024 ) , and automated customer service ( Roziere et al. 2023 ; Peng et al. 2023 ) . The rapid adoption of LLM-based solutions has led to an exponential increase in demand for efficient serving systems, with modern deployments handling millions of requests daily ( Lammertyn 2024 ) . This surge in demand has driven the development of various LLM serving systems ( Kwon et al. 2023a ; Yu et al. 2022 ; Zhong et al. 2024 ; Patel et al. 2024 ; Liu et al. 2024a ; Liu et al. 2024c ; Li et al. 2024 ; Qin et al. 2025 ; Agrawal et al. 2024 ) , which aim to optimize serving performance in increasingly resource-intensive scenarios. Figure 1 . Illustration of (a) vLLM with offloading; (b) Layered PIM; and (c) PAM. System illustration is on the left side of each subfigure and the performance breakdown is on the right side of each subfigure. The x-axis of the histogram is batch size. The red solid line indicates the maximum capacity of HBM and the red dotted line indicates the maximum DDR capacity. Despite these advancements, existing solutions are constrained by two major limitations. First, many systems rely on large-scale, high-performance hardware infrastructures, such as tightly interconnected GPU clusters and expensive High Bandwidth Memory (HBM), to improve processing capability. While effective, these approaches often come with exorbitant costs and energy consumption, making them impractical for cost-sensitive environments. Second, these systems are predominantly compute-centric, prioritizing the optimization of compute-bound operations. However, key-value (KV)-related operations, including attention computation and KV cache storage, constitute the primary bottleneck in LLM serving systems. Attention computation, which generally accounts for over 80% of runtime, is inherently bandwidth-bound as it requires loading all previously generated KV (referred to as KV-tokens in this paper) to generate each new token ( Vaswani 2017 ) . KV cache, storing precomputed key-value vectors to avoid redundant computations, can occupy over 95% of the total memory capacity ( Kwon et al. 2023a ; Hong et al. 2024 ; Li et al. 2024 ) . Together, these operations demand both high memory bandwidth and vast memory capacity, creating a fundamental mismatch with compute-optimized designs. Recent studies have explored various hardware ( Park et al. 2024a ; Zhou et al. 2022 ; Wu et al. 2024b ; Park et al. 2024b ; Pan et al. 2024 ) and software ( Aminabadi et al. 2022 ; Alizadeh et al. 2023 ; Team 2023b ; Jiang et al. 2024 ) techniques to alleviate these bottlenecks. Among these, Processing-In-Memory (PIM) architectures 1 1 1 For simplicity, DRAM-based and Flash-based PIM/Near Data Processing are collectively referred to as PIM in this paper. ( Wu et al. 2024b ; Seo et al. 2024 ; Kim et al. 2023b ; Lee et al. 2022 ; Nayak et al. 2024 ; Kwon et al. 2022 ) have gained prominence. These architectures integrate computational units directly into memory devices, providing increased bandwidth. Systems such as AttAcc! ( Park et al. 2024a ) , which leverage HBM-PIM ( Lee et al. 2021 ) , achieve up to 9 \times higher aggregated memory bandwidth than NVIDIA DGX A100 ( NVIDIA 2021 ) , significantly improving attention computations performance. However, the high cost of large-scale HBM-PIM deployments renders them impractical for scaling KV cache capacity to handle millions of requests. Alternatively, as shown in Figure 1 (a), offloading-based approaches ( Lee et al. 2024 ; Sheng et al. 2023 ; Song et al. 2023 ; Xue et al. 2024 ; Hwang et al. 2024 ) transfer KV data to inexpensive host memory (e.g., DDR) or non-volatile memory (e.g., SSDs) to expand memory capacity. While cost-effective, these systems suffer from limited bandwidth, as SSDs typically offer less than 5 GB/s compared to the 2 TB/s bandwidth of GPUs with HBM. While these approaches have shown promise, they fail to simultaneously address the dual demands for bandwidth and capacity. Instead of relying on isolated memory levels or compute-centric optimizations, we argue for a hierarchically coordinated approach that leverages the strengths of heterogeneous memory devices. A straightforward solution is to enable in-situ processing in the current hierarchical memory system, i.e. layered processing-in-memory (L-PIM) system as we defined. As shown in Figure 1 (b), L-PIM distributes KV tokens across HBM, DDR, and SSD, providing large capacity and enhanced bandwidth ( Lockerman et al. 2020 ; Schwedock et al. 2022 ; Wang et al. 2023 ; Fujiki et al. 2022 ; Xiao et al. 2018 ) . However, simply putting PIM devices together shows that a new bottleneck emerges in attention computation, e.g. SSDs (SSD-Attn), as memory usage increases. This bottleneck arises from a mismatch between the memory access intensity and the provided bandwidth of different memory devices in the L-PIM system. Specifically, since all KV tokens participate in attention computation, the lowest-level memory, such as SSDs, which store the majority of KV tokens, endures the highest frequency of data accesses. However, their limited bandwidth poses significant challenges for efficient data retrieval, leading to substantial performance degradation. For example, SSD-PIM solutions (

Read the complete text