PiKV: KV Cache Management System for Mixture of Experts
Dong Liu, Yanxuan Yu, Ben Lengerich, Ying Nian Wu, Xuhong Wang · 2025
Read PiKV: KV Cache Management System for Mixture of Experts by Dong Liu, Yanxuan Yu, Ben Lengerich, Ying Nian Wu, Xuhong Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06526.txt.
Introduction
Dong Liu and co-authors attack the memory bill that mixture-of-experts language models run up while answering. These models split computation across specialist subnetworks, yet their key-value caches stay dense and synchronized across every graphics card, so storage and chatter dominate at long contexts. PiKV shards the cache the way the model shards its experts.
Caches are partitioned by expert across GPUs, a routing layer trims token-to-cache traffic, and a scheduler keeps only the entries each query actually needs. Compression modules squeeze what remains. The authors have released PiKV as open-source software and describe it as a living project, aiming to become the complete cache management system for mixture-of-experts serving.
Catalog record
| CREATOR | Dong Liu, Yanxuan Yu, Ben Lengerich, Ying Nian Wu, Xuhong Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06526
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PiKV: KV Cache Management System for Mixture of Experts Dong Liu, Yanxuan Yu, Ben Lengerich, Ying Nian Wu, Xuhong Wang Abstract . As large language models continue to scale up in both size and context length, the memory and communication cost of key-value (KV) cache storage has become a major bottleneck in multi-GPU and multi-node inference. While MoE-based architectures sparsify computation across experts, the corresponding KV caches remain dense and globally synchronized, resulting in significant overhead. We introduce PiKV , a parallel and distributed KV cache serving framework tailored for MoE architecture. PiKV leverages expert-sharded KV storage to partition caches across GPUs, PiKV routing to reduce token-to-KV access, and a PiKV Scheduling to adaptively retain query-relevant entries. To further reduce memory usage, PiKV integrates PiKV Compression modules the caching pipeline for acceleration. PiKV is recently publicly available as an open-source software library: https://github.com/NoakLiu/PiKV . Experiments details is recorded at: https://github.com/NoakLiu/PiKV/Experimental_Results . We also have PiKV integrated with Nvidia kvpress for acceleration, details see https://github.com/NoakLiu/PiKVpress . PiKV is still a living project, aiming to become a comprehesive KV Cache management system for MoE Architectures. † † authors: . PVLDB Reference Format: PVLDB, 14(1): XXX-XXX, 2024. doi:XX.XX/XXX.XX † † This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing [email protected] . Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 14, No. 1 ISSN 2150-8097. doi:XX.XX/XXX.XX PVLDB Artifact Availability: The source code, data, and/or other artifacts have been made available at https://github.com/NoakLiu/PiKV . 1. Introduction Large Language Models (LLMs) have become the foundation of modern AI applications, powering virtual assistants, code generation, document analysis, and multi-turn reasoning. With increasing demand for longer sequences and sparse expert models ( Bai et al. 2025 ; Rajbhandari et al. 2020 ; Achiam et al. 2023 ) , there is huge demand to deploy sparsely-gated Mixture-of-Experts (MoE) structures ( Lepikhin et al. 2020 ; Du et al. 2022 ) to reduce computation costs at scale. However, serving such models introduces significant system-level challenges. During inference, each token generation requires attending to the entire KV cache from prior tokens. For a 7B-scale MoE model with 128K context and 16 experts, the full KV cache can occupy > 24GB of memory and incur excessive communication latency across GPUs and nodes. Even with FlashAttention-style optimizations ( Dao et al. 2022 ) , the need to load and attend to dense KV structures becomes the dominant bottleneck, especially in autoregressive decoding. Prior works ( Zhang et al. 2023 ; Gao et al. 2024 ) have shown that a small fraction of tokens contribute disproportionately to the final attention output, motivating selective cache access. Yet most methods either use static heuristics or ignore the underlying system cost of accessing KV entries across distributed compute nodes. In this work, we ask a deeper question: Can we design a KV caching system that is both sparsity-aware and system-optimized for distributed MoE inference? Figure 1. PiKV Framework Figure 2. KV cache memory usage comparison. Left: absolute memory usage of different methods; Right: memory reduction percentage compared to Full KV cache. Figure 3. Latency performance comparison. Left: latency at different sequence lengths; Right: speedup ratio of PiKV compared to other methods. We propose PiKV , a parallel distributed KV caching system tailored for sparse mixture of expert models training and inference. As shown in Figure 1 , PiKV includes three synergistic components: (1) an expert-sharded distributed KV cache layout across multi-GPU or multi-node compute, (2) a sparse expert routing layer that dynamically selects top- k experts per query, and (3) an adaptive stream scheduler that uses activity-based eviction to retain only high-utility KV entries. Our experimental results demonstrate significant improvements in both memory efficiency and inference latency. As shown in Figures 2 and 3 , PiKV achieves up to 3.9 \times memory reduction and 1.7 \times latency improvement compared to full KV cache approaches, while maintaining competitive accuracy across various benchmarks. To further reduce memory and bandwidth cost, PiKV compresses KV representations using modular schemes such as LoRA ( Hu et al. 2022 ) , PyramidKV ( Cai et al. 2024 ) , and Duo ( Chen et al. 2024 ) . We track metadata and usage patterns of each KV shard to guide eviction and cache streaming policies, enabling efficient inference under both static and streaming contexts. • We present a novel system architecture that combines sparse expert routing and distributed KV cache layout with query-aware streaming scheduling. • We propose compression-aware KV caching, integrating multiple compression schemes and eviction policies into a unified system-level framework. • We validate efficiency of PiKV in KV Cache Management of MoE Architectures, achieving significant improvements in memory, latency, and end-to-end generation efficiency. 2. Preliminary 2.1. Sparse MoE Inference Meets Long-Context Bottlenecks The rise of long-context language models has transformed downstream tasks such as multi-document reasoning, conversational memory, and retrieval-augmented generation. However, inference with context lengths beyond 32K tokens introduces prohibitive memory and latency overhead, especially when deployed with Mixture-of-Experts (MoE) models ( Lepikhin et al. 2020 ; Fedus et al. 2022 ) , which already impose routing and communication complexities. Despite the potential for sparse computation, current MoE systems suffer from dense KV cache replication , non-adaptive expert selection , and cache scheduling agnostic to query dynamics . Formally, consider a sequence of L tokens and E experts, each storing key-value (KV) representations of dimension d . Standard dense caching incurs a total memory cost of \mathcal{O}(L\cdot d\cdot E) per GPU if all experts are replicated. Even with top- k routing (where k\ll E ), expert KV stores are either globally synchronized or locally incomplete, leading to redundant storage or poor attention recall. Challenge 1: Expert-Sharded KV Fragmentation. Token-level routing naturally shards the KV cache across devices. However, naive per-expert partitioning breaks temporal locality and introduces fragmented access patterns: \text{KV}_{t}^{(e)}\in\mathbb{R}^{k\times d},\quad\text{for }e\in\mathcal{R}(q_{t}) where \mathcal{R} is the router assigning experts to token t . Efficient access thus requires query-aware retrieval across heterogeneous and often incomplete cache segments. Challenge 2: Latency Bottleneck from Sparse Lookup. Sparse MoE inference reduces compute, but not necessarily latency: each query q_{t} must attend over k expert KV sets, often spread across GPUs. Without localized cache routing and pre-filtering, the expected latency is: \mathbb{E}[\text{Latency}]\sim\mathcal{O}(k\cdot T_{\text{lookup}}+T_{\text{sync}}) where T_{\text{sync}} arises from inter-GPU communication. Challenge 3: Non-coordinated Routing, Compression, and Scheduling. Prior systems treat expert routing ( Du et al. 2022 ) , KV compression ( Liu et al. 2025 ; Liu 2024 ) , and cache scheduling ( Xiao et al. 2024 ; Tang et al. 2024 ) as disjoint modules. This results in inconsistent memory policies: a router might route to an expert whose KV cache has already evicted the relevant tokens. PiKV bridges this gap. We introduce a unified framework where: • Routing is query- and cache-aware , enabling locality-sensitive top- k expert selection; • Compression is hierarchical and expert-partitioned , minimizing redundancy without affecting reuse; • Scheduling is jointly optimized with routing , based on token-level saliency and inter-expert redundancy. By reframing the inference stack around the KV cache