DynaKV : Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphones
Tuowei Wang, Minxing Huang, Fengzu Li, Ligeng Chen, Jinrui Zhang, Ju Ren · 2025
Read DynaKV : Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphones by Tuowei Wang, Minxing Huang, Fengzu Li, Ligeng Chen, Jinrui Zhang, Ju Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07427.txt.
Introduction
Phones struggle to run long language model replies because memory fills up fast. Tuowei Wang and colleagues explain that the key-value cache, which stores context during generation, grows in step with text length. Moving parts of it to flash storage helps, but fixed groupings drift out of date as decoding proceeds.
DynaKV adapts the cache layout as generation moves forward, keeping needed entries close and dropping waste. The design also respects phone limits on bandwidth, storage speed, and memory. Tests report accurate long replies with far less strain on device memory. The work targets practical on-device assistants that can sustain long conversations.
Catalog record
| CREATOR | Tuowei Wang, Minxing Huang, Fengzu Li, Ligeng Chen, Jinrui Zhang, Ju Ren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07427
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DynaKV : Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphones Tuowei Wang, Minxing Huang, Fengzu Li, Ligeng Chen, Jinrui Zhang, Ju Ren Abstract . As the demand for human-like reasoning, multi-turn dialogues, and long-form responses grows, large language models (LLMs) are increasingly expected to support efficient and effective long-sequence decoding. However, due to limited DRAM capacity, long-sequence LLM decoding on smartphones is constrained by the key–value cache (KVCache), whose memory footprint increases linearly with sequence length. Retrieval-based methods mitigate DRAM pressure by offloading KVCache to flash and retrieving query-relevant entries through cluster-based indexing. Unfortunately, as decoding progresses, KVCache distribution shifts render static or local cluster updates progressively misaligned, excluding essential entries or fetching redundant ones. These issues are further exacerbated by smartphone-specific limitations in bandwidth, IOPS, and memory capacity. We propose DynaKV , the first adaptive KVCache management approach that jointly addresses accuracy and efficiency for long-sequence decoding on smartphones. DynaKV integrates three key techniques: (1) Migration-Free Cluster Adaptation, which adaptively splits clusters during retrieval without incurring additional transfers; (2) Continuity-Centric Flash Management, which co-locates correlated entries and clusters and employs a dual-head layout for efficient updates; and (3) Memory-Efficient Cache Design, which virtualizes cache space across DRAM and flash and extends replacement policies to align with cluster-level access patterns. Evaluations demonstrate that DynaKV improves retrieval accuracy and reduces end-to-end latency compared to state-of-the-art solutions, achieving average gains of 1.38 × 1.38\times in accuracy and 1.47 × 1.47\times speedups. Furthermore, the insights of DynaKV naturally extend to other long-context workloads and multi-tier memory hierarchies, underscoring its broader applicability. 1. Introduction The emergence of large language models (LLMs) has fundamentally transformed the processing of long-context data. Beyond understanding extensive inputs such as documents ( document-1 ; document-2 ) or codebases ( codebase-1 ; codebase-2 ) , LLMs are increasingly expected to generate coherent and long-form outputs, particularly on mobile devices like smartphones . On one hand, state-of-the-art LLMs are moving toward reasoning-oriented designs ( openai-o1 ; deepseek-r1 ; gemini2.5 ) , leveraging structured thought processes to strengthen problem-solving capabilities. On the other hand, mobile applications such as advanced conversational agents demand both detailed response generation ( longlamp ; hellobench ; seed-story ) and sustaining multi-turn dialogues ( mt-bench ; mt-bench++ ; mt-bench-101 ) . These trends together underscore the critical need to enable efficient and effective long-sequence LLM decoding on smartphones. Similar to other long-context scenarios, the limited memory capacity on smartphones poses a major challenge for long-sequence LLM decoding. Since the memory usage of the key–value cache (KVCache) scales linearly with sequence length, it quickly becomes the dominant bottleneck at longer contexts. To mitigate this issue, prior studies have investigated the contextual sparsity inherent in the attention mechanism ( attention-sparsity ; deja-vu ; jenga ; long-exposure ) . In practice, each query vector interacts significantly with only a small, query-dependent subset of key and value vectors. Building on this property, a variety of retrieval-based methods ( pqcache ; clusterkv ; infllm ; retrieval-attention ) have been proposed for memory-constrained settings, offloading the entire KVCache to external storage and retrieving only the relevant entries into main memory on demand. Figure 1. Illustration of three KVCache cluster management strategies. (a) Static Update: new entries are greedily appended to their nearest existing clusters. (b) Local Update: new entries are re-clustered independently of existing clusters. (c) DynaKV : adaptively appends new entries or splits existing clusters to maintain accuracy and efficiency. To balance retrieval efficiency and accuracy, the KVCache is typically partitioned into clusters based on embedding similarity, which then serve as the fundamental retrieval units. Since global clustering is both computationally and memory-intensive, it is performed only once during the prefill phase. Newly generated KV entries are either greedily appended to the nearest existing clusters ( Static Update ) ( pqcache ; squeezed-attention ) or periodically re-clustered within a local context window ( Local Update ) ( clusterkv ; shadowkv ) , as illustrated in Figure 1 . These methods remain effective when the prefill proportion is relatively large. However, the situation changes in long-sequence decoding, where generated tokens dominate the inference context. Specifically, we observe that as the decode length increases, the distribution of the KVCache undergoes continual transformation, exhibiting clustering patterns that diverge substantially from the initial ones, a phenomenon we term KVCache Distribution Shift . Unfortunately, existing clustering methods fail to effectively accommodate this dynamic shift, resulting in suboptimal KVCache retrieval. In particular, excluding essential entries degrades inference accuracy, while transferring redundant entries wastes I/O bandwidth. This observation motivates a critical question: Can we develop an adaptive KVCache management that preserves both accuracy and efficiency of KVCache retrieval on smartphones? However, this is not a low-hanging fruit, as it stems from the inherent misalignment between the adaptive demands of clustering mechanisms and the stringent constraints of smartphone memory hierarchies. The presence of KVCache clustering shift necessitates access to global information for adaptive clustering, which unavoidably incurs additional data transfers from external storage to main memory. Given the hardware characteristics of smartphones, such I/O operations are readily bottlenecked by both limited bandwidth , restricted Input/Output Operations Per Second (IOPS) , and constrained memory capacity . Our analysis highlights three critical technical challenges that must be tackled: (1) Bandwidth-Limited Adaptation. Adaptive clustering requires continuous updates to existing clusters as new KV entries are generated. Since external storage lacks compute capability, the relevant clusters must first be transferred into main memory before being updated. The resulting frequent transfers can quickly lead to I/O performance bottlenecks. (2) IOPS-Bounded Flash Access. Retrieved KV entries exhibit scattered and dynamic access patterns across different queries. Placing the KVCache naively in strict sequence order results in fragmented flash accesses, exacerbating the IOPS bottleneck. Alternatively, organizing it by clusters improves continuous reads but raises challenges for frequent updates. (3) Memory-Constrained Caching. Retaining frequently retrieved KV entries in main memory is essential for reducing transfer overhead. However, the distinctive access patterns of KVCache render standard replacement policies ineffective. Moreover, cluster-based organization inflates cache demands, which is incompatible with the limited memory capacity. In this paper, we propose DynaKV , an efficient approach that enables long-sequence LLM decoding on smartphones with adaptive KVCache management. DynaKV employs three key techniques to tackle these challenges, respectively: (1) Migration-Free Cluster Adaptation. DynaKV design a variance-based self-adaptive clustering algorithm that preserves clustering effectiveness while maintaining cluster granularity as coarse as possible. To avoid extra I/O overhead, DynaKV adopts a delayed-split strategy that integrates cluster splitting naturally into the existing retrieval workflow. (2) Continuity-Centric Flash Management. To facilitate continuous read access and mitigate IOPS bottlenecks, DynaKV places all entries within a cluster as well as highly-correlated clusters continuously in flash. To support frequent adaptations, DynaKV employs a dual-head layout that allows cluster updates without interfering with each other. (3) Memory-Efficient Cache Design. DynaKV constructs a virtual cache space that unifies DRAM and flash. By leveraging idle bandwidth during computation, cache clusters are seamlessly swapped between the two tiers. To further improve cache hit rate, DynaKV augments standard replacement policies to align with cluster-based retrieval. We evaluate DynaKV on three smartphones with distinct hardware configurations, benchmarking a diverse range of LLMs varying in structures and scales. The results demonstrate that DynaKV outperforms the state-of-the-art solutions in