Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
Kunjun Li, Zigeng Chen, Cheng-Yen Yang, Jenq-Neng Hwang · 2025
Read Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression by Kunjun Li, Zigeng Chen, Cheng-Yen Yang, Jenq-Neng Hwang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19602.txt.
Introduction
Li, Chen, Yang, and Hwang tackled the memory hunger of a new image generation method in twenty twenty-five. Visual autoregressive models build pictures scale by scale, from coarse to fine. Quality and flexibility are strong, but the stored key-value cache grows explosively during generation, wasting memory.
Their framework, ScaleKV, compresses that cache using two observations: layers differ in how much cache they need, and attention patterns change with scale. Layers split into drafters, which spread attention widely and need more cache, and refiners, which focus locally and need little. ScaleKV sizes the cache per layer and per scale accordingly. On a leading text-to-image model, memory fell to ten percent of the original while picture fidelity stayed at pixel level, the authors report.
Catalog record
| CREATOR | Kunjun Li, Zigeng Chen, Cheng-Yen Yang, Jenq-Neng Hwang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19602
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression Kunjun Li, Zigeng Chen, Cheng-Yen Yang, Jenq-Neng Hwang Abstract Visual Autoregressive (VAR) modeling has garnered significant attention for its innovative next-scale prediction approach, which yields substantial improvements in efficiency, scalability, and zero-shot generalization. Nevertheless, the coarse-to-fine methodology inherent in VAR results in exponential growth of the KV cache during inference, causing considerable memory consumption and computational redundancy. To address these bottlenecks, we introduce ScaleKV, a novel KV cache compression framework tailored for VAR architectures. ScaleKV leverages two critical observations: varying cache demands across transformer layers and distinct attention patterns at different scales. Based on these insights, ScaleKV categorizes transformer layers into two functional groups: drafters and refiners. Drafters exhibit dispersed attention across multiple scales, thereby requiring greater cache capacity. Conversely, refiners focus attention on the current token map to process local details, consequently necessitating substantially reduced cache capacity. ScaleKV optimizes the multi-scale inference pipeline by identifying scale-specific drafters and refiners, facilitating differentiated cache management tailored to each scale. Evaluation on the state-of-the-art text-to-image VAR model family, Infinity, demonstrates that our approach effectively reduces the required KV cache memory to 10% while preserving pixel-level fidelity . Code is available at https://github.com/StargazerX0/ScaleKV . Figure 1 : We introduce a new KV cache compression framework for Visual Autoregressive modeling that preserves pixel-level fidelity. On Infinity-8B, it achieves 10x memory reduction from 85 GB to 8.5 GB with negligible quality degradation (GenEval score remains at 0.79 and DPG score marginally decreases from 86.61 to 86.49 ). 1 Introduction Figure 2 : By implementing scale-aware layer budget allocation, ScaleKV enables differentiated cache management tailored to each layer’s computational demands at every scale. Recent advances in Autoregressive (AR) models [ 51 , 23 , 69 , 31 ] have achieved impressive image quality and multi-modal capabilities [ 34 , 64 , 62 , 59 , 61 ] for unified vision understanding and generation. However, the token-by-token generation approach requires numerous decoding steps. Visual Autoregressive (VAR) modeling [ 54 ] has revolutionized this process through next-scale prediction, enabling models to decode multiple tokens in parallel. Building upon this framework, several approaches [ 16 , 53 ] have demonstrated promising results for VAR-based text-to-image generation. Despite these advancements, VAR models face a fundamental scalability challenge due to exponential growth in token sequence across scales. Unlike traditional next-token prediction models, which process one token per step with linear KV cache growth, VAR models must preserve KV states from all previous token maps. Generating a 1024 \times 1024 image requires processing over 10K tokens across multiple scales, creating severe memory bottlenecks—with KV cache alone consuming approximately 85 GB of memory when generating 1024 \times 1024 images with a batch size of 8 using Infinity-8B [ 16 ] , a text-to-image VAR model. Figure 3 (a) illustrates this complexity, where each scale contributes a new KV cache entry of size h_{k}\times w_{k} . The total cache requirement grows cubically with the number of scales n , while the computational complexity of attention reaches \mathcal{O}(n^{4}) . These memory constraints and increased inference latency significantly impede practical deployment. To address these inefficiencies, we analyze the specific properties of VAR’s next-scale prediction paradigm. First, we observe that cache demands vary significantly across transformer layers. Next, we find that different scales exhibit distinct attention patterns. These findings indicate that VAR requires both layer-adaptive and scale-specific cache management strategies for optimal performance. Our Approach. Inspired by these observations, we propose Scale-Aware KV Cache (ScaleKV), a simple yet highly effective method that significantly reduces inference memory while maintaining high generation quality. Our approach categorizes transformer layers into two functional groups: Drafters and Refiners . Drafters distribute attention across multiple scales to access global information from preceding tokens, requiring greater cache capacity. In contrast, refiners focus attention on current token map to process local details, necessitating substantially reduced cache storage. As illustrated in Figure 2 , ScaleKV optimizes multi-scale inference by identifying scale-specific drafters and refiners, facilitating differentiated cache management to their computational demands at each scale. Extensive evaluation demonstrates the effectiveness of our method. As shown in Figure 1 , compared to the original Infinity-8B model, ScaleKV achieves negligible quality degradation (GenEval score remains at 0.79 and DPG score decreases slightly from 86.61 to 86.49) while requiring merely 10% of the original GPU memory consumption. These results validate that ScaleKV effectively addresses the fundamental memory bottlenecks that have constrained the practical deployment of VAR models. In conclusion, we introduce ScaleKV, a novel KV cache compression framework for VAR. ScaleKV categorizes transformer layers into drafters and refiners and implements scale-aware layer budget allocation. Through extensive experiments, our method achieves significant memory reduction while preserving pixel-level fidelity, enabling efficient deployment in resource-constrained environments. 2 Related Works Autoregressive Visual Generation. Early works [ 6 , 55 ] pioneered pixel-by-pixel image generation, later enhanced by VQVAE [ 56 ] and VQGAN [ 8 ] through image patch quantization. Recent advances include GPT-style models [ 52 , 34 ] , mixture-of-experts [ 23 ] , linear attention [ 26 ] , diffusion-autoregressive hybrids [ 64 , 82 , 14 ] , and masked approaches [ 26 , 4 , 38 ] . However, autoregressive approaches suffer from substantial inference latency due to sequential token generation. VAR [ 54 ] overcomes this limitation through hierarchical parallel decoding, and has been extended to text-to-image synthesis [ 16 , 75 , 53 , 40 , 27 , 31 ] , audio synthesis [ 45 ] , and 3D content creation [ 74 ] . Efficient Visual Generation. For diffusion models, efficiency optimization methods are already well-developed. [ 47 , 72 , 37 , 48 , 70 ] focus on reducing sampling steps while [ 29 , 81 , 9 , 73 , 68 , 28 , 49 , 24 ] optimize models through quantization [ 24 ] , pruning [ 9 ] or knowledge distillation [ 19 ] . Several approaches [ 41 , 60 , 77 , 67 , 50 , 25 , 39 , 80 ] skip redundant computations during the denoising process. However, research into memory optimization for VAR image generation remains in its early stages. LiteVAR [ 65 ] and FastVAR [ 15 ] enhance inference speed but do not effectively address the fundamental memory bottlenecks, while CoDe [ 7 ] improves memory efficiency through collaborative decoding but requires a secondary VAR model. In contrast, our approach directly reduces memory consumption and can be integrated with existing techniques to further enhance overall efficiency. KV Cache Compression. Current KV cache compression techniques for large language models (LLMs) and vision-language models (VLMs) primarily utilize quantization [ 36 , 71 , 22 , 17 ] , eviction [ 79 , 35 , 42 , 46 , 30 , 44 ] , and merging [ 78 , 33 , 57 , 58 ] strategies. Quantization reduces precision but faces granularity limitations; eviction removes less important tokens using attention metrics while optimizing allocation within budgets [ 12 , 66 , 10 , 11 ] ; and merging consolidates redundant KV pairs. However, primarily designed for single-sequence processing in LLMs and VLMs, existing techniques fail to accommodate the multi-scale operational characteristics of VAR and the varying attention patterns