Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach
Yaoxin Yang, Peng Ye, Xudong Tan, Chongjun Tu, Maosen Zhao, Jia Hao, Tao Chen · 2025
Read Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach by Yaoxin Yang, Peng Ye, Xudong Tan, Chongjun Tu, Maosen Zhao, Jia Hao, Tao Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16786.txt.
Introduction
Yaoxin Yang and six co-authors compress the key-value cache of multimodal language models, whose memory grows with every visual token and throttles inference. Prior compression scored entries by attention weight, which clashes with fast attention kernels and ignores what the value vectors contribute to the output.
Looking at the cache matrices themselves, the authors find their frequency-domain energy concentrated at low frequencies. Entries that deviate strongly from that smooth principal component, dubbed Outlier KVs, turn out to carry critical features: deleting them hurts badly. Their FlashCache framework detects and preferentially keeps those outliers and allocates each layer's budget dynamically. On several multimodal models it decoded up to about 1.7 times faster while using eighty percent less cache memory at equal task performance.
Catalog record
| CREATOR | Yaoxin Yang, Peng Ye, Xudong Tan, Chongjun Tu, Maosen Zhao, Jia Hao, Tao Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-16786
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach Yaoxin Yang, Peng Ye, Xudong Tan, Chongjun Tu, Maosen Zhao, Jia Hao, Tao Chen Abstract Multimodal large language models suffer from substantial inference overhead since multimodal KV Cache grows proportionally with the visual input length. Existing multimodal KV Cache compression methods mostly rely on attention score to reduce cache size, which makes them are incompatible with established efficient attention kernels (e.g., FlashAttention) and ignores the contribution of value vectors to the attention output. In this work, we revisit multimodal KV Cache compression from the perspective of the KV matrices’ distribution. First, we observe that frequency-domain energy of multimodal KV matrices is predominantly concentrated in low-frequency and extract this principal energy via a low-pass filter. Further, we find that removing KV pairs that deviate substantially from this principal energy leads to a pronounced performance drop, which we define as Outlier KVs . Considering Outlier KVs are more likely to encode features critical for inference, we propose FlashCache , a frequency-domain–guided, Outlier-KV-aware KV Cache compression framework. First, we introduce an Outlier KV Recognition Module that models the principal component of multimodal KV matrices in the frequency domain and preferentially retains KV pairs that significantly deviate from it. Furthermore, Dynamic Budget Allocation Module is designed to adaptively determine the per-layer KV Cache size to retain more Outlier KVs. Experiments on multiple MLLMs and benchmarks demonstrate that FlashCache outperforms state-of-the-art multimoal KV compression methods, achieving up to 1.69× faster decoding with 80% lower KV memory usage while maintaining task performance. 1 Introduction Figure 1 : Frequency-domain energy distribution of the KV matrices. We observe that the frequency-domain energy of KV matrices is predominantly concentrated at low frequency, with high frequency components occupying a relatively small proportion. The red dashed box in the figure highlights the low-frequency concentration phenomenon. Experiments are conducted on MileBench 29 with Qwen2.5-VL-7B. Multimodal large language models (MLLMs) 1 ; 47 ; 7 ; 19 ; 22 ; 36 exhibit strong multimodal perception and reasoning, yet their practical deployment is hindered by high computation and memory costs, driven primarily by the substantial volume of visual tokens at inference time 17 ; 15 . To address this, many techniques, such as token pruning 6 ; 13 ; 30 ; 45 , model quantization 42 ; 41 , and KV Cache optimizations 2 ; 14 ; 18 , have been proposed to accelerate MLLM inference. Among them, KV Cache improves decoding efficiency by storing the Keys and Values of previously seen tokens, avoiding repeated prefix computation during decoding, thereby garnering significant attention 28 ; 11 . Despite achieving impressive progress, as context length grows (e.g., multi-image, high-resolution and video settings 29 ; 25 ; 37 ; 31 ), the length of multimodal KV Cache increases accordingly, leading to substantial GPU memory overhead and a sharp slowdown during the decoding stage. Recently, several works have attempted to prune redundant portions of long-context KV Cache in MLLMs to further reduce memory usage and speed up decoding. LOOK-M 34 observes extensive redundancy in visual KVs within long multimodal contexts and compresses the KV Cache by screening and merging visual KV pairs with low attention score during the prefill stage. MEDA 33 uses cross-modal attention entropy to allocate an appropriate KV Cache size per MLLM layer. However, these approaches all rely on attention score to shrink the KV Cache, which raises several issues: (1) efficient attention kernels (e.g., FlashAttention 8 ) typically do not explicitly output the full attention score, and recomputing attention score introduces additional overhead; (2) since attention score are obtained via Query–Key dot products, directly compressing KV Cache via attention score ignores the information contribution of the Value matrix 26 . This naturally leads to the question: how can we perform efficient multimodal KV Cache compression without relying on attention score ? Figure 2 : The impact on model performance of prioritizing the removal of KV pairs under different removal ratios. We find discarding KV pairs with larger deviations leads to faster performance drop than random removal or removing small-deviation pairs. Experiments are conducted on MileBench 29 with Qwen2.5-VL-7B. In this work, we revisit the problem from the perspective of the data distribution of the visual KV matrices themselves. Inspired by the common use of frequency-domain analysis in image processing 32 , we transform visual KV matrices into the frequency domain and analyze their energy distribution (i.e., the average power spectrum). As shown in Fig. 1 , we observe that the frequency-domain energy of KV matrices is predominantly concentrated at low frequencies, with high-frequency components occupying a relatively small proportion. Further, noting that removing outliers in model quantization often causes steep performance drops 9 ; 39 ; 10 , we hypothesize that a similar outlier phenomenon may exist for KV Cache compression. To investigate, we apply a low-pass filter to obtain smoothed and dominant KV matrices (termed as Base KV ), then measure the deviation of the original KV matrices from Base KV and remove KV pairs with different deviations. We find that, discarding KV pairs with larger deviations leads to faster performance drop than random removal or removing small-deviation pairs (Fig. 2 ), whereas discarding small-deviation KV pairs has smaller impact on performance. These results indicate KV pairs that differ substantially from the Base KV are more likely to carry features critical to model inference, which we define as Outlier KVs . Building on this novel finding, we propose FlashCache , a frequency-domain-guided Outlier-KV-aware compression approach with two core components. First, we introduce an Outlier KV Recognition Module that efficiently selects Outlier KVs in each layer. Concretely, we apply Discrete Cosine Transform (DCT) 3 to map the KV matrices to the frequency domain and retain only the dominant low-frequency components; we then perform the inverse DCT to return to the time domain and obtain the smoothed Base KV ; after that, we compute the mean squared error between each original KV pair and its Base KV counterpart to select Outlier KVs. Second, we further observe differences in the low-frequency concentration phenomenon across each layer of the model in Fig. 4 , and propose a frequency-guided Dynamic Budget Allocation Module to dynamically determine the KV Cache size per layer. For each layer, we analyze the KV matrices in the frequency domain and compute the ratio of outlier-information energy to the total KV energy; we normalize these ratios into weights and allocate distinct per-layer retention quotas under a global cache budget. Across multiple models and datasets, FlashCache reduces KV usage and accelerates decoding under the same context length while preserving model performance. Our contributions are as follows: • To the best of our knowledge, we are the first to analyze multimodal KV Cache compression from a frequency-domain perspective. First, we find that the frequency energy of original KV matrices concentrates primarily in low frequency. Further, by applying a low-pass filter to obtain a smoothed Base KV , we observe that preferentially discarding KV pairs that differ most from the Base KV causes rapid performance degradation. It indicates that such KV pairs are more likely to encode features crucial for inference, which we define as Outlier KVs . • Based on this novel finding, we propose FlashCache