PruneHal: Reducing Hallucinations in Multi-modal Large Language Models through Adaptive KV Cache Pruning

Fengyuan Sun, Hui Chen, Xinhao Xu, Dandan Zheng, Jingdong Chen, Jun Zhou, Jungong Han, Guiguang Ding · 2025

Read PruneHal: Reducing Hallucinations in Multi-modal Large Language Models through Adaptive KV Cache Pruning by Fengyuan Sun, Hui Chen, Xinhao Xu, Dandan Zheng, Jingdong Chen, Jun Zhou, Jungong Han, Guiguang Ding (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19183.txt.

Introduction

Fengyuan Sun and seven co-authors present PruneHal, a method to cut hallucinations in multimodal language models by pruning their memory cache. Such models often invent objects in images, and the authors trace much of that failure to attention being spread too thin across redundant visual tokens, so key visual cues get ignored.

By adaptively removing unneeded cached visual entries during generation, the model focuses on the informative tokens instead. Unlike fixes that need extra training data or outside information, pruning adds little cost. Sharper visual attention should make image descriptions more faithful to what is actually shown.

Catalog record

CREATORFengyuan Sun, Hui Chen, Xinhao Xu, Dandan Zheng, Jingdong Chen, Jun Zhou, Jungong Han, Guiguang Ding
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19183

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PruneHal: Reducing Hallucinations in Multi-modal Large Language Models through Adaptive KV Cache Pruning Fengyuan Sun, Hui Chen, Xinhao Xu, Dandan Zheng, Jingdong Chen, Jun Zhou, Jungong Han, Guiguang Ding Abstract While multi-modal large language models (MLLMs) have made significant progress in recent years, the issue of hallucinations remains a major challenge. To mitigate this phenomenon, existing solutions either introduce additional data for further training or incorporate external or internal information during inference. However, these approaches inevitably introduce extra computational costs. In this paper, we observe that hallucinations in MLLMs are strongly associated with insufficient attention allocated to visual tokens. In particular, the presence of redundant visual tokens disperses the model’s attention, preventing it from focusing on the most informative ones. As a result, critical visual cues are often under-attended, which in turn exacerbates the occurrence of hallucinations. Building on this observation, we propose PruneHal , a training-free, simple yet effective method that leverages adaptive KV cache pruning to enhance the model’s focus on critical visual information, thereby mitigating hallucinations. To the best of our knowledge, we are the first to apply token pruning for hallucination mitigation in MLLMs. Notably, our method don’t require additional training and incurs nearly no extra inference cost. Moreover, PruneHal is model-agnostic and can be seamlessly integrated with different decoding strategies, including those specifically designed for hallucination mitigation. We evaluate PruneHal on several widely used hallucination evaluation benchmarks using four mainstream MLLMs, achieving robust and outstanding results that highlight the effectiveness and superiority of our method. Our code will be publicly available. 1 Introduction Recent advancements in multi-modal large language models (MLLMs) have led to significant breakthroughs, enabling these models to effectively tackle a wide range of complex visual tasks ( Wang et al. 2024d ; Bai et al. 2023 ; Achiam et al. 2023 ; Chen et al. 2024b ; Dai et al. 2023 ; Liu et al. 2023b ; Liu et al. 2024a ; Lu et al. 2024 ; Yao et al. 2024 ) . Despite MLLMs’ strong performance across various visual tasks, their practicality remains limited due to hallucinations. The issues of hallucinations ( Bai et al. 2024 ; Rohrbach et al. 2018 ; Li et al. 2023b ) often cause MLLMs to output content that is inconsistent with visual input, severely undermining their reliability. Several studies have explored mitigating hallucinations in MLLMs from different perspectives. Some works ( Gunjal et al. 2024 ; Zhou et al. 2023 ; Liu et al. 2023a ; Yu et al. 2024 ) focus on further training on specifically designed datasets or alignment to reduce the model’s propensity to generate hallucinated content, typically by leveraging additional annotated data or specialized supervision. In contrast, training-free approaches ( Leng et al. 2024 ; Huang et al. 2024 ; Liu et al. 2024b ; Wang et al. 2024c ; Xu et al. 2025 ) aim to alleviate hallucinations during inference, often by designing specific decoding strategies. Although these methods have demonstrated effectiveness, they incur additional training costs or computational overhead during inference. In this paper, we first note that hallucinations in MLLMs are closely associated with the model’s insufficient attention to visual tokens. Previous works ( Chen et al. 2024a ; Liu et al. 2024b ) have demonstrated that in MLLMs, visual tokens constitute most of the input tokens, yet they receive little attention during the forward pass of the self-attention ( Vaswani et al. 2017 ) . It is reasonable to hypothesize that the generation of hallucinated content may also be associated with this phenomenon. To intuitively validate this, we conduct a qualitative experiment. Specifically, we attempt to directly amplify the visual attentions in attention maps during model inference, immediately before the model generates hallucinated content (details are provided in Appendix. C.1 ). As shown in Fig. 1 , increasing MLLMs’ attention to visual information enables it to generate content that aligns with the image during inference process, thereby reducing hallucinations. Figure 1: An example where the hallucinated word “bed”, which is not presented in the image, is correctly changed to “window”, by amplifying the visual attention. More examples can be found in Appendix. C.3 . To illustrate this quantitatively, we present a comparison between the visual attention in the inference steps that lead to hallucinations and the average visual attention across all inference steps (details are provided in Appendix. C.2 ). As illustrated in Fig. 2 , it is clear that across the shallow, middle, and deep layers of LLaVA-v1.5-7B, the generation of hallucinated contents are strongly associated with lower attention scores on visual tokens (intuitively, most of the blue scatter points lie above the red ones). These evidences reveal that the issue of lower attention scores received by visual tokens is potentially one of the cause of generating hullucinated contents during decoding. (a) Layer 1 (b) Layer 16 (c) Layer 30 Figure 2: Average visual attention in language model of LLaVA-v1.5-7B. Each blue scatter point represents the mean attention scores received by all visual tokens in a caption, averaged over all decoding steps. The red scatter points represent the mean attention values when hallucinated content is generated in captions. Attention scores are averaged over all attention heads. Images and captions are selected from the validation set of MSCOCO 2014. Meanwhile, existing works ( Chen et al. 2024a ; Yang et al. 2025 ) have pointed out that visual tokens in MLLMs exhibit widespread redundancy, with only a small number of key tokens containing critical visual information needed for inference. We reasonably hypothesize that hallucinations in MLLMs are highly correlated with visual tokens’ redundancy . In particular, the redundancy of visual tokens could disperse models’ attention and cause important tokens to be under-attended. This further reduces attention to critical and informative visual tokens, exacerbating the hallucination phenomenon. Based on these insights, we propose PruneHal , a novel framework that leverages KV cache pruning to mitigate hallucinations in MLLMs. Initially, we apply a simple KV cache pruning method that retains the top-K most important visual tokens. We show that this top-K-based approach is both simple and effective, enabling the model to focus on key visual tokens. However, this strategy struggles to determine the optimal time for pruning, often leading to excessive pruning. This, in turn, results in the loss of crucial visual information, negatively impacting the overall performance of the model. To address this, we propose an adaptive pruning strategy that tracks the historical visual attention distribution to prune redundant visual tokens more effectively. This allows PruneHal to achieve a balanced trade-off as inference progresses, mitigating hallucinations while preserving output quality. To verify the effectiveness of the proposed method, we conducted extensive experiments on multiple widely-used MLLMs, including LLaVA ( Liu et al. 2023b ) , InstructBLIP ( Dai et al. 2023 ) , and Qwen-VL ( Bai et al. 2023 ) , across various benchmarks and hallucination metrics. The results demonstrated that PruneHal significantly improves performance on a range of MLLMs, benchmarks and datasets, showing favourable robustness. Additionally, we found that PruneHal is fully compatible with existing decoding strategies designed specifically for hallucination mitigation. Our contribution can be summarized as follows: 1. Our experiments reveal a strong link between hallucinations in MLLMs and the insufficient attention given to

Read the complete text