INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling
Xin Dong, Shichao Dong, Jin Wang, Jing Huang, Li Zhou, Zenghui Sun, Lihua Jing, Jinsong Lan, Xiaoyong Zhu, Bo Zheng · 2025
Read INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling by Xin Dong, Shichao Dong, Jin Wang, Jing Huang, Li Zhou, Zenghui Sun, Lihua Jing, Jinsong Lan, Xiaoyong Zhu, Bo Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05056.txt.
Introduction
Xin Dong and co-authors present INTER, a 2025 method to cut false details in vision-language models. Such models can describe things that are not in the picture, a failure called hallucination. Humans rarely do this, perhaps because they weigh how sight and sound interact before speaking.
Tests showed the models also hold some of that cross-signal understanding, just more weakly. INTER guides the model's word choices to reuse that understanding at answer time, with no extra training data. On six benchmarks covering questions and captions, it improved five top models by up to 3.4 percent over the best rival decoding method.
Catalog record
| CREATOR | Xin Dong, Shichao Dong, Jin Wang, Jing Huang, Li Zhou, Zenghui Sun, Lihua Jing, Jinsong Lan, Xiaoyong Zhu, Bo Zheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05056
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling Xin Dong, Shichao Dong, Jin Wang, Jing Huang, Li Zhou, Zenghui Sun, Lihua Jing, Jinsong Lan, Xiaoyong Zhu, Bo Zheng Abstract Hallucinations in large vision-language models (LVLMs) pose significant challenges for real-world applications, as LVLMs may generate responses that appear plausible yet remain inconsistent with the associated visual content. This issue rarely occurs in human cognition. We argue that this discrepancy arises from humans’ ability to effectively leverage multimodal interaction information in data samples. Specifically, humans typically first gather multimodal information, analyze the interactions across modalities for understanding, and then express their understanding through language. Motivated by this observation, we conduct extensive experiments on popular LVLMs and obtained insights that surprisingly reveal human-like, though less pronounced, cognitive behavior of LVLMs on multimodal samples. Building on these findings, we further propose INTER : Inter action Guidance Sampling, a novel training-free algorithm that mitigate hallucinations without requiring additional data. Specifically, INTER explicitly guides LVLMs to effectively reapply their understanding of multimodal interaction information when generating responses, thereby reducing potential hallucinations. On six benchmarks including VQA and image captioning tasks, INTER achieves an average improvement of up to 3.4% on five LVLMs compared to the state-of-the-art decoding strategy. The codes are released on Github . 1 1 1 ∗ Equal Contribution. 2 2 2 † Corresponding authors. 1 Introduction Figure 1 : Comparisons with existing decoding strategies on the state-of-the-art LVLM InternVL2.5-MPO [ 10 ] . Our approach INTER achieved optimal results across six benchmarks. Besides, the detailed output of INTER in a case of complex inputs is presented. The hallucinated words are highlighted in red. Large Vision-Language Models (LVLMs) have demonstrated remarkable versatility across a wide range of tasks, from image captioning to complex reasoning [ 3 , 31 , 63 , 10 , 38 , 13 ] . These models blend visual and textual information, enhancing our ability to interpret and interact with the world. However, LVLMs experience hallucinations that hinder their applications, which means that responses are not aligned with the given input [ 25 , 16 , 30 , 11 , 50 , 55 , 8 ] . Previous methods made attempts to address this issue with additional training and fine-grained data [ 36 , 65 , 67 , 29 , 50 , 66 ] , but such approaches often demand substantial human effort and computational resources. In parallel, other methods were proposed to explore efficient training-free methods to mitigate hallucinations in LVLMs [ 16 , 25 , 17 , 11 , 40 , 72 , 1 , 42 ] . These methods aimed to enhance the model’s focus on the input visual or textual information by adjusting the logit distribution during the autoregressive generation process, thereby reinforcing the link between the output and uni-modal information. Despite these efforts, the role of multimodal interactions information in shaping LVLMs’ hallucinated responses remains largely underexplored, even though it naturally plays a critical role in human cognition for accurate reasoning. Specifically, when presented with a textual prompt and an associated image, humans typically first gather multimodal information, analyze the interactions between modalities to form a conceptual understanding, and then provide textual responses based on this understanding. Motivated by this intuition, we propose to first investigate whether LVLMs exhibit similar cognitive behavior when processing multimodal data, aiming to uncover the potential causes behind their hallucination issues. To this end, we designed several metrics based on the Harsanyi dividend [ 22 ] to explicitly quantify the influence of image-text multimodal interactions in LVLMs’ responses. The Harsanyi dividend was originally proposed in game theory, which measures the interactions among different players. Associated with the Shapley value [ 48 ] , the Harsanyi dividend theoretically satisfies the efficiency , linearity , dummy , symmetry axioms, which further enhances its trustworthiness on the analyses of LVLMs [ 44 , 45 , 56 ] . In this way, we derive the following key insights through extensive analysis: • Insight 1 : LVLMs implicitly capture multimodal interactions from input samples and leverage such interactions for decision-making to some extent. • Insight 2 : LVLMs exhibit a tendency to primarily apply their understanding of multimodal interactions to the generation of a few key tokens, rather than uniformly across all response tokens. • Insight 3 : The understanding of multimodal interactions in LVLMs positively influences the quality of generated responses, with stronger interactions leading to greater accuracy. To our surprise, the above findings suggest that LVLMs possess a human-like—albeit less pronounced—understanding of multimodal interactions, which inspired us to design efficient methods for reducing hallucinated responses of LVLMs. To mitigate the hallucinations of LVLMs, we propose a simple yet effective method named INTER : Inter action Guidance Sampling, which aims to reapply their captured multimodal interaction understanding ( cf . Insight 1 ) more accurately and effectively in LVLMs’ reasoning process. Specifically, we first design a variance-based filtering module termed as the Interactive Guided Locator in INTER to automatically detect key tokens that significantly contribute to the accuracy of responses ( cf . Insight 2 ). After that, we design the Interaction Probability Modifier in INTER which guides the sampling of these key tokens to rely more on multimodal interactions ( cf . Insight 3 ). In this way, LVLMs can reduce potential hallucinations in their responses, improving their overall performance. As shown in Fig. 1 , our INTER achieved better performance on the state-of-the-art LVLM InternVL2.5-MPO (8B) [ 10 ] across six widely-used benchmarks [ 33 , 39 , 7 , 19 , 46 , 37 ] . In experiments, we also verified the effectiveness of INTER on other popular LVLMs [ 13 , 38 , 63 , 3 , 9 ] . Specifically, INTER boosted the performance of state-of-the-art decoding strategies by an average of 4.1% and 2.6% on CHAIR [ 46 ] and MME [ 19 ] benchmark, respectively. Our contributions can be summarized as follows: • From a novel game-theoretic view, we propose to investigate the roles of multimodal interactions in shaping LVLMs’ hallucinated responses. Through extensive analyses, we obtain several new insights accordingly. • We propose Interaction Guidance Sampling (INTER), a novel plug-and-play sampling rectification method for eliminating hallucinations, that accurately and effectively guides LVLMs to explicitly reapply their understanding of multimodal interactions in responses. • Extensive experiments demonstrate that INTER successfully improved the performance of various LVLMs upon six benchmarks without requiring additional training, outperforming state-of-the-art methods by a large margin. 2 Related Work 2.1 Large Vision-Language Models (LVLMs) The rapid advancement of Large Vision-Language Models (LVLMs) has become a pivotal area in artificial intelligence research. These models aim to generate contextually grounded responses through multimodal understanding. Modern architectures typically adapt existing Large Language Models (LLMs) as text decoder for generation. Prominent examples include LLaVA-v1.5 [ 38 ] and InstructBLIP [ 13 ] , both built on Vicuna 7B [ 12 ] , as well as Qwen-VL [ 3 ] and mPLUG-Owl2 [ 63 ] , which utilize Qwen 7B [ 2 ] and LLaMA 2 7B [ 54 ] respectively. Scalability efforts are exemplified by InternVL2 [ 10 ] , which explores parameter configurations from 1B to 108B. Despite these advancements, a critical challenge