Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention
Jianfei Zhao, Feng Zhang, Xin Sun, Chong Feng, Zhixing Tan · 2025
Read Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention by Jianfei Zhao, Feng Zhang, Xin Sun, Chong Feng, Zhixing Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20032.txt.
Introduction
Jianfei Zhao, Feng Zhang, Xin Sun, Chong Feng, and Zhixing Tan present Vision-Guided Attention, a training-free cure for hallucination in multimodal models. Such models often invent objects because their visual attention localizes poorly. The authors notice the models can extract correct visual meaning from image tokens, yet fail to use that strength during later reasoning.
Their method first builds precise visual grounding from the semantic content of the visual tokens, then guides the model's focus toward the right regions. For captioning, the guidance updates during generation by suppressing regions already described. Each token needs only one forward pass, so the delay is negligible, and the method works with fast attention kernels. Benchmarks show top dehallucination results across several models.
Catalog record
| CREATOR | Jianfei Zhao, Feng Zhang, Xin Sun, Chong Feng, Zhixing Tan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-20032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention Jianfei Zhao, Feng Zhang, Xin Sun, Chong Feng, Zhixing Tan Abstract Visual attention serves as the primary mechanism through which MLLMs interpret visual information; however, its limited localization capability often leads to hallucinations. We observe that although MLLMs can accurately extract visual semantics from visual tokens, they fail to fully leverage this advantage during subsequent inference. To address this limitation, we propose V ision- G uided A ttention (VGA), a training-free method that first constructs precise visual grounding by exploiting the semantic content of visual tokens, and then uses this grounding to guide the model’s focus toward relevant visual regions. In image captioning, VGA further refines this guidance dynamically during generation by suppressing regions that have already been described. In VGA, each token undergoes only a single forward pass, introducing a negligible latency overhead of just 4.36%. In addition, VGA is fully compatible with efficient attention implementations such as FlashAttention. Extensive experiments across diverse MLLMs and multiple hallucination benchmarks demonstrate that VGA achieves state-of-the-art dehallucination performance. Further analysis confirms that explicit visual guidance plays a crucial role in enhancing the visual understanding capabilities of MLLMs. 1 1 1 The code is available at: https://github.com/beta-nlp/VGA . 1 Introduction Multimodal Large Language Models (MLLMs) extend the capabilities of large language models (LLMs) into the visual domain, enabling tasks such as visual question answering and image captioning. However, their visual understanding remains limited, as they often exhibit hallucinations—producing outputs that contradict the actual visual content. This hallucination issue significantly affects the usability and reliability of MLLMs. Figure 1 : A diagram of vision‑guided attention. We first leverage the semantic features embedded in visual tokens to establish visual grounding, and then guide the model’s visual attention toward the relevant image regions. Various approaches have been proposed to mitigate hallucinations, such as constructing targeted datasets 21 or designing specialized loss functions 24 ; 17 . However, the rapid iteration of model architectures has led to diminishing returns for training-based methods. In contrast, training-free hallucination mitigation techniques offer greater practical value due to their plug-and-play nature and strong generalization capability. In particular, a line of research focuses on optimizing visual attention—a mechanism that serves as the primary conduit through which multimodal large language models (MLLMs) interpret visual information 31 ; 28 —providing one of the most direct pathways to reducing hallucinations. Previous studies 16 ; 27 ; 34 ; 25 have underscored the essential role of visual attention in MLLMs and shown that strengthening visual attention can substantially mitigate hallucination. However, these approaches depend on the inherent quality of visual attention itself, which is fundamentally limited in its ability to accurately localize critical visual regions 6 ; 8 , ultimately resulting in suboptimal performance. While employing external tools 35 or performing additional forward passes 3 is feasible, these approaches introduce extra computational overhead and reduce practical usability. Furthermore, because they rely on optimization based on attention distributions, they are incompatible with FlashAttention 4 , which does not expose explicit attention weights, thereby restricting their applicability. Consequently, optimizing visual attention remains a significant challenge. To our surprise, we found that the model can accurately extract semantic features from visual tokens, enabling token-level visual grounding. Based on this observation, we propose Vision-Guided Attention (VGA), a novel strategy for directing visual attention by leveraging the semantic features of visual tokens to identify the most informative ones and allocate greater attention to them. Figure 1 illustrates the overall framework of VGA. More importantly, VGA does not require computing attention weights for individual tokens, making it fully compatible with Flash Attention. In short, the our main contributions are: • We introduce Visual Semantic Confidence (VSC), which leverages the semantic features of visual tokens to achieve precise visual grounding. • We propose Vision-Guided Attention (VGA), which uses the visual grounding produced by VSC to guide visual attention, enabling the MLLM to focus on the most relevant visual tokens. • VGA is compatible with Flash Attention, and each token requires only a single forward pass, adding just 4.36% inference latency (measured on LLaVA‑7B). • Extensive experiments across diverse benchmarks and multiple MLLMs verify the effectiveness and generalization capability of VGA. Further analysis shows that VSC provides accurate visual grounding; moreover, by leveraging this grounding to guide visual attention, VGA effectively mitigates hallucinations in MLLMs. 2 Related Work Constructing high-quality datasets 2 ; 21 or designing hallucination-targeted optimization objectives 24 ; 17 can directly improve model reliability, but the substantial training cost makes these approaches less cost‑effective. As a result, training‑free dehallucination methods have gained increasing attention. Integrating well‑established external tools can effectively enhance the reliability of MLLMs 29 ; 35 , but this also increases the overall system cost. Consequently, optimizing the model’s internal representations has become a more appealing alternative. Such methods mitigate hallucinations by adjusting internal components—such as attention weights, logits, or hidden states—to suppress hallucinatory signals. A line of contrastive decoding–based approaches 9 ; 33 ; 30 ; 12 directly reduces hallucinations in MLLMs by attenuating hallucinatory features in the logits. However, these methods generally require an additional forward pass to activate or expose such features. Other studies 23 ; 37 ; 10 edit semantic features in the embedding space to enhance vision‑related representations; yet the high dimensionality and complexity of this space make such vector manipulations prone to introducing noise. Attention mechanisms serve as the primary means by which MLLMs interpret visual features, and the distribution of attention weights offers an intuitive, interpretable reflection of the model’s visual behavior. Several studies 16 ; 36 have directly increased the model’s attention to visual content, thereby empirically validating the critical role of visual attention in MLLMs. Other works 26 ; 32 ; 5 have investigated functional heterogeneity among attention heads and sought to coordinate their roles accordingly. Additionally, some approaches 8 ; 27 attribute hallucinations to anomalous attention patterns and propose targeted refinements to address them. While these methods effectively mitigate hallucinations by optimizing visual attention, they largely overlook the inherent variability among visual tokens—specifically, which tokens are most worthy of the model’s attention. Some approaches 25 ; 34 leverage historical attention distributions to assess the importance of visual tokens and use this information to refine the current attention distribution. However, these methods rely entirely on the model’s own visual behavior possessing significant limitations. More importantly, optimizing visual attention typically requires access to the full attention distribution in advance, making such approaches incompatible with attention acceleration techniques like Flash Attention 4 . We propose Vision-Guided Attention, which first leverages the semantic features of visual tokens to identify informative tokens and then guide the model’s attention to them. Moreover, our method does not rely on attention weights, making it fully compatible with attention acceleration techniques. 3 Visual Semantic Confidence Figure 2 : Top-3 tokens in visual logits from LLaVA-7B. In MLLMs, an input image is first divided into patches, producing visual tokens v_{1},\cdots,v_{m} . A vision encoder followed by a projector maps each token to a visual embedding E_{v_{i}} . These visual embeddings are then concatenated with textual embeddings to form a single sequence, which is fed into the base LLM. The final hidden states are passed through an unembedding