Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation

Zheng Qi, Chao Shang, Evangelia Spiliopoulou, Nikolaos Pappas · 2025

Read Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation by Zheng Qi, Chao Shang, Evangelia Spiliopoulou, Nikolaos Pappas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22067.txt.

Introduction

Zheng Qi, Chao Shang, Evangelia Spiliopoulou, and Nikolaos Pappas try to stop vision-language models from hallucinating content that neither the picture nor the question supports. Earlier fixes boosted attention to image regions, but attention itself often pools on irrelevant areas, and boosting sight alone can leave the model misreading what the user actually asked.

Their method, GIFT, watches where the model's visual attention moves in a helpful direction while it absorbs the question. Builds a saliency map from those gaze shifts before answering begins. That map then guides a balanced fusion of image and query, strengthening the right regions instead of amplifying whatever already glows. The authors present the approach as a simple correction to a specific failure: models answering from language habits rather than from what they were shown.

Catalog record

CREATORZheng Qi, Chao Shang, Evangelia Spiliopoulou, Nikolaos Pappas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22067

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation Zheng Qi, Chao Shang, Evangelia Spiliopoulou, Nikolaos Pappas Abstract Vision language models (VLMs) often generate hallucination, i.e., content that cannot be substantiated by either textual or visual inputs. Prior work primarily attributes this to over-reliance on linguistic prior knowledge rather than visual inputs. Some methods attempt to mitigate hallucination by amplifying visual token attention proportionally to their attention scores. However, these methods overlook the visual attention sink problem, where attention is frequently misallocated to task-irrelevant visual regions, and neglect cross-modal fusion balance by enhancing only visual attention without adjusting attention to the user query. This can result in amplifying incorrect areas while failing to properly interpret the user query. To address these challenges, we propose a simple yet effective method called G aze Sh i ft-Guided Cross-modal F usion Enhancemen t ( GIFT ). GIFT pre-computes a holistic visual saliency map by tracking positive changes in visual attention, or “gaze shifts” , during user query comprehension, and leverages this map to amplify attention to both salient visual information and the user query at each decoding step. This reduces the impact of visual attention sink, as irrelevant tokens exhibit minimal shifts, while ensuring balanced cross-modal fusion for well-integrated representation. Extensive experiments show that GIFT effectively mitigates hallucination in VLMs across both generative and classification tasks, achieving up to 20.7% improvement over greedy decoding, while maintaining general vision-language performance with low computational overhead. 1 Introduction Vision language models (VLMs) ( Li et al. 2023c ; Liu et al. 2023b ; Zhu et al. 2023 ; Liu et al. 2024b ; Hurst et al. 2024 ; Wang et al. 2024b ; Bai et al. 2025 ) have recently achieved remarkable progress on tasks that require joint reasoning over textual and visual information, such as visual question answering, visual reasoning, and image captioning. Despite these advances, VLMs remain prone to generating hallucination, i.e., content that cannot be substantiated by either textual or visual inputs ( Liu et al. 2024a ) . This issue poses serious challenges, particularly in high-stakes domains such as biomedicine ( Li et al. 2023b ; Chen et al. 2024b ) , autonomous driving ( Wang et al. 2023 ; Li et al. 2025 ) , and robotics ( Chen et al. 2024a ; Li et al. 2024 ) , where factual accuracy and reliability are critical for safe and effective operation. Recent analyses suggest that these failures are primarily due to vision language models (VLMs) over-relying on linguistic prior knowledge while under-utilizing visual inputs ( Wang et al. 2024a ; Zhang et al. 2024 ) . To mitigate this, inference-time interventions have been proposed to enhance visual grounding by highlighting visual signals based on visual saliency, i.e., the relevance of specific visual regions to the task at hand. For instance, Yin et al. 2025 enhances attention allocated to visual tokens during decoding in proportion to their attention scores. While this approach strengthens the contribution of visual information, it does not account for the balance between visual and query signals during cross-modal fusion. Consequently, the model may attend to relevant regions but misinterpret the query, forming inaccurate integrated representations. Moreover, this approach does not address the issue of visual attention sink ( Kang et al. 2025 ) , where attention is persistently misallocated to irrelevant visual tokens, potentially amplifying incorrect regions throughout generation. To mitigate the issue of visual attention sink, existing methods recalibrate visual token attention ( Kang et al. 2025 ; Zhu et al. 2025b ) , but they fail to address the broader problem of low overall visual contribution during decoding. To address these limitations, we propose an inference-time hallucination mitigation method for VLMs called G aze Sh i ft-Guided Cross-modal F usion Enhancemen t ( GIFT ). Drawing inspiration from human vision, we hypothesize that VLMs, like humans, dynamically shift their “visual gaze” when processing information-rich words in a user query. By tracking positive changes in visual token attention, i.e., “gaze shifts”, over these information-rich query tokens at the layer exhibiting the largest positive change, GIFT pre-computes a holistic visual saliency map that captures task-relevant regions prior to decoding, requiring pre-filling only up to that layer. This mechanism also mitigates the impact of visual attention sink, as irrelevant regions exhibit minimal or no attention shift. During decoding, GIFT leverages this saliency map to proportionally amplify attention to salient visual tokens in critical cross-modal fusion layers, where the model attends strongly to both visual and query tokens. In contrast to Yin et al. 2025 , which only increases attention to visual tokens, GIFT also adjusts attention to query tokens based on the overall visual attention amplification ratio, maintaining cross-modal balance and forming well-integrated representations. In summary, our main contributions are three-fold: • We introduce a novel mechanism that captures a holistic view of salient visual regions while effectively mitigating the visual attention sink problem. This mechanism pre-computes a task-relevant visual saliency map prior to decoding by tracking positive shifts in visual attention, i.e., “gaze shifts”, as the VLM processes information-rich words in a user query. • We propose GIFT, a lightweight inference-time hallucination mitigation method that leverages the precomputed saliency map to guide visual attention enhancement while proportionally scaling attention to query tokens to preserve cross-modal fusion balance. • We show that GIFT consistently mitigates hallucination across VLM architectures and model sizes, achieving gains of up to 20.7% on CHAIR, 15.9% on MMHal-Bench, and 3.0% improvement on POPE, while preserving general vision-language performance with low computational overhead. Extensive ablation studies further validate the contribution of each component. Figure 1: Examples of visual saliency maps from LLaVA 1.5 7B. The vanilla method ( Static ) averages visual token attention over all query tokens, while the proposed method ( Shift ) averages positive changes in visual token attention over information-rich query tokens. Shift more effectively highlights task-relevant visual regions and mitigates the impact of visual attention sink, where irrelevant regions receive erroneously high saliency scores. 2 Related Work VLM Hallucination Mitigation. A key cause of hallucination in VLMs is over-reliance on linguistic prior knowledge rather than visual inputs ( Wang et al. 2024a ; Zhang et al. 2024 ) . To address this, training-based approaches have introduced specialized learnable modules ( Zhao et al. 2024 ) or curated data augmentations ( Liu et al. 2023a ; Pi et al. 2024 ; Chen et al. 2025 ) to encourage stronger reliance on visual features. While effective, these methods often suffer from high computational costs and limited scalability. Another line of work focuses on inference-time mitigation, which can be broadly categorized into three types: (1) Contrastive decoding ( Leng et al. 2024 ; Liu et al. 2024c ; Huo et al. 2024 ; Wang et al. 2025 ; Zhu et al. 2025a ) , which reduces over-reliance on knowledge priors by contrasting the output distributions of two inputs, one with the original visual inputs and one with perturbed or absent visual inputs. However, this approach incurs significant computational overhead due to the need for generating counterpart outputs. Our method instead strengthens visual contributions directly in intermediate layers, eliminating the need for generating alternatives. (2) Visual input modification, which manipulates the

Read the complete text