Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations
Boxu Chen, Ziwei Zheng *, Le Yang, Zeyu Geng · 2025
Read Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations by Boxu Chen, Ziwei Zheng *, Le Yang, Zeyu Geng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17812.txt.
Introduction
Boxu Chen and six co-authors build VaLSe, a frame that first reads how a sight-language model uses an image, then steers it away from made-up things. Their paper is from two thousand twenty-five.
Such models can name things that are not in the image. Past fixes cut that flaw but left its cause dim. VaLSe draws maps that show how each part of the image sways each word the model writes. Those maps bare where the model's eye truly rests. The team then nudges the model's inner states toward the parts that count. That pulls its words back to the scene and cuts the false things.
Catalog record
| CREATOR | Boxu Chen, Ziwei Zheng *, Le Yang, Zeyu Geng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17812
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations Boxu Chen, Ziwei Zheng *, Le Yang, Zeyu Geng Abstract Large Vision-Language Models (LVLMs) have achieved remarkable success but continue to struggle with object hallucination (OH), generating outputs inconsistent with visual inputs. While previous work has proposed methods to reduce OH, the visual decision-making mechanisms that lead to hallucinations remain poorly understood. In this paper, we propose VaLSe, a Vision-aware Latent Steering framework that adopts an interpretation-then-mitigation strategy to address OH in LVLMs. By tackling dual challenges of modeling complex vision-language interactions and eliminating spurious activation artifacts, VaLSe can generate visual contribution maps that trace how specific visual inputs influence individual output tokens. These maps reveal the model’s vision-aware focus regions, which are then used to perform latent space steering, realigning internal representations toward semantically relevant content and reducing hallucinated outputs. Extensive experiments demonstrate that VaLSe is a powerful interpretability tool and an effective method for enhancing model robustness against OH across multiple benchmarks. Furthermore, our analysis uncovers limitations in existing OH evaluation metrics, underscoring the need for more nuanced, interpretable, and visually grounded OH benchmarks in future work. Code is available at: https://github.com/Ziwei-Zheng/VaLSe . 1 Introduction Recent advances in large language models (LLMs) [ 1 , 2 , 3 ] have accelerated the development of Large Vision-Language Models (LVLMs), such as LLaVA [ 4 , 5 ] , InstructBLIP [ 6 ] , MiniGPT-4 [ 7 ] , and Qwen2-VL [ 8 , 9 ] . However, LVLMs are prone to object hallucination [ 10 , 11 , 12 , 13 ] , often generating outputs that are inconsistent with visual inputs, which raises serious concerns about the reliability and safety of LVLMs. Recent efforts to mitigate hallucinations have explored a range of strategies, including end-to-end fine-tuning [ 14 , 15 , 16 ] , post-processing of model outputs [ 17 , 18 , 13 , 19 ] , and latent feature steering [ 11 , 20 , 21 ] , all of which have shown promising results on open-source LVLMs. Nevertheless, a critical limitation remains [ 10 ] : there still lacks an effective method to trace how visual inputs influence the decision-making processes of LVLMs. As a result, the underlying mechanisms of hallucination and the factors triggering it remain poorly understood. Interpreting open-ended responses from LVLMs introduces several key challenges. (1) Complex vision-language interaction: The intricate alignment between vision encoders and LLMs creates difficulty in disentangling modality contributions and leads to poor interpretable results [ 22 , 23 ] . (2) Activation artifacts: Recent studies [ 24 , 25 , 26 ] reveal that some neurons produce disproportionately high activations regardless of the input, which distort visualization results (see Figure 1 (a)). These challenges hinder the development of reliable interpretation methods for LVLMs, making it difficult to explore why a hallucinated word is generated and to determine whether a correct prediction is a correct answer or a “guessing one”. Figure 1: The proposed VaLSe can effectively (a) eliminate artifacts and provide high quality visualization results, and then (b) mitigate OH by vision-aware latent steering. With the ability of mitigating OH, VaLSe can further provide in-depth analysis of (c) how a word token is generated based on visual information and (d) inferring why a hallucinated words is generated. To address these issues, we propose VaLSe, a novel V ision- a ware L atent S t e ering framework for LVLMs. Figure 1 provides an overview of VaLSe. To trace the influence of visual inputs on output tokens, VaLSe models complex vision-language interactions via visual contribution maps and eliminates artifact activations by contrasting targeted tokens with non-semantic special tokens, resulting in higher-quality, interpretable visualization of LVLM’s output. Moreover, with the interpretable results, VaLSe can reinforce the LVLM’s attention to semantically relevant regions in the image by latent feature steering using the generated visual contribution maps, enhancing its visual grounding and effectively mitigating OH (shown in Figure 1 (b)). Through comprehensive experiments, we demonstrate that VaLSe effectively mitigates OH without compromising general ability. More importantly, VaLSe offers a new perspective for studying OH by providing fine-grained interpretability into the model’s decision-making process. As illustrated in the visualization results in Figure 1 , which highlight the focus regions of LLaVA-1.5 during response generation, benchmark ground-truth answers alone are insufficient for determining whether hallucination has occurred. On one hand, a model may produce a correct response while attending to irrelevant image regions, indicating it relied on language priors rather than visual cues. For example, in the bottom-left panel of Figure 1 (c), the model correctly predicts the word “two” without attending to the correct relevant visual evidence. On the other hand, as revealed through visualization (Figure 1 (d)), a word flagged as hallucinated by metrics (e.g., the CHAIR metric [ 27 ] ) may actually be a visually grounded and accurate description. These findings highlight not only the importance of understanding the internal mechanisms behind hallucinated outputs but also the need for more sophisticated and comprehensive benchmarks to evaluate OH in LVLMs. The main contributions are summarized as follows: • We propose a novel vision-aware latent steering method that follows an interpretation-then-mitigation strategy, enabling internal analysis of the generation process behind hallucinated words and effectively reducing OH in LVLMs. • VaLSe generates high-quality visual contribution maps across different LVLMs, enabling deeper analysis of their decision-making processes. Our analysis reveals limitations in existing OH evaluation metrics, highlighting the need for more nuanced visually grounded assessment methods. • Experiments demonstrate the effectiveness of VaLSe in OH mitigation. Moreover, both qualitative and quantitative evaluations demonstrate the superiority of our method in visualization for LVLMs. 2 Related Work Large Visual-Language Models (LVLMs) Based on successes of LLMs, large vision-language models (LVLMs) have made significant progress in recent years. These models typically integrate a vision encoder with an LLM via fusion modules, such as a linear projection layer [ 4 ] or a Q-former [ 7 ] . Recent LVLMs, such as LLaVA [ 4 , 5 ] , MiniGPT-4 [ 7 ] , mPLUG-Owl [ 28 , 29 ] , Qwen-VL [ 1 , 8 ] , LLaVA-Phi [ 30 ] and DeepSeek-VL [ 31 ] have shown to be capable for complex image understanding and reasoning. Despite these advancements, modern LVLMs continue to face significant security and robustness challenges, notably object hallucination [ 10 ] . Mitigation of Object Hallucination Various approaches have been proposed to address this issue. Given that hallucinations may stem from data biases and the knowledge gap between visual and linguistic information, recent studies have explored fine-tuning LVLMs for robustness [ 14 , 32 ] , cross-modality matching [ 15 , 16 ] , and preference alignment [ 33 , 34 ] . To avoid the high cost of fine-tuning, post-processing strategies have been developed to revise model outputs using external tools, such as LURE [ 13 ] and visual-guided refiners [ 35 , 36 , 19 ] . Other approaches aim to debias strong language priors during decoding [ 17 , 37 , 18 , 38 , 39 , 40 ] , while feature-steering methods [ 11 , 21 , 41 ] learn latent shift directions to adjust