Hallucination-aware Intermediate Representation Edit in Large Vision-Language Models

Wei Suo, Hanzu Zhang, Lijun Zhang, Ji Ma, Peng Wang & Yanning Zhang · 2026

Read Hallucination-aware Intermediate Representation Edit in Large Vision-Language Models by Wei Suo, Hanzu Zhang, Lijun Zhang, Ji Ma, Peng Wang & Yanning Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29405.txt.

Introduction

Suo and co-authors present HIRE, a method that edits vision-language models to cut hallucinations. These models sometimes state facts that clash with the image. Retraining fixes this at high cost, and contrastive decoding doubles inference work. HIRE instead finds the internal features tied to hallucination. It then edits those features on the fly to remove the false content. The added compute cost stays small. Benchmarks show top results, the authors report. The method also gives strong control over hallucinations.

Catalog record

CREATORWei Suo, Hanzu Zhang, Lijun Zhang, Ji Ma, Peng Wang & Yanning Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48196
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.29405

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hallucination-aware Intermediate Representation Edit in Large Vision-Language Models Wei Suo, Hanzu Zhang, Lijun Zhang, Ji Ma, Peng Wang & Yanning Zhang Abstract Large Vision-Language Models have demonstrated exceptional performance in multimodal reasoning and complex scene understanding. However, these models still face significant hallucination issues, where outputs contradict visual facts. Recent research on hallucination mitigation has focused on retraining methods and Contrastive Decoding (CD) methods. While both methods perform well, retraining methods require substantial training resources, and CD methods introduce dual inference overhead. These factors hinder their practical applicability. To address the above issue, we propose a framework for dynamically detecting hallucination representations and performing hallucination-eliminating edits on these representations. With minimal additional computational cost, we achieve state-of-the-art performance on existing benchmarks. Extensive experiments demonstrate the effectiveness of our approach, highlighting its efficient and robust hallucination elimination capability and its powerful controllability over hallucinations. Code is available at https://github.com/ASGO-MM/HIRE . 1 Introduction Large Vision-Language Models (LVLMs) ( 5 ; 13 ; 35 ; 65 ) have made remarkable advancements in recent years, demonstrating the ability to generate context-aware language outputs based on visual understanding. By integrating visual information into Large Language Models (LLMs), LVLMs have demonstrated strong capabilities in tasks such as multimodal reasoning ( 38 ) and complex scene understanding ( 39 ) . However, LVLMs currently face the issue of hallucination, where the generated responses contradict the actual visual content. Recently, many approaches ( 27 ; 34 ; 61 ; 63 ; 40 ) have been proposed to mitigate hallucinations in LVLMs, which can be broadly categorized into retraining methods and Contrastive Decoding (CD) methods. As illustrated in Fig. 1 (a), retraining methods aim to alleviate hallucinations by constructing specialized datasets that target hallucination phenomena and introducing new training paradigms ( 23 ; 18 ; 37 ) . In contrast, Fig. 1 (b) illustrates the paradigm of CD methods, which mitigate hallucination during inference. The method can work by contrasting the output token probabilities of the original response with those of a weakened variant that is more susceptible to hallucination. CD methods effectively reduce hallucinations without requiring retraining or additional data ( 27 ; 22 ; 41 ) . Although both methods have shown promising results in hallucination mitigation, they still face some notable limitations: 1) Retraining methods require substantial data collection and computational resources for partial or full model fine-tuning. However, both the high cost of dataset construction and the heavy computational burden ( i.e., they need to retrain the LVLM) limit the practical applicability of these methods. 2) Contrastive decoding methods, while more efficient in design, introduce high computational overhead. They require two forward passes per inference—one for generating the original output and another for producing a deliberately weakened variant prone to hallucination—which significantly increases latency. Moreover, they uniformly adjust the probability of all tokens without distinguishing whether a token is actually prone to hallucination. For instance, common tokens like “in,” “on,” or “from” rarely cause such issues. The one-size-fits-all adjustment leads to unnecessary computation and may harm output coherence. 3) Finally, hallucinations are not inherently harmful in tasks like creative writing ( 25 ) , where appropriate hallucination can enhance expressiveness. However, most existing methods lack the ability to control the degree of hallucination, limiting their applicability in broader use cases where flexible generation is desired. Recent studies on LLMs ( 3 ; 10 ) have shown that a model’s internal representations encode cues about the truthfulness of statements, enabling hallucination detection without external knowledge. Building on these insights, recent research on LVLMs ( 31 ; 16 ) demonstrates that similar techniques can be applied to multimodal settings, where hallucination detection can be effectively performed through simple classifiers trained on intermediate features. This suggests that authentic and hallucinatory features in LVLMs are clearly separable in the latent space. This raises a natural question: Can we leverage this inherent separation to effectively eliminate hallucinations? Based on the above discussion, we propose HIRE ( H allucination-aware I ntermediate R epresentation E dit), a feature-editing framework that dynamically detects and mitigates hallucinations at the intermediate representation level. As shown in Fig. 1 (c), instead of costly retraining, we propose to mitigate hallucinations at the feature level without modifying the LVLM’s weights. Specifically, we first introduce a module, Editor , that learns to identify and isolate hallucination-related components by modeling both semantic invariance and hallucinatory differences between authentic and hallucinated responses, while preserving the underlying semantic information. Moreover, since uniformly editing all tokens leads to unnecessary computational overhead, we design a lightweight Router to selectively edit only tokens with high hallucination risk. Lastly, to enable more flexible hallucination control, we introduce a Hallucination Regulator , which allows dynamic control over hallucination levels via a simple hyperparameter. Overall, our main contributions are: (1) We introduce a new paradigm for hallucination mitigation, which reduces fabricated content by modifying representations without retraining models or doubling inference costs. (2) We propose a new framework, HIRE, that dynamically detects and edits intermediate representations with high hallucination. Meanwhile, our method can control the degree of hallucinations to adapt to different user requirements by adjusting the editing intensity. (3) We validate the effectiveness of the proposed method through extensive experiments and achieve state-of-the-art performance on three benchmarks. Figure 1: Comparison of mainstream hallucination mitigation paradigms. (a) Retraining-based methods: Constructing hallucination-specific datasets and training frameworks. (b) Contrastive decoding methods: Comparing the original probability distribution with a perturbed one. (c) Our method: Editing intermediate representations of LVLMs. 2 Related Work 2.1 Large Vision-Language Models Building on the success of Large Language Models (LLMs) ( 4 ; 8 ; 12 ; 51 ) and cross-modal learning ( 45 ; 14 ) , Large Vision-Language Models (LVLMs) achieve breakthrough performance by integrating visual perception and language generation capabilities, excelling in tasks such as image captioning ( 19 ) , visual question answering ( 2 ) , and multimodal reasoning ( 38 ) . A typical LVLM architecture comprises three core components: a visual encoder for hierarchically image feature extraction like CLIP ( 45 ) ; a cross-modal alignment module implemented via linear projection layers or advanced architecture like Q-Former ( 29 ) ; a large language model fine-tuned through instruction tuning for context-aware text generation. Despite these advancements, LVLMs inherently suffer from hallucination, where the generated text exhibits semantic inconsistencies with the input visual content. 2.2 Mitigating Hallucinations in LVLMs Recently, multiple strategies have been proposed to address hallucination in LVLMs, which can be broadly categorized into retraining methods and Contrastive Decoding (CD) methods. Retraining methods retrain LVLMs using carefully constructed datasets and training paradigms specifically designed to address hallucinations. HA-DPO ( 63 ) fine-tunes LVLMs using style-consistent hallucination sample pairs to promote non-hallucinatory outputs. HDPO ( 18 ) further targets diverse causes of hallucinations by constructing specialized preference pairs for visual distraction, long-context generation, and multimodal conflicts. In contrast, CD methods mitigate hallucinations by comparing the output distributions from the original and perturbed inputs, without requiring any model parameter updates. VCD ( 27 ) introduces noise into visual inputs to amplify hallucinations and suppresses them by contrasting perturbed and original token distributions. SID ( 22 ) retains only the least important visual tokens after shallow processing to amplify vision-text association

Read the complete text