When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models

Francesco Ortu, Zhijing Jin, Diego Doimo, Alberto Cazzaniga, [10px] University of Trieste AREA Science Park · 2025

Read When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models by Francesco Ortu, Zhijing Jin, Diego Doimo, Alberto Cazzaniga, [10px] University of Trieste AREA Science Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13868.txt.

Introduction

Francesco Ortu and co-authors catch vision-language models in the act of choosing what they see over what they know in a 2025 paper. Shown a picture that breaks common sense, such a model must pick a side: its stored knowledge, or its eyes. Picking wrong in either direction breeds hallucination and shaky answers.

Their test set, WHOOPS-AHA!, poses counterfactual picture questions built to force that clash. Reading the model's internal scores, the team found a small set of attention heads that referee the fight. Nudging those heads tips the verdict toward memory or toward the image, on demand. The same heads also point at the exact image regions driving a visual override, more precisely than gradient-based maps. The work offers both a diagnosis and a steering wheel for cross-modal conflict.

Catalog record

CREATORFrancesco Ortu, Zhijing Jin, Diego Doimo, Alberto Cazzaniga, [10px] University of Trieste AREA Science Park
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13868

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models Francesco Ortu, Zhijing Jin, Diego Doimo, Alberto Cazzaniga, [10px] University of Trieste AREA Science Park Abstract Vision-language models (VLMs) increasingly leverage diverse knowledge sources to address complex tasks, often encountering conflicts between their internal parametric knowledge and external information. Knowledge conflicts can result in hallucinations and unreliable responses, but the mechanisms governing such interactions remain unknown. To address this gap, we analyze the mechanisms that VLMs use to resolve cross-modal conflicts by introducing a dataset of multimodal counterfactual queries that deliberately contradict internal commonsense knowledge. We localize with logit inspection a small set of heads that control the conflict. Moreover, by modifying these heads, we can steer the model towards its internal knowledge or the visual inputs. Finally, we show that attention from such heads pinpoints localized image regions driving visual overrides, outperforming gradient-based attribution in precision. 1 1 1 Code and Dataset: francescortu/Seeing-Knowing † † footnotetext: Correspondence: [email protected] , [email protected] , [email protected] , [email protected] 1 Introduction Vision–language models (VLMs) Alayrac et al. 2022 ; Li et al. 2022 ; Liu et al. 2023 ; Team 2024 ; Deitke et al. 2024 have shown remarkable versatility in various multimodal tasks, from image understanding to image generation. They draw on their ability to combine two key sources of information: a rich set of world knowledge acquired during pretraining, and contextual cues provided in the input prompts. However, these two sources can sometimes contradict each other, for example when the pretraining knowledge becomes outdated Lazaridou et al. 2021 ; Luu et al. 2022 or when prompts include intentionally misleading visual information Liu et al. 2024d . Such conflicts often lead to hallucinations in model responses Cui et al. 2023 ; Liu et al. 2024a ; Guan et al. 2024 , yet the internal mechanisms responsible for these errors remain poorly understood. Xu et al. 2024 . Figure 1: Overview of Our Approach. ( Top ) We construct prompts that induce a conflict between a vision-language model’s internal factual knowledge and counterfactual visual context. ( Bottom ) We then analyze which components in the model mediate this tension, identifying attention heads and visual patches that favor factual or visually grounded predictions. In this work, we analyze how VLMs resolve conflicts between visual input and internal knowledge by framing the problem through counterfactual image-text pairs. We prompt the VLMs with images depicting unusual or absurd scenes taken from the Whoops! dataset Guetta et al. 2023 , followed by a sentence encouraging a typical knowledge-based continuation. As shown in Fig. 1 , each input prompt is associated with a counterfactual pair of completions. For instance, the model may be shown an image of a wolf howling at the sun, a scene that contradicts commonsense knowledge, and asked to complete the prompt accordingly (see top-left panel). We construct the dataset such that VLMs, when prompted with text alone, generate commonsense responses while in the presence of the image, change their prediction to align with the visual context, even when it contradicts their internal knowledge. Building on the approach of Ortu et al. 2024 , we identify which internal components of the model contribute the most to factual versus counterfactual predictions. We find that a small subset of attention heads mediates this competition, and targeted interventions on these heads can reliably alter the model’s outputs. We also show that these heads are more effective than gradient-based methods in identifying the most important parts of an image to resolve multimodal conflicts in VLMs. In summary, our contributions are as follows: 1. We construct Whoops-aha! , a dataset that combines images containing counterfactual scene elements and commonsense textual queries, designed to analyze conflicts between visual context and internal knowledge (Sec. 4.1 ); 2. We identify the attention heads that promote factual and counterfactual responses, ranking their importance with logit attribution (Sec. 4.2 ); 3. By reweighting these heads, we show that we can control the tendency of the model to rely on the visual evidence or its internal knowledge and vice versa (Sec. 4.3 ); 4. We demonstrate that direct attention attribution from conflict-resolution heads provides more accurate identification of counterfactual image regions than traditional gradient-based attribution methods (Sec. 4.4 ). 2 Related Work Most prior work on knowledge conflicts has focused on language models and unimodal tasks, leaving the multimodal domain underexplored Xu et al. 2024 . The analyses of knowledge conflicts in language models have largely been behavioral, showing that when resolving conflicts between contextual and internal knowledge, language models can overrely on their internal knowledge or contextual information, depending on factors such as model size Longpre et al. 2021 and conflicting external information Chen et al. 2022 . Wang et al. 2024 found that even SOTA language models often fail to report inconsistencies between in-context information and their internal knowledge. Few works have analyzed the internal mechanisms underlying conflict resolution. Ortu et al. 2024 identified two heads that mediate between factual and counterfactual information, while Jin et al. 2024 showed that pruning specific heads can steer the model’s reliance toward internal or contextual sources. In the multimodal domain, studies on VLMs have primarily focused on benchmark construction and black-box evaluation ( Le et al. 2023 ; Han et al. 2024 ; Golovanevsky et al. 2025b ; Guan et al. 2024 ) . For example, Han et al. 2024 introduced a dataset probing contextual knowledge conflicts introduced by deceptive visual elements in prompts, while Golovanevsky et al. 2025b proposed NOTICE, using semantically corrupted image pairs to analyze attention heads behavior in LLaVA and BLIP. Additionally, Le et al. 2023 introduced COCO-Counterfactuals , a dataset comprising minimally-edited counterfactual image pairs, and Liu et al. 2024c developed ConflictVis to evaluate conflicts between visual input and parametric knowledge. However, both studies limited their analyses to evaluating model behavior and prompt structures, without investigating the internal mechanisms by which models resolve such conflicts. During the final phase of this project (June 2025), we discovered concurrent research by Golovanevsky et al. 2025a , which introduces steering vectors to control model predictions and examines how varying visual input affects the competition between modalities. Their approach uses pairs of images, one consistent with the model’s internal knowledge and one modified to introduce a counterfactual variation, focusing on simple object attributes such as color or size. Our approach differs in that we use images depicting complex scenes that contradict common sense, combined with captions specifically designed to generate commonsense responses aligned with the model’s internal knowledge, thereby generating a conflict. 3 Background and Methods 3.1 Model Architectures This study investigates how visual input interacts with the model’s internal knowledge during text generation in VLMs. Given a sequence of k image-text tokens, a VLM encodes the image using a vision encoder and the text using an embedding matrix, producing the residual stream \mathbf{x}\in\mathbb{R}^{d\times k} , where d is the hidden dimension of the model. We denote the residual stream at position i and layer l as \mathbf{x}_{i}^{l} . The residual stream is processed through L layers, each composed of an attention block \mathbf{a}^{l} and an MLP block \mathbf{m}^{l} . After the final layer, it is projected to the vocabulary space via an unembedding matrix W_{U}\in\mathbb{R}^{d\times|V|} .

Read the complete text