Uncovering and Mitigating Transient Blindness in Multimodal Model Editing

Xiaoqi Han, Ru Li, Ran Yi, Hongye Tan, Zhuomin Liang, Víctor Gutiérrez-Basulto, Jeff Z. Pan · 2025

Read Uncovering and Mitigating Transient Blindness in Multimodal Model Editing by Xiaoqi Han, Ru Li, Ran Yi, Hongye Tan, Zhuomin Liang, Víctor Gutiérrez-Basulto, Jeff Z. Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13243.txt.

Introduction

Xiaoqi Han and 6 co-authors examine how editing multimodal models can quietly break their use of images. Editing aims to fix wrong facts, but older tests use random or weakly matched images and can overstate success. The authors build a fuller locality test with three image settings and seven data types.

Their dynamic test, De-VQA, reveals a flaw they call transient blindness, where an edited model leans on text and ignores the picture. Token checks show edits shift text tokens far more than image tokens. A new loss that guards locality cuts the flaw and improved locality by about seventeen percent on average over baselines.

Catalog record

CREATORXiaoqi Han, Ru Li, Ran Yi, Hongye Tan, Zhuomin Liang, Víctor Gutiérrez-Basulto, Jeff Z. Pan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-13243

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Uncovering and Mitigating Transient Blindness in Multimodal Model Editing Xiaoqi Han, Ru Li, Ran Yi, Hongye Tan, Zhuomin Liang, Víctor Gutiérrez-Basulto, Jeff Z. Pan Abstract Multimodal Model Editing (MMED) aims to correct erroneous knowledge in multimodal models. Existing evaluation methods, adapted from textual model editing, overstate success by relying on low-similarity or random inputs, obscure overfitting. We propose a comprehensive locality evaluation framework, covering three key dimensions: random-image locality, no-image locality, and consistent-image locality , operationalized through seven distinct data types, enabling a detailed and structured analysis of multimodal edits. We introduce De-VQA, a dynamic evaluation for visual question answering , uncovering a phenomenon we term transient blindness , overfitting to edit-similar text while ignoring visuals. Token analysis shows edits disproportionately affect textual tokens. We propose locality-aware adversarial losses to balance cross-modal representations. Empirical results demonstrate that our approach consistently outperforms existing baselines, reducing transient blindness and improving locality by 17% on average. Code and Appedix with Additional Results — https://github.com/sev777/DE-VQA Introduction The rapid advancement of large language models (LLMs), such as ChatGPT and Deepseek ( 29 ; 23 ) , has driven their widespread adoption as sources of factual knowledge ( 14 ; 35 ; 40 ) for downstream tasks ( 24 ) , and multimodal LLMs have further extended these capabilities to vision-language tasks, with strong performance in cross-modal understanding ( 23 ; 18 ; 42 ) . However, a critical challenge for multimodal LLMs is knowledge obsolescence, making regular updates essential to maintain accuracy. To address this, model editing ( 28 ; 6 ; 21 ; 11 ) has emerged as an efficient solution to correct factual inaccuracies without costly retraining. Figure 1: Current locality evaluation focuses only on low-similarity data (a), while the edited model fail on high-similarity cases (b). Multimodal model editing (MMED) specifically targets updating a multimodal model’s predictions for specific image-text pairs while preserving performance on unrelated inputs. Recent works by 5 have introduced dedicated datasets and adapted evaluation metrics for this task. These metrics address two core aspects: (1) Factual updating, assessed through dimensions like reliability , generalization to comprehensively measure an edit’s effectiveness; (2) Side-effect evaluation, referred to as locality , where current methods simply sample random text or image-text questions (see Figure 1 (a)). Building on this foundation, recent methods ( 13 ; 20 ; 7 ) have proposed more challenging factual evaluation protocols to better assess an editor’s editing capacity. However, they largely overlook the locality aspect, an essential factor in multimodal editing. By directly adopting locality metrics from text-only settings, these methods fail to account for the unique challenges of multimodal inputs, where such simplistic extensions are often inadequate. Specifically, existing locality evaluation focuses only on whether the final output stays the same, without examining whether the model’s inference process or modality usage has changed. As shown in Figure 1 , after we update the output for question-image pair, the model cannot output the correct answer “Black” instead of “Green”. Even when presented with a rephrased question “What color are the shoes of this boy?” alongside an image of yellow shoes (cf. Figure 1 (b)), the edited model still outputs “Black”, disregarding the contradictory visual evidence. More critically, when given only the edited text (with no accompanying image), the model still incorrectly responds with “Black”, indicating that the post-edit model tends to overfit to the edited fact and ignore visual evidence when faced with semantically similar inputs. This issue is particularly problematic in multimodal models, where both image and text are jointly used for prediction. An edit that targets only the textual representation can lead the model to over-rely on language and disregard visual cues, even when the output appears correct. Such behavior reflects a breakdown in cross-modal balance, a key aspect of model fidelity in multimodal settings. To address this issue, we propose a novel evaluation framework for analyzing the locality of multimodal model editing (MMED). We introduce De-VQA , a Dynamic Evaluation framework for visual question answering (VQA), which automatically selects adversarial samples that are similar (but not identical) to the editing data in either the text or image modality. To assess a post-edit model’s multimodal ability, we define three locality metrics: Random-Image Locality (RI-Loc) , No-Image Locality (NI-Loc) , and Consistent-Image Locality (CI-Loc) . These are quantified using seven distinct data types designed to probe different aspects of locality. Using De-VQA, we uncover the phenomenon of transient blindness , a degradation in multimodal locality where edits to a model’s textual knowledge cause it to temporarily “ignore” or under-utilize visual inputs during inference. To investigate the root causes of transient blindness, we employ token attribution tracing and find that current editing methods disproportionately alter textual representations while leaving visual representations largely unaffected. This imbalance in hidden state updates causes edited models to over-rely on textual cues, giving rise to transient blindness. We introduce an adversarial loss that amplifies the influence of visual inputs during editing, thereby reducing this effect. Experiments on two datasets and across two multimodal models demonstrate that our method effectively mitigates transient blindness, improving both the robustness and reliability of the edited models. In summary, our main contributions are as follows: (1) We propose De-VQA , a dynamic evaluation framework designed to detect breakdowns in locality during multimodal model editing (MMED). It introduces three evaluation dimensions operationalized through seven carefully constructed data types. Together, these components form the first comprehensive benchmark for assessing locality preservation in MMED. (2) We characterize locality failures as transient blindness : a phenomenon in which post-edit models overfit to textual inputs resembling the edit, while neglecting visual information. Our analysis reveals that transient blindness stems from imbalanced updates between the textual and visual modalities during the editing process. (3) We propose a locality loss that balances cross-modal updates. Extensive experiments show that our approach consistently mitigates transient blindness and improves locality preservation by 17% on average across multiple models and datasets, while maintaining edit accuracy. Multimodal Model Editing Task Definition. Multimodal model editing (MMED) aims to correct the output of a multimodal model f(\cdot;\theta) with parameters \theta without full retraining or fine-tuning, while preserving the model’s original outputs for unrelated inputs. MMED employs an editing function g(\cdot) to achieve g(f(x_{e}))=a , where x_{e} is the input for the edits and a is the desired output. For inputs x_{o} unrelated to x_{e} , the goal is to maintain their outputs unchanged, i.e. g(f(x_{o}))=y=f(x_{0}) . Note: x_{*} is composed of the input image x^{i}_{*} and the input text x^{t}_{*} . Evaluation Metrics. To evaluate the performance of an editing method g(\cdot) , prior work uses the following datasets: the editing dataset D_{e} composed of tuples \langle x^{i}_{e},x^{t}_{e},y_{e},a\rangle , the semantically equivalent dataset D_{g} composed of tuples \langle x^{i}_{g},x^{t}_{g},y_{g},a\rangle and the locality dataset D_{o} composed of tuples \langle x^{i}_{o},x_{o}^{t},y_{o}\rangle . They use the following metrics: • Reliability measures how effectively g(\cdot) updates the output of model to a instead of y_{e} , for each data element in D_{e} . This is calculated as: \textrm{Rel}=\mathbb{E}_{\langle x^{i}_{e},x^{t}_{e},y_{e},a\rangle\in D_{e}}[\mathbf{1}_{g(f(x^{i}_{e},x^{t}_{e}),y_{e})=a}]. (1) • Generality assesses the consistency of predictions from the post-edit model g(f(\cdot)) when presented with semantically equivalent inputs from D_{g} . This includes modified text and images, known as text-generality (T-Gen) and image-generality (I-Gen)

Read the complete text