Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs
Chi Zhang, Wenxuan Ding, Jiale Liu, Mingrui Wu, Qingyun Wu, Ray Mooney · 2026
Read Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs by Chi Zhang, Wenxuan Ding, Jiale Liu, Mingrui Wu, Qingyun Wu, Ray Mooney (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19202.txt.
Introduction
Chi Zhang and five co-authors test whether false words can outshout true pictures in visual models. Vision language models do well on visual question tests, but their strength when text misleads them is little studied. Past work on falsehood focused on text only worlds, leaving the clash between senses untested.
The team builds a test set, ConText-VQA, where the text and the image tell opposed stories. It shows how the models settle the clash: sight or words. A model that drops what it sees for what it is told would fail in news, medicine, and safety checks where captions can lie.
Catalog record
| CREATOR | Chi Zhang, Wenxuan Ding, Jiale Liu, Mingrui Wu, Qingyun Wu, Ray Mooney |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-19202
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs Chi Zhang, Wenxuan Ding, Jiale Liu, Mingrui Wu, Qingyun Wu, Ray Mooney Abstract Vision-Language Models (VLMs) have shown strong multimodal reasoning capabilities on Visual-Question-Answering (VQA) benchmarks. However, their robustness against textual misinformation remains under-explored. While existing research has studied the effect of misinformation in text-only domains, it is not clear how VLMs arbitrate between contradictory information from different modalities. To bridge the gap, we first propose the ConText-VQA (i.e., Conf licting Text ) dataset, consisting of image-question pairs together with systematically generated persuasive prompts that deliberately conflict with visual evidence. Then, a thorough evaluation framework is designed and executed to benchmark the susceptibility of various models to these conflicting multimodal inputs. Comprehensive experiments over 11 state-of-the-art VLMs reveal that these models are indeed vulnerable to misleading textual prompts, often overriding clear visual evidence in favor of the conflicting text, and show an average performance drop of over 48.2 % 48.2\% after only one round of persuasive conversation. Our findings highlight a critical limitation in current VLMs and underscore the need for improved robustness against textual manipulation. 1 Introduction Recent advancements in Vision-Language Models (VLMs) have demonstrated their remarkable capabilities, including complex reasoning ( 45 ; 31 ; 44 ) , knowledge integration ( 60 ; 67 ) , and creativity generation ( 21 ) . However, similar to Large Language Models (LLMs), VLMs exhibit vulnerabilities in various cases. They are shown to be prone to hallucination ( 49 ; 37 ; 61 ) , sensitive to adversarial perturbations ( 38 ; 68 ) , and struggle with compositional understanding ( 15 ) , among many such issues. Previous work has shown that LLMs are vulnerable to external information that conflicts with their parametric knowledge ( 52 ; 56 ; 11 ) , including carefully crafted adversarial prompts ( 20 ; 57 ) and persuasive misinformation ( 58 ; 65 ) . However, while existing research has focused on manipulating low-level visual features, the robustness of VLMs in face of misinformation is less studied, and semantic attack is under-explored as an insidious vulnerability. Figure 1: VLMs are susceptible to textual misinformation that conflicts with visual evidence, causing them to fail on questions they would otherwise answer correctly. Consequently, we raise an important research question: How robust are VLMs when presented with misleading textual information that conflicts with visual evidence, especially on questions they initially answer correctly? We argue that for VLMs to be robust against textual misinformation, they should effectively balance evidence from multiple modalities, maintain fidelity to understanding and processing visual evidence, and properly ground their responses in the given evidence. Ensuring VLM robustness against textual misinformation is crucial for their reliable deployment in real-world applications. For instance, in autonomous driving, a system must reconcile potentially conflicting user instructions (e.g., voice commands) with its visual perception of the environment for safe operation ( 72 ) . Likewise, in content moderation, VLMs need to accurately evaluate visual materials even when they are accompanied by misleading textual descriptions, in order to prevent the spread of harmful content like hate speech ( 1 ) . In the critical domain of medical diagnostics, a model analyzing a radiological scan also must prioritize the visual data over a potentially erroneous summary in a patient’s record to avoid a misdiagnosis ( 48 ) . These scenarios validate concerns on the reliability of these models when misinformation leads to significant consequences. To this end, we introduce ConText-VQA , a benchmark consisting of VQA problems accompanied by persuasive textual misinformation, which are systematically generated with VLMs, using the strategies of repetition, logical appeal, credibility appeal, and emotional appeal. With ConText-VQA , we propose a framework to assess the robustness of VLMs against textual misinformation and persuasion. Specifically, we start with problems that the VLMs can answer correctly without misleading inputs. Then, we sample a Non-Fact from the distractor options and use a strong VLM to generate misleading persuasion. The persuasion is fed into the VLMs together with the original problem, and we verify the response change and confidence shift in this case to understand how the model belief changes when provided with contradictory multi-modal information. Our experiments with 11 state-of-the-art VLMs reveal that these models are indeed vulnerable to misleading textual persuasion, and show an average performance drop of 48.2\% in the presence of misinformation. To conclude, our contributions are as follows: • To the best of our knowledge, this is the first work to investigate the effect of misinformation on VLMs using persuasive conversations. While prior studies have explored persuasion in text-only domains, our work offers novel insight by systematically benchmarking how they impact a VLM’s arbitration between conflicting visual and textual modalities. • We propose the ConText-VQA dataset by filtering baseline image-question pairs and applying a carefully crafted prompt template to generate misleading questions that explicitly contradict the visual evidence. • We develop a framework to benchmark open-source and proprietary state-of-the-art VLM’s performance against textual manipulation. Experimental results show that VLMs are indeed vulnerable to misleading prompts, with an average performance drop of over 48.2\% after only one round of persuasive conversation. 2 Related Work Hallucination and Misinformation in VLMs Despite the increasing capabilities of VLMs, a wide range of work has revealed that they are prone to hallucination , especially where their textual output contradicts the visual input ( 16 ; 3 ; 2 ; 17 ; 13 ) . Vulnerability to hallucination significantly impairs their performance and reliability, and various methods have been proposed to mitigate this problem ( 49 ; 37 ; 61 ; 43 ) . While these works often focus on how VLMs generate unfaithful text spontaneously, our work investigates how they are affected by external textual misinformation that conflicts with visual evidence. This is a critical distinction, as we specifically probe the models’ decision-making process when faced with contradictory signals, particularly for questions they would otherwise answer correctly. A similar contemporary work ( 40 ) explores how models can be misled by semantic faithfulness to text, causing them to overlook visual consistency, particularly in tasks like scene text recognition. In comparison, our paper places greater emphasis on evaluating the model’s overall robustness to conflicting multi-modal inputs. Adversarial Attacks from a Multi-modal Perspective LLMs have long been known to be vulnerable to adversarial attacks ( 59 ; 64 ; 58 ) , where carefully crafted inputs can cause them to produce erroneous outputs. With the growing popularity of VLMs ( 35 ; 33 ; 46 ) , the introduction of the visual modality brings in new attack vectors. An emerging line of work studies the adversarial vulnerability of such multi-modal LLMs ( 70 ; 68 ; 18 ; 4 ; 12 ; 71 ) . Much of the existing research on VLM security has focused on vision-specific attacks. These include methods like applying low-level, often imperceptible, perturbations to images to fool the model or training learnable tokens that can be inserted to trigger a desired output ( 28 ; 73 ; 51 ) . There have also been works on mitigating the effect of multi-modal misinformation ( 53 ; 29 ) . However, targeted textual manipulations to