Images Amplify Misinformation Sharing in Vision-Language Models
Alice Plebe, Timothy Douglas, Diana Riazi, R. Maria del Rio-Chanona · 2025
Read Images Amplify Misinformation Sharing in Vision-Language Models by Alice Plebe, Timothy Douglas, Diana Riazi, R. Maria del Rio-Chanona (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13302.txt.
Introduction
People share news more readily when a picture rides along, and the picture makes even false stories feel truer. Alice Plebe, Timothy Douglas, Diana Riazi, and Maria del Rio-Chanona ask whether vision-language models fall for the same pull. They built a set of fact-checked political stories, true and false, each paired with its image.
A crafted prompting method got four leading models to give sharing decisions instead of refusing. Adding the image raised resharing of false news by about fourteen and a half percent, and of true news by about five. Giving a model a dark, antisocial persona made false sharing worse; a Republican persona made it less sensitive to truth. One smaller model resisted the image effect best. The authors warn that multimodal systems inherit human-like biases.
Catalog record
| CREATOR | Alice Plebe, Timothy Douglas, Diana Riazi, R. Maria del Rio-Chanona |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 111 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13302
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Images Amplify Misinformation Sharing in Vision-Language Models Alice Plebe, Timothy Douglas, Diana Riazi, R. Maria del Rio-Chanona Abstract As language and vision–language models (VLMs) become central to information access and online interaction, concerns grow about their potential to amplify misinformation. Human studies show that images boost the perceived credibility and shareability of information, raising the question of whether VLMs exhibit the same vulnerability. We present the first study examining how images influence VLMs’ propensity to reshare news content, how this effect varies across model families, and how persona conditioning and content attributes modulate such behavior. We develop a jailbreaking-inspired prompting strategy that bypasses VLMs’ default refusals to engage with controversial news, allowing them to generate resharing decisions across diverse topics and elicited traits, including antisocial ones. We evaluate four state-of-the-art VLMs on a novel multimodal dataset of fact-checked political news from PolitiFact, paired with images and ground-truth veracity labels. Our experiments show that image presence increases resharing rates by 14.5% for false news and 5.3% for true news. Persona conditioning further modulates this effect: Dark Triad traits amplify resharing of false news, whereas Republican-aligned profiles reduce sensitivity to veracity. Among the tested models, Claude-3-Haiku demonstrates the greatest robustness to visual misinformation. These findings reveal that VLMs replicate human-like biases in response to images, underscoring emerging risks for multimodal AI systems. They point to the need for evaluation frameworks and mitigation strategies that account for visual influence and persona-driven variability, particularly in sociotechnical settings where AI systems shape public discourse and information sharing. 1 Department of Industrial Engineering, University of Trento, Trento, Italy, 2 Computer Science Department, University College London, London, WC1E 6BT, United Kingdom [email protected] Code and Dataset — https://github.com/3lis/misinfo˙vlm Introduction Figure 1: In humans, image presence is known to increase perceived truthfulness of information. We investigate whether a similar effect applies to VLMs’ willingness to share news. We compare sharing-likelihood ratings when models are prompted on news items with and without accompanying images, while conditioning on personas, framing prompts in the third person, and eliciting step-by-step reasoning. False and misleading news distorts elections, undermines public-health campaigns, and has been identified as a major global risk ( Lewandowsky 2024 ) . Social media has become a primary source of news ( Tobitt 2024 ; Aubin and Liedke 2024 ) , where false claims spread faster than accurate information ( Vosoughi et al. 2018 ) and visual content drives higher engagement ( Li and Xie 2020 ) . At the same time, large language models (LLMs) are increasingly embedded in the news ecosystem, powering content recommendation systems at major outlets like Forbes and the Financial Times ( Forbes 2023 ; Financial Times 2023 ) and automated agents that curate and reshare news on social media. Research demonstrates that visual elements increase humans’ perceived credibility of false claims ( Newman et al. 2020 ; Newman and Schwarz 2024 ; Tseng and Yuan 2023 ; Seo 2020 ) . As vision–language models (VLMs), which process both text and images, become more widely deployed, this raises a critical question: do VLMs exhibit similar visual biases when deciding what content to reshare or recommend? If so, systems designed to filter and curate information may instead amplify visually-rich misinformation, posing a new threat to trustworthy news dissemination. Several studies have examined the factors driving misinformation spread in both LLMs and humans. In LLMs, misinformation can arise from hallucinations that promote conspiracy theories ( Bandara 2024 ) , from echo chamber effects that reinforce existing views ( Nehring et al. 2024 ) , and from sycophantic behavior, where false claims receive immediate affirmation ( Sharma et al. 2023 ) . In humans, susceptibility to misinformation is influenced not only by visual cues, but also by antisocial personality traits, which increase the likelihood of believing and sharing false information ( Calvillo et al. 2024 ; Peter et al. 2024 ) , as well as by demographic factors and extreme political orientations ( Imhoff et al. 2022 ) . Yet, the role of image presence in shaping misinformation within VLMs remains unexplored, as does its interaction with persona conditioning. Given that visual content significantly increase humans’ propensity to share news, and that LLMs can reflect biases observed in humans ( Hu et al. 2025 ; Acerbi and Stubbersfield 2023 ) , it is crucial to understand how VLMs handle misinformation when images are involved. It is equally important to examine how personality traits elicited in these systems might influence visual misinformation sharing, especially as LLMs become increasingly personalized to individual users ( Przystalski 2025 ) . This work addresses these issues by investigating the following research questions. RQ1: Are VLMs, like humans, more inclined to share a news item when it includes an image? RQ2: Is this pattern consistent across different model families? RQ3: How does persona conditioning, specifically through personality traits and demographic characteristics, modulate a model’s willingness to reshare content when an image is present? RQ4: How do news attributes such as veracity, topic, and image content interact to shape that decision? To answer these questions, we make two key methodological contributions. First, we develop a prompting strategy based on third-person instructions to elicit responses from VLMs. This is particularly necessary when models are conditioned on antisocial traits and presented with controversial or conspiratorial news items, as LLMs are designed to reject harmful or sensitive queries. Inspired by jailbreaking techniques, our approach redirects the toxic behavior to a fictional third-party user rather than the model itself, enabling more natural responses. Second, we curate a multimodal dataset of paired image–text news items from PolitiFact , annotated with expert veracity labels. While related datasets exist, none fully meet the requirements of this study: many lack paired images, omit reliable ground-truth labels, or predate the release of current VLMs. By focusing on recent news, our dataset reduces the risk that models rely on memorized factual knowledge, ensuring that resharing decisions reflect sensitivity to visual content. We evaluate four representative VLMs: two open-weight models (LLaVA-1.6 and Qwen2-VL) and two proprietary models (GPT-4o-mini and Claude-3-Haiku). Each model is developed by a different provider and reflects distinct architectural choices and training methodologies. This diversity is essential for assessing how multimodal misinformation risks manifest across differing governance and safety regimes. Proprietary models are subject to centralized corporate oversight and safety enforcement, whereas open-weight models operate with minimal regulatory control after release. Moreover, differences in training paradigms directly influence alignment strategies, data curation practices, and the handling of sensitive content, making cross-model comparison critical for understanding systemic risk. We report four main findings. First, VLMs are more likely to reshare news items when an image is present than when the same content is presented as text alone, with the effect stronger for false news (14.5% increase) than for true news (5.3%). Second, this pattern holds across all tested model families, though effect sizes vary by architecture: GPT-4o-mini and Qwen2-VL are the most affected, while Claude-3-Haiku shows the greatest robustness. Third, persona conditioning shapes resharing behavior: Dark Triad traits ( Paulhus and Williams 2002 ) amplify the spread of false news, whereas a Republican political stance reduces sensitivity to veracity, leading to similar sharing rates for both true and false content. Finally, news-level attributes such as