Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models

Jiaying Wu Fanxiao Li Min-Yen Kan Bryan Hooi · 2025

Read Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models by Jiaying Wu Fanxiao Li Min-Yen Kan Bryan Hooi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15489.txt.

Introduction

Jiaying Wu and co-authors teach machines to spot the intent behind misleading news, in this 2025 paper. Fake stories mislead not only through false facts but through the slant a creator plans on purpose. The team built a benchmark of twelve thousand image-caption pairs, each grounded in a trusted source article, with the creator's aim and plan modeled by a guided simulation.

The set covers both misleading and straight cases, with tricks played on pictures, text, or both. It supports three tasks: flagging misleading intent, naming its source, and inferring what the creator wanted. Fourteen top vision-language models struggled, leaning on shallow cues such as polish or surface fit. Models trained on the new set carried their gains to real-world fake news tests.

Catalog record

CREATORJiaying Wu Fanxiao Li Min-Yen Kan Bryan Hooi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15489

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models Jiaying Wu Fanxiao Li Min-Yen Kan Bryan Hooi Abstract The real-world impact of misinformation stems from the underlying misleading narratives that creators seek to convey. As such, interpreting misleading creator intent is essential for multimodal misinformation detection (MMD) systems aimed at effective information governance. In this paper, we introduce an automated framework that simulates real-world multimodal news creation by explicitly modeling creator intent through two components: the desired influence and the execution plan . Using this framework, we construct DeceptionDecoded , a large-scale benchmark comprising 12,000 image-caption pairs aligned with trustworthy reference articles. The dataset captures both misleading and non-misleading intents and spans manipulations across visual and textual modalities. We conduct a comprehensive evaluation of 14 state-of-the-art vision-language models (VLMs) on three intent-centric tasks: (1) misleading intent detection , (2) misleading source attribution , and (3) creator desire inference . Despite recent advances, we observe that current VLMs fall short in recognizing misleading intent, often relying on spurious cues such as superficial cross-modal consistency, stylistic signals, and heuristic authenticity hints. Our findings highlight the pressing need for intent-aware modeling in MMD and open new directions for developing systems capable of deeper reasoning about multimodal misinformation. 1 1 1 Link to DeceptionDecoded : [GitHub Repo] . Content Warning: this paper contains potentially harmful text and images. 1 Introduction Figure 1: The importance of detecting misleading creation intent in multimodal news beyond surface-level cross-modal consistency. For instance, a malicious creator can craft captions that are semantically consistent with the image yet deliberately convey a false narrative. Multimodal misinformation, which combines persuasive text with compelling visuals, poses significant threats to public understanding and can lead to serious societal harm Zhou and Zafarani 2020 ; Alam et al. 2022 ; Do Nascimento et al. 2022 . A growing body of research in multimodal misinformation detection (MMD) focuses on identifying cross-modal misalignment , particularly in two major forms: (1) out-of-context (OOC) misinformation Yuan et al. 2023 ; Qi et al. 2024 , where images and captions from unrelated events or time periods are falsely paired; and (2) multimodal media manipulation Shao et al. 2023 ; Liu et al. 2025 , involving subtle changes such as shifts in facial expressions or caption phrasing that alter the perceived message. However, current MMD benchmarks often simulate such misalignments in artificial ways, either by mismatching image and text with CLIP-based Radford et al. 2021 similarity scores Luo et al. 2021 or through sentiment word substitutions Sudhakar et al. 2019 ; Shao et al. 2023 . These arbitrary strategies create a fundamental gap between constructed benchmarks and the complexity of real-world MMD. Beyond surface-level cross-modal misalignment, the key to effective real-world misinformation governance lies in detecting and understanding misleading creator intent Appelman et al. 2022 ; Jaidka et al. 2025 . Many misinformation campaigns are deliberately crafted to advance specific agendas, often without the audience’s awareness, yet still manage to substantially influence public opinion Ecker et al. 2022 ; Broda and Strömbäck 2024 . For example, as illustrated in Figure 1 , a slight modification to a news caption can lead readers to attribute significant casualties to government negligence, ultimately eroding institutional trust. Although preliminary efforts have explored intent interpretation Da et al. 2021 ; Gabriel et al. 2022 ; Wang et al. 2023 , they are often limited to unimodal settings and fail to capture the synergy between visual and textual cues. Moreover, these approaches rely on reader inference of creator intent, which are prone to error due to the deceptive framing of such content. These limitations highlight a pressing need for transparent modeling of ground-truth creator intents in multimodal content, along with systematic benchmarks for detecting misleading communicative objectives. To this end, we establish an intent-guided framework for simulating intent-aware multimodal news reporting. Specifically, we define diverse combinations of desired influence and execution plans, two core components that characterize real-world communication strategies Hallahan et al. 2007 ; Paul 2011 . Building on this, we introduce DeceptionDecoded , a large-scale benchmark designed to reveal misleading creator intents in multimodal news. Constructed upon real-world image-caption pairs from the VisualNews Liu et al. 2021 repository, DeceptionDecoded contains 12,000 instances created with misleading and non-misleading intent, spanning a broad spectrum of misleading intent types and framing strategies. Each instance is paired with a trustworthy reference article to serve as contextual grounding, and the quality of the generated data are further validated through human evaluation (Section 3.4 ). The benchmark supports evaluation of three intent-centric tasks related to MMD: (1) Misleading Intent Detection , which determines whether a news piece was intentionally crafted to mislead; (2) Misleading Source Attribution , which identifies whether the misleading signal originates from the image or the text; and (3) Creator Desire Inference , which infers the creator’s intended area of societal impact, such as political polarization or public health disruption. Using DeceptionDecoded , we evaluate 14 representative vision-language models (VLMs). Despite recent advances in multimodal reasoning Hurst et al. 2024 ; Anthropic 2025c , we find that state-of-the-art VLMs still struggle with intent understanding in the context of multimodal news . For instance, even relatively strong models such as GPT-4o-mini OpenAI 2024 achieve only 18.8% accuracy when misleading intent is subtly expressed through caption manipulation. More concerningly, our analysis reveals that VLMs tend to rely on spurious features when detecting misleading intent, including superficial image–text consistency, stylistic patterns in the text, and heuristic cues suggesting authenticity. This over-reliance leads to significant performance degradation under simple adversarial manipulations, such as stylistic reframing Chen and Shu 2024 ; Wu et al. 2024 and persuasive prompting attacks Zeng et al. 2024 . Our benchmark and empirical analyses establish a foundation for future MMD research that targets deeper semantic reasoning and intent-aware understanding beyond shallow cross-modal consistency patterns. Benchmark Task Setup Content Modality Creator Intent News Context Textual Visual EMU Da et al. 2021 visual manipulation understanding - ✔ ✔ (viewer-perceived) - Fakeddit Nakamura et al. 2020 mixed-source MMD ✔ ✔ - - MRF Gabriel et al. 2022 reader perception reasoning ✔ - ✔ (reader-perceived) - NewsINT Wang et al. 2023 news intent interpretation ✔ - ✔ (reader-perceived) - PolitiFact Shu et al. 2020 MMD on social media ✔ ✔ - ✔ GossipCop Shu et al. 2020 MMD on social media ✔ ✔ - ✔ NewsCLIPpings Luo et al. 2021 OOC misinformation detection ✔ ✔ - - DGM4 Shao et al. 2023 multimedia manipulation detection ✔ ✔ - - MMFakebench Liu et al. 2025 mixed-source MMD ✔ ✔ - - DeceptionDecoded (Ours) misleading intent detection ✔ ✔ ✔ (creator-produced) ✔ Table 1: Comparison between DeceptionDecoded and prior benchmarks on misinformation-related tasks. 2 Related Work 2.1 Multimodal Misinformation Detection With the increasing prevalence of persuasive visual–textual misinformation on online platforms, research has expanded from detecting purely textual misinformation Rashkin et al. 2017 ; Chen and Shu 2024 ; Wu et al. 2024 to tackling multimodal misinformation detection (MMD). A growing body of work emphasizes misinformation arising from subtle cross-modal misalignments, such as out-of-context (OOC) misinformation Qi et al. 2024 and multimedia manipulation Shao et al. 2023 ; Liu et al. 2025 . Recent advances leverage VLMs like LLaVA Liu

Read the complete text