VLDBench Evaluating Multimodal Disinformation with Regulatory Alignment
Shaina Raza, Ashmal Vayani, Aditya Jain, Aravind Narayanan, Vahid Reza Khazaie, Syed Raza Bashir, Elham Dolatabadi, Gias Uddin, Christos Emmanouilidis, Rizwan Qureshi, Mubarak Shah · 2025
Read VLDBench Evaluating Multimodal Disinformation with Regulatory Alignment by Shaina Raza, Ashmal Vayani, Aditya Jain, Aravind Narayanan, Vahid Reza Khazaie, Syed Raza Bashir, Elham Dolatabadi, Gias Uddin, Christos Emmanouilidis, Rizwan Qureshi, Mubarak Shah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11361.txt.
Introduction
Shaina Raza and co-authors build VLDBench, a benchmark for spotting disinformation that mixes text and images. Their 2025 paper notes that most safety tests cover text alone, or false content shared without intent. Crafted propaganda and conspiracy pieces that mimic real news have had far less study.
The benchmark holds about sixty-two thousand labeled text-image pairs in thirteen classes, drawn from fifty-eight news outlets. Twenty-two experts spent over five hundred hours checking labels through a semi-automated pipeline. Tests of leading language and vision models showed that adding image cues lifted detection scores by five to thirty-five points. Data and code are released for further work.
Catalog record
| CREATOR | Shaina Raza, Ashmal Vayani, Aditya Jain, Aravind Narayanan, Vahid Reza Khazaie, Syed Raza Bashir, Elham Dolatabadi, Gias Uddin, Christos Emmanouilidis, Rizwan Qureshi, Mubarak Shah |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-11361
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VLDBench Evaluating Multimodal Disinformation with Regulatory Alignment Shaina Raza, Ashmal Vayani, Aditya Jain, Aravind Narayanan, Vahid Reza Khazaie, Syed Raza Bashir, Elham Dolatabadi, Gias Uddin, Christos Emmanouilidis, Rizwan Qureshi, Mubarak Shah Abstract Detecting disinformation that blends manipulated text and images has become increasingly challenging, as AI tools make synthetic content easy to generate and disseminate. While most existing AI-safety benchmarks focus on single-modality misinformation (i.e., false content shared without intent to deceive), intentional multimodal disinformation, such as propaganda or conspiracy theories that imitate credible news; remains largely unaddressed. In this work, we introduce the V ision- L anguage D isinformation Detection Bench mark ( VLDBench ), the first large-scale resource supporting both unimodal (text-only) and multimodal (text + image) disinformation detection. VLDBench comprises approximately 62,000 labeled text–image pairs across 13 categories, curated from 58 news outlets. Using a semi-automated pipeline followed by expert review, 22 domain experts invested over 500 hours to produce high-quality annotations with substantial inter-annotator agreement. Evaluation of state-of-the-art LLMs and VLMs on VLDBench shows that adding visual cues improves detection accuracy, with gains ranging from 5 points for strong baselines (e.g., LLaMA-3.2-11B-Vision 74.82% vs. LLaMA-3.2-1B-Instruct 70.29%) to 25–30 points for smaller families (e.g., LLaVA-v1.5-Vicuna7B 72.32% vs. Vicuna-7B-v1.5 55.21%), reflecting complementary evidence from images (e.g., meme-like visuals, image–text consistency) that text alone cannot capture. We provide data and code for evaluation, fine-tuning and robustness tests to support disinformation analysis. Developed in alignment with the AI Goverance frameworks (MIT AI Risk Repository), VLDBench offers a principled foundation for advancing trustworthy disinformation detection in multimodal media. Project: https://vectorinstitute.github.io/VLDBench/ Data: https://huggingface.co/datasets/vector-institute/VLDBench Code: https://github.com/VectorInstitute/VLDBench Journal: Elsevier Issue: new Issue: “collapse” Issue: boil all tap water immediately Issue: Share this now Issue: new Issue: “boil all tap water,” “share this now” Issue: “collapse” 1 Introduction The global proliferation of false or misleading content has reached alarming levels: approximately 60% of people worldwide report encountering false narratives on digital platforms 1 1 1 Fake news statistics (2024) , and 94% of journalists view fabricated news as a significant threat to public trust 2 2 2 Pew Research Center (2022) Report on Journalism . The World Economic Forum 3 3 3 World Economic Forum, Global Risks Report 2024 identifies misinformation (unintentionally shared false information) and disinformation (deliberately deceptive content) among the leading global risks for 2024. Distinguishing between these two AI risks is crucial: while misinformation spreads without malicious intent [ 62 ] , disinformation is strategically weaponized, exploiting digital ambiguity to erode societal trust 4 4 4 7 Common Forms of Information Disorder . In this work, we restrict our scope to disinformation manifested in multimodal (text–image) contexts. Generative AI (GenAI) has recently played a paradoxical role, as it can both generate and detect disinformation [ 7 ] . For instance, text-to-image models can produce convincing but fabricated visuals of public figures at events that never occurred, while the same models can also be leveraged to flag image–text inconsistencies, such as detecting when an old protest photo is falsely paired with a recent political headline. Although recent advances have improved AI-driven detection methods, addressing disinformation in real-world news (as illustrated in Figure 1 ) remains challenging. While most research continues to focus on text-only cases, multimodal disinformation: such as repurposed images used to reinforce fabricated narratives; is still significantly underexplored, posing complex technical challenges and serious societal implications. Figure 1 : Schematic illustration of the disinformation flow: fabricated narratives and multimodal manipulation distort real-world information, producing myths and deliberate falsehoods and ultimately eroding public trust. Governments and technology companies are increasingly addressing disinformation through emerging regulatory frameworks. Policies such as the EU Action Plan [ 19 ] , the U.S. Digital Fairness Act 5 5 5 New York State Senate — Bill A3308 (2023) , and Canada’s AI initiatives [ 30 ] (summarized in Table 2 ) emphasize transparency, accountability, and responsible AI deployment. In parallel, industry leaders—including Google 6 6 6 Google AI Principles , Meta 7 7 7 Meta Principles , Microsoft 8 8 8 Microsoft AI Fairness Checklist , and Anthropic [ 2 ] , have launched technical countermeasures through collaborative initiatives, outlined in Table 3 . The MIT AI Risk Repository [ 67 ] , which synthesizes over 1,600 risks across 65 AI frameworks, explicitly identifies disinformation as a systemic threat requiring robust governance mechanisms (see Appendix A.1 ). However, despite these efforts, governance frameworks and industry pledges remain largely principle-level; they lack practical benchmarking resources that operationalize and audit performance on multimodal disinformation detection. This gap motivates VLDBench, a governance-aligned, human-verified benchmark for evaluating text–image disinformation. Existing safety benchmarks for large language models (LLMs), such as SafetyBench [ 87 ] and AISB [ 17 ] ; primarily target general harms (e.g., toxicity, bias). Large Vision–language models (LVLMs) safety benchmarks, including MM-SafetyBench [ 41 ] , MultiTrust [ 86 ] (social bias), VLGuard [ 90 ] , ALM-Bench [ 76 ] (cultural bias), and Rainbow Teaming [ 60 ] , emphasize robustness and alignment but the specific threat of disinformation is not under specific topics. Some benchmarks, such as Misinformer [ 52 ] , Vision–Language Deception Detection [ 54 ] , and deepfake-oriented resources [ 37 ] , incorporate multimodal elements, yet intentional multimodal disinformation (e.g., propaganda or health scams pairing misleading text with incongruent or manipulated imagery) remains out of scope. GenAI is reshaping how people access and produce news, with text-image systems able to fabricate persuasive content at scale. Policymakers now warn not only about user deception but also about damage to the digital commons : The European Commission’s JRC Generative AI Outlook (2025) 9 9 9 The European Commission’s JRC Generative AI Outlook (2025) cautions that “AI generated errors and disinformation can pollute open knowledge repositories (e.g., Wikipedia), creating costly verification burdens". Therefore, in this paper we introduce VLDBench , a multimodal benchmark for news style text-image disinformation detection, including robustness stress tests and governance- aware reporting. Our goal is to offer a rigorous, open resource to measure and mitigate risks before synthetic content contaminates shared knowledge ecosystems. To our knowledge, VLDBench is first comprehensive, human-verified benchmark aligned with emerging AI governance standards, supporting accountability, transparency, and robustness in disinformation detection across both unimodal (text-only LLMs) and multimodal (text-image) systems. Table 1 compares our benchmark with prior efforts, showing that existing studies primarily target misinformation and seldom address disinformation. Table 1 : Comparison of VLDBench with contemporary datasets. The Annotation ( ) means Manual and ( , ) indicates Hybrid (Human+AI). Access ? ( ✓ ) refers to open-source and ( ) indicates Request required. The Real ? is defined as ( ✓ ) if it’s real-world data and ( ✗ ) if it’s a Synthetic data. * Multiple includes Politics, National, Business & Finance, International, Local/Regional, Entertainment, Opinion/Editorial, Health, Other, Sports, Technology, Weather & Environment, Science . Dataset Primary Task Multimodal Real? Category Annotation type Access? LIAR [ 79 ] Misinfo. (Text) ✗ ✓ Politics ✓ FakeNewsNet [ 66 ] Misinfo. (Text+Image) ✓ ✓ Politics, Social BuzzFace [ 61 ] Misinfo. (Text) ✗ ✓ Politics ✓ FEVER [ 72 ] Misinfo. (Text) ✗ ✓ Social Media RealNews [ 84 ] Misinfo. (Text) ✗ ✓ General News Nela-GT [ 32 ] Misinfo. (Text) ✗ ✓ Politics,