M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset
Jiahui Geng, Jonathan Tonglet*, Iryna Gurevych · 2025
Read M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset by Jiahui Geng, Jonathan Tonglet*, Iryna Gurevych (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23508.txt.
Introduction
Jiahui Geng, Jonathan Tonglet, and Iryna Gurevych built a new dataset for checking false claims that mix text with images. Older collections were small, covered one or two languages, handled a single task, or leaked evidence in ways that made tests too easy. Real fact-checking, the authors note, is broader and messier than those sets suggest.
Their dataset, M4FC, pairs 4,982 images with 6,980 claims checked by professionals from 22 organizations across many cultures and regions. Claims appear in one or two of ten languages, and the set supports six tasks, from spotting fake images to predicting the final verdict. The authors report baseline scores for each task and study how earlier tasks feed into the final verdict. They release the data and code publicly.
Catalog record
| CREATOR | Jiahui Geng, Jonathan Tonglet*, Iryna Gurevych |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23508
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset Jiahui Geng, Jonathan Tonglet*, Iryna Gurevych Abstract Existing real-world datasets for multimodal automated fact-checking have multiple limitations: they contain few instances, focus on only one or two languages and tasks, suffer from evidence leakage, or depend on external sets of news articles for sourcing true claims. To address these shortcomings, we introduce M4FC, a new real-world dataset comprising 4,982 images paired with 6,980 claims. The images, verified by professional fact-checkers from 22 organizations, represent diverse cultural and geographic contexts. Each claim is available in one or two out of ten languages. M4FC spans six multimodal fact-checking tasks: visual claim extraction, claimant intent prediction, fake detection, image contextualization, location verification, and verdict prediction. We provide baseline results for all tasks and analyze how combining intermediate tasks influence downstream verdict prediction performance. We make our dataset and code available. 1 1 1 github.com/UKPLab/M4FC 1 Introduction Since 2020, approximately 80% of online misinformation verified by fact-checkers has been multimodal, combining text with images, videos, or audio ( Dufour et al. 2024 ) . Debunking multimodal misinformation is challenging and time-consuming for human fact-checkers ( Silverman 2013 ; Mossou and Higgins 2021 ; Khan et al. 2023 ; Khan et al. 2024 ) . It typically involves performing several sub-tasks and utilizing specialized tools like satellite imagery providers and sun angle calculators ( Mossou and Higgins 2021 ; Khan et al. 2023 ; Khan et al. 2024 ) . Additionally, it requires expert knowledge of specific languages and cultural contexts ( Silverman 2013 ) . This work specifically targets the two most common categories of image-based misinformation ( Dufour et al. 2024 ) : (1) out-of-context (OOC) misinformation, where authentic images are paired with false claims about their date, location, or the people and events depicted; and (2) false claims paired with manipulated or fake images. Figure 1: An out-of-context instance from M4FC. The six AFC tasks are shown in blue and their output in gray. Multimodal automated fact-checking (AFC) assists human fact-checkers by automating parts of their workflow ( Akhtar et al. 2023 ) . Several datasets have been proposed for multimodal AFC. Synthetic datasets, on the one hand, are large and scalable ( Luo et al. 2021 ; Shao et al. 2024 ) but cannot fully mimic real-world misinformation. On the other hand, real-world datasets consist of claims verified by human fact-checkers, but these datasets tend to be much smaller ( Zlatkova et al. 2019 ; Hu et al. 2023 ; Papadopoulos et al. 2024 ; Tonglet et al. 2024 ) . Existing real-world datasets suffer from additional limitations. While multimodal misinformation is a global problem, these datasets cover only one or two languages and lack geographic diversity. Furthermore, the human fact-checking workflow involves several intermediate tasks before providing a verdict. However, the real-world datasets fail to account for most of these intermediate tasks. In this work, we introduce M4FC , a m ultimodal, m ultilingual, m ulticultural, and m ultitask real-world f act- c hecking dataset, with 4,982 images and 6,980 claims. M4FC addresses several limitations of existing resources. It contains labels for six multimodal AFC tasks organized into a pipeline, as illustrated in Figure 1 . This pipeline allows for automating a larger part of the fact-checking workflow and studying the interaction between intermediate tasks and verdict prediction. Images, claims, task labels, and metadata are drawn from articles produced by 22 fact-checking organizations across 17 countries. As a result, M4FC offers broad cultural and geographic coverage and spans 10 claim languages: Arabic, Dutch, English, French, German, Portuguese, Spanish, Tamil, Telugu, and Turkish. We introduce two new multimodal AFC tasks. Visual claim extraction addresses the mandatory step of formulating verifiable claims based on a screenshot of a social media post containing the image. Location verification validates candidate locations for images using two types of evidence previously overlooked in multimodal AFC research: maps and satellite images. We provide baseline results for all tasks. Furthermore, we evaluate how different sequential combinations of intermediate tasks affect verdict prediction performance. In summary, our contributions are as follows: (1) we introduce M4FC, a large-scale real-world multimodal, multilingual, multicultural, and multitask fact-checking dataset; (2) we introduce two new challenging multimodal AFC tasks; and (3) we conduct comprehensive experiments, including analyses of how combining intermediate tasks impacts verdict prediction performance. 2 Related work Claims Evidence Dataset # tasks # images # claims # languages Data split Source Leak? Type Weibo ( Jin et al. 2017 ) 1 9,528 9,528 1 (Zh) Event FC & News - - Fauxtography ( Zlatkova et al. 2019 ) 1 1,233 1,233 1 (En) Random FC & News No Text COSMOS (test) ( Aneja et al. 2023 ) 1 1,700 3,400 1 (En) No split FC & News - - MR2 ( Hu et al. 2023 ) 1 14,700 14,700 2 (En, Zh) Event FC & News Yes Text, Image Ookpik ( Pham et al. 2024 ) 1 545 1,090 1 (En) No split FC & News - - Post-4V ( Geng et al. 2024 ) 1 186 186 1 (En) No split FC - - VERITE ( Papadopoulos et al. 2024 ) 1 662 1,000 1 (En) No split FC Yes Text, Image 5Pils ( Tonglet et al. 2024 ) 1 1,676 1,676 1 (En) Temporal FC No Text 5Pils-OOC ( Tonglet et al. 2025 ) 2 624 1,248 1 (En) No split FC No Text, Image AVerImaTeC ( Cao et al. 2025 ) 1 1,297 1,297 1 (En) Temporal FC No Text, Image XFacta ( Xiao et al. 2025 ) 1 2,400 2,400 1 (En) Random CN & News Unknown Text, Image M4FC (ours) 6 4,982 6,980 10 Temporal FC No Text, Map, Satellite Table 1: Comparison of real-world multimodal AFC datasets. En and Zh indicate claims in English and Chinese, while FC, News, and CN stand for fact-checking articles, news articles, and X Community Notes, respectively. 2.1 Synthetic multimodal AFC datasets Several synthetic datasets have been proposed for multimodal AFC ( Sabir et al. 2018 ; Müller-Budack et al. 2020 ; Luo et al. 2021 ; Shao et al. 2024 ; Liu et al. 2025 ) . While synthetic datasets are large and scalable, they often suffer from biases. Papadopoulos et al. 2025 demonstrated that simple models can exploit these biases to achieve near SOTA performance, raising concerns about the reliability of synthetic data for evaluating progress in multimodal AFC. 2.2 Real-world multimodal AFC datasets Real-world datasets, summarized in Table 1 , collect claims from fact-checking articles. Because human fact-checking is time-intensive, limited articles are available, and real-world datasets are smaller than synthetic ones. They also often lack true claims ( Nielsen and McConville 2022 ) , as fact-checkers prioritize articles about false claims. To compensate, many datasets sample image-caption pairs from reputable news outlets and treat them as true claims ( Jin et al. 2017 ; Zlatkova et al. 2019 ; Aneja et al. 2023 ; Hu et al. 2023 ) . However, this introduces a distribution mismatch ( Nielsen and McConville 2022 ) , as the content, resolution, and topics of news images may differ from those in online misinformation. Existing datasets cover one or two claim languages and