Multimodal Climate Disinformation Detection: Integrating Vision-Language Models with External Knowledge Sources
Marzieh Adeli Shamsabad, Hamed Ghodrati · 2026
Read Multimodal Climate Disinformation Detection: Integrating Vision-Language Models with External Knowledge Sources by Marzieh Adeli Shamsabad, Hamed Ghodrati (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16108.txt.
Introduction
Marzieh Adeli Shamsabad and Hamed Ghodrati build a tool to spot false climate claims in mixed media. Their 2026 preprint notes how false posts now use strong pics and clips. Text checks alone miss tricks that sit in the image. False claims spread fast on social feeds. Late acts on climate can stem from such false posts.
The pair join vision and language models with facts from known sources. The joint check weighs words, pics and source facts as one. Proof from firm sources helps flag claims that clash with known facts. The tool aims to catch false posts that look true at a glance. It serves teams who guard climate facts online.
Catalog record
| CREATOR | Marzieh Adeli Shamsabad, Hamed Ghodrati |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-16108
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Climate Disinformation Detection: Integrating Vision-Language Models with External Knowledge Sources Marzieh Adeli Shamsabad, Hamed Ghodrati Abstract Climate disinformation has become a major challenge in today’s digital world, especially with the rise of misleading images and videos shared widely on social media. These false claims are often convincing and difficult to detect, which can delay actions on climate change. While vision-language models (VLMs) have been used to identify visual disinformation, they rely only on the knowledge available at the time of training. This limits their ability to reason about recent events or updates. The main goal of this paper is to overcome that limitation by combining VLMs with external knowledge. By retrieving up-to-date information such as reverse image results, online fact-checks, and trusted expert content, the system can better assess whether an image and its claim are accurate, misleading, false, or unverifiable. This approach improves the model’s ability to handle real-world climate disinformation and supports efforts to protect public understanding of science in a rapidly changing information landscape. 1 Introduction Climate change is one of the most urgent challenges of our time. Although scientific research has built a strong foundation for understanding the crisis, public response is often disrupted by the spread of disinformation. In recent years, this has increasingly taken the form of images and videos that are altered, taken out of context, or even generated by artificial intelligence. When combined with misleading text, such content can make false claims appear convincing and trustworthy, especially on fast-moving platforms like social media. This new type of disinformation is not just textual, it is visual and multimodal. It spreads quickly, triggers emotional reactions, and is often more difficult to verify. In the context of climate communication, where public trust in evidence is critical, this creates a serious risk. This paper addresses that challenge by going beyond current vision-language models, which rely solely on internal knowledge acquired during training. These models are unable to reason effectively about recent events or unseen claims. To overcome this limitation, we propose a system that integrates external knowledge sources, allowing the model to access up-to-date information and make more accurate decisions. The main objectives of this work are: 1. Detect climate-related disinformation by analyzing both images and text. 2. Combine multiple external knowledge sources including reverse image search, web search using Google and GPT, and climate-specific fact-checking websites to verify image–claim pairs. 3. Use GPT-4o, the latest model with both vision and language understanding, to analyze image–text pairs in context and classify them as accurate, misleading, false, or unverifiable. The goal is to build a system that improves disinformation detection and supports public understanding of climate science in an online environment where facts are frequently distorted. 2 Prior Work The spread of climate disinformation has become a major concern as people increasingly rely on online platforms for news and scientific information. Researchers have addressed this problem from different angles, developing methods to detect misleading narratives, creating datasets that capture the specific features of climate communication, and testing the capabilities of large language models in this domain. The study by Coan et al. [ 3 ] used a hierarchical deep learning model trained on more than eighty seven thousand annotated paragraphs to classify over two hundred forty nine thousand documents into a detailed taxonomy of climate claims. Their analysis revealed patterns that linked certain misinformation themes to funding from conservative donors. Meddeb et al. [ 7 ] focused on the French media landscape, creating a dataset of more than two thousand three hundred climate-related news articles. By combining a CamemBERT model with handcrafted linguistic features, they achieved strong classification results and applied LIME to make the model’s reasoning more transparent. In a broader perspective, Herasimenka et al. [ 5 ] reviewed thirty eight studies on manipulative climate communication across social media. They found that most work concentrated on platforms such as Twitter and Facebook, rarely tested interventions, and paid little attention to visual platforms like Instagram or TikTok. Other researchers have developed specialised detection systems. Rojas et al. [ 8 ] introduced Augmented CARDS, which first distinguished between convinced and contrarian tweets before assigning contrarian content to specific categories. Applied to more than five million climate tweets, this approach improved both binary and detailed classification compared to earlier systems. Fore et al. [ 4 ] examined how large language models can be influenced by false climate claims and evaluated strategies to reduce such effects. They showed that unlearning techniques could substantially lower the rate of contradictions, while retrieval-augmented generation improved truthfulness without modifying model weights. Allaham et al. [ 1 ] compared sixteen large language models against expert-labelled data, showing that fine-tuning GPT 3.5 turbo led to higher agreement with experts than both GPT 4o and a BERT-based classifier. Beyond work that focuses specifically on climate content, a growing body of research in multimodal misinformation detection has explored how external evidence can improve factual verification. LEMMA by Xuan et al. [ 11 ] demonstrated how combining the internal reasoning of GPT 4V with documents retrieved through web search and image provenance from reverse image search can enhance classification accuracy. The CMIE framework [ 6 ] tackled the challenge of out-of-context misinformation by retrieving contextual evidence from the web and image titles, identifying semantic links between image–text pairs, and weighting the most relevant evidence. Shen et al. [ 9 ] developed GAMED, which refined multimodal features through expert networks, balanced the importance of each modality, and incorporated semantic knowledge from ERNIE2.0 to achieve high accuracy on benchmark datasets. Wu et al. [ 10 ] proposed DECEPTIONDECODED, a benchmark where each claim is paired with a trusted reference article. Their results showed that even when evidence is available, leading models often fail to use it effectively to identify misleading intent. Together, these studies show that climate disinformation detection has advanced considerably, yet most work remains focused on text, specific languages, or a limited set of platforms. At the same time, multimodal approaches that integrate external evidence have rarely been applied to climate-related claims, and even when such evidence is available, models often struggle to make full use of it. This gap points to the need for approaches that bring together climate-specific detection with effective external knowledge integration, enabling more accurate and context-aware verification across different media formats. 3 Approach 3.1 Dataset The experiments in this work are based on the CliME (Climate Change Multimodal Evaluation) dataset [ 2 ] , which contains 2,579 climate-related social media posts collected from Twitter and Reddit. Each post consists of a short textual claim, an accompanying image, and a manually validated description. The claim corresponds to the main text of the post, along with the supportive image. The descriptions, generated using DeepSeek Janus Pro and reviewed by experts, summarize the overall message conveyed by both the claim and the image. The dataset covers a diverse range of multimodal content, including memes, infographics, and posts expressing either skepticism or support toward climate change. Since the original dataset does not include factuality labels, we automatically extracted labels by jointly considering the claim, the accompanying image, and the expert-validated description. The extraction process was carried out using GPT-4o with four distinct role based prompt designs, each