ClimateViz : A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts
Ruiran Su, Jiasheng Si, Zhijiang Guo, Janet B. Pierrehumbert · 2025
Read ClimateViz : A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts by Ruiran Su, Jiasheng Si, Zhijiang Guo, Janet B. Pierrehumbert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08700.txt.
Introduction
Ruiran Su and co-authors build a benchmark for checking facts against scientific charts in this 2025 paper. Fact-checking research has leaned on text and tables. Charts carry much of science's hard evidence, though. Their set, ClimateViz, pairs claims with expert-made climate charts. Tens of thousands of claims link to nearly three thousand charts. Each claim is labeled supported, refuted, or undecidable from its chart.
Explanations come as small knowledge graphs of trends and comparisons. Top multimodal models were tested with no and few examples. Current models struggled, the authors report. Reading axes, trends, and causes from charts is harder than it looks. The benchmark gives the field a hard, fair test. It is a dataset and evaluation paper.
Catalog record
| CREATOR | Ruiran Su, Jiasheng Si, Zhijiang Guo, Janet B. Pierrehumbert |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08700
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ClimateViz : A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts Ruiran Su, Jiasheng Si, Zhijiang Guo, Janet B. Pierrehumbert Abstract Scientific fact-checking has largely focused on textual and tabular sources, neglecting scientific charts—a primary medium for conveying quantitative evidence and supporting statistical reasoning in research communication. We introduce ClimateViz , the first large-scale benchmark for scientific fact-checking grounded in real-world, expert-curated scientific charts. ClimateViz comprises 49,862 claims paired with 2,896 visualizations, each labeled as support, refute, or not enough information. To enable interpretable verification, each instance includes structured knowledge graph explanations that capture statistical patterns, temporal trends, spatial comparisons, and causal relations. We conduct a comprehensive evaluation of state-of-the-art multimodal large language models, including proprietary and open-source systems, under zero-shot and few-shot settings. Our results show that current models struggle to perform fact-checking when statistical reasoning over charts is required: even the best-performing systems, such as Gemini 2.5 and InternVL 2.5, achieve only 76.2–77.8% accuracy in label-only output settings, which is far below human performance (89.3% and 92.7%). While few-shot prompting yields limited improvements, explanation-augmented outputs significantly enhance performance in some closed-source models, notably o3 and Gemini 2.5. We released our dataset and code alongside the paper. 1 1 1 https://github.com/Albasu120491/ClimateViz 1 Introduction Scientific fact-checking—the task of assessing the validity of scientific claims through cross-referencing with established literature, empirical observations, or experimental data ( Wadden et al. 2020 ; Vladika and Matthes 2023 ) —is essential for maintaining the integrity of research findings, combating misinformation, and preserving public confidence in scientific discourse ( Wadden et al. 2022 ) . However, the rapid accumulation of scholarly findings and the increasing demand for domain-specific expertise often exceed the capacity of manual verification, making scientific fact-checking a critical focus in the NLP community. Figure 1: (a) A sample from ClimateViz showing a scientific chart, caption, claim, and its label. (b) Knowledge graph creation pipeline: raw triplets are extracted by a multimodal LLM and canonicalized. (c) Subgraph of relevant facts representing structured reasoning from the chart to the claim. Significant progress has been made with the development of benchmarks such as SciFact ( Wadden et al. 2020 ) , SciFact-Open ( Wadden et al. 2022 ) , and SciTab ( Lu et al. 2023 ) . Despite these advances, existing resources exhibit critical limitations in scope. Specifically, prior benchmarks predominantly focus on verifying scientific claims against textual ( Wadden et al. 2020 ; Diggelmann et al. 2021 ; Sarrouti et al. 2021 ; Saakyan et al. 2021 ) or tabular ( Mohr et al. 2022 ; Lu et al. 2023 ) evidence, particularly from literature abstracts or tables. These claims are typically validated using semantic or structural logical reasoning between claims and corresponding evidence. In contrast, real-world scientific findings often involve claims that are intrinsically tied to quantitative data. In such contexts, charts serve as both visual and statistical representations, summarizing complex numerical information, revealing trends, supporting quantitative reasoning, and effectively communicating scientific insights ( Huang et al. 2024 ) . However, chart-based verification is largely absent from prior fact-checking benchmarks—despite requiring explicit chart understanding and statistical reasoning over visualized data, beyond what semantic or structural inference alone can handle ( Akhtar et al. 2023b ; Akhtar et al. 2024 ) . More specifically, scientific claims often involve not only raw data observations but also interpretations of patterns, anomalies, correlations, and statistical aggregation. For example, the claim in Figure 1 implies a statistical relationship between ice sheet mass loss and sea-level rise . Verifying such information is inherently challenging and requires: (i) visual understanding of charts with non-uniform temporal granularity (e.g., dual-axis time series), spatial dimensions (e.g., geographic insets), and uncertainty bands; (ii) nuanced statistical reasoning to interpret temporal trends, quantify cumulative change, and relate ice mass loss to sea-level rise; and (iii) cross-modal reasoning to establish logical coherence between the linguistic embedding of scientific claims and the perceptual features of the chart. This example illustrates the depth of reasoning needed to understand complex scientific findings, yet the absence of benchmarks that require such statistical reasoning severely limits the evaluation of models designed for general scientific understanding. In this paper, we introduce ClimateViz , a novel dataset sourced from reputable climate institutions (e.g., the National Oceanic and Atmospheric Administration 2 2 2 https://www.noaa.gov/ and the UK Met Office 3 3 3 https://www.metoffice.gov.uk/ ), designed to advance scientific fact-checking with a focus on statistical reasoning over charts. ClimateViz comprises 49,862 claim–chart–knowledge graph (KG) triplets, each accompanied by relevant metadata (e.g., chart caption, chart type). Claims are systematically constructed based on 2,896 expert-curated scientific charts and annotated with one of three labels: support , refute , or not enough information (NEI) . A key innovation of the dataset is the inclusion of chart-specific knowledge graphs that provide structured, interpretable explanations for fact verification. These KGs capture core scientific information aligned with the claims, such as quantities, trends, spatial and temporal contexts, and causal relationships—enabling explicit multi-hop reasoning. To construct ClimateViz , we launched a large-scale project on the citizen science platform Zooniverse 4 4 4 https://www.zooniverse.org/ , which ensured scientifically literate annotations. Each chart was independently annotated by six contributors, and the resulting claims were reviewed and verified by two domain experts to ensure correctness and quality. We utilize ClimateViz as a diagnostic benchmark to evaluate the zero-shot and in-context learning capability across a varied range of state-of-the-art models, including open- and closed-source language models, and chart-based vision-language models. Comprehensive experiments reveal that all models struggle with verifying claims over scientific charts when statistical reasoning is necessary. Furthermore, the integration of a chart-specific knowledge graph proves beneficial when models are provided with both scientific charts and supplementary KG data. In addition, while models generate semantically plausible explanatory triplets, they typically fail to produce properly canonicalized outputs. These findings underscore the unique challenges posed by ClimateViz and highlight the need for further advances in models capable of statistical reasoning, structured explanation generation, and deep scientific understanding from visual evidence. Dataset Modality Domain #Claims Source SciFact ( Wadden et al. 2020 ) Text Biomedical 1.4K Medical literature Climate-FEVER ( Diggelmann et al. 2021 ) Text Climate 1.5K Wikipedia HealthVer ( Sarrouti et al. 2021 ) Text Health 14K News COVID-Fact ( Saakyan et al. 2021 ) Text COVID-19 4.1K News CoVERT ( Mohr et al. 2022 ) Table COVID-19 10K Social media SciTab ( Lu et al. 2023 ) Table CS 1.2K Scientific papers ClimateViz Chart Climate 49.8K Expert-curated scientific charts Table 1: Comparison of scientific fact-checking datasets by modality, domain, claim volume, and source. ClimateViz is the first chart-based benchmark at this scale, grounded in real expert-curated scientific charts. 2 Related Work Scientific Fact-checking Benchmarks. Several benchmarks have been proposed to advance automated scientific fact-checking, primarily focusing on textual evidence (see Table 1 ). SciFact ( Wadden et al. 2020 ) introduced claim verification against biomedical research abstracts, while Climate-FEVER ( Diggelmann et al. 2021 ) extended claim verification to the climate domain using Wikipedia articles. Other datasets, such as HealthVer ( Sarrouti et al. 2021 ) and COVID-Fact ( Saakyan et al. 2021 ) , collected claims from health news and pandemic-related sources, respectively. More recently, CoVERT ( Mohr et al. 2022 )