Is this chart lying to me? Automating the detection of misleading visualizations

Jonathan Tonglet, Jan Zimny, Tinne Tuytelaars, Iryna Gurevych · 2025

Read Is this chart lying to me? Automating the detection of misleading visualizations by Jonathan Tonglet, Jan Zimny, Tinne Tuytelaars, Iryna Gurevych (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-21675.txt.

Introduction

Jonathan Tonglet, Jan Zimny, Tinne Tuytelaars, and Iryna Gurevych build tools to catch charts that lie, in a 2025 paper. Misleading charts are a strong driver of misinformation online. They break design rules to distort data. Both people and large AI models get fooled by them. Automatic detection could protect readers at scale. The field lacked a large, open dataset for training. The team fills that gap.

Their benchmark holds over twenty-six hundred real charts. Each is labeled with which of twelve tricks it uses. A synthetic set adds nearly fifty-eight thousand generated charts. Those were built from real data tables with a plotting library. The team tested top multimodal models, rule systems, and axis readers. All struggled with the task. The authors release the data and code openly. They frame detection as an unsolved, practical problem.

Catalog record

CREATORJonathan Tonglet, Jan Zimny, Tinne Tuytelaars, Iryna Gurevych
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-21675

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Is this chart lying to me? Automating the detection of misleading visualizations Jonathan Tonglet, Jan Zimny, Tinne Tuytelaars, Iryna Gurevych Abstract Misleading visualizations are a potent driver of misinformation on social media and the web. By violating chart design principles, they distort data and lead readers to draw inaccurate conclusions. Prior work has shown that both humans and multimodal large language models (MLLMs) are frequently deceived by such visualizations. Automatically detecting misleading visualizations and identifying the specific design rules they violate could help protect readers and reduce the spread of misinformation. However, the training and evaluation of AI models has been limited by the absence of large, diverse, and openly available datasets. In this work, we introduce Misviz, a benchmark of 2,604 real-world visualizations annotated with 12 types of misleaders. To support model training, we also release Misviz-synth, a synthetic dataset of 81,814 visualizations generated using Matplotlib and based on real-world data tables. We perform a comprehensive evaluation on both datasets using state-of-the-art MLLMs, rule-based systems, and fine-tuned classifiers. Our results reveal that the task remains highly challenging. We release Misviz, Misviz-synth, and the accompanying code. 1 1 1 github.com/UKPLab/arxiv2025-misviz 1 Introduction Misleading visualizations are charts that distort the underlying data, typically by violating design principles, leading readers to draw inaccurate conclusions ( Tufte and Graves-Morris 1983 ; Pandey et al. 2015 ; Lauer and O’Brien 2020 ; McNutt et al. 2020 ; Lo et al. 2022 ; Lisnic et al. 2023 ; Lan and Liu 2025 ) . While many arise from unintentional design errors, misleading visualizations are also deliberately crafted by malicious actors to spread disinformation and manipulate public understanding, especially during crises such as the COVID-19 pandemic, where misleading charts circulated widely on social media ( Correll and Heer 2017 ; Tartaglione and de Wit 2025 ; Lisnic et al. 2023 ) . Prior work has shown that both humans ( Pandey et al. 2014 ; Pandey et al. 2015 ; O’Brien and Lauer 2018 ; Yang et al. 2021 ; Ge et al. 2023 ; Rho et al. 2023 ) and MLLMs ( Bendeck and Stasko 2025 ; Chen et al. 2025 ; Pandey and Ottley 2025 ; Tonglet et al. 2025 ) are deceived by such visualizations in question-answering tasks. The deceptive features in these charts, or misleaders ( Lisnic et al. 2023 ) , often reside in subtle details easily missed by readers, such as axis tick intervals. Furthermore, misleaders are highly diverse: the latest taxonomies identify over 70 distinct types spanning a wide variety of chart types, including bar charts, pie charts, and maps ( Lo et al. 2022 ; Lan and Liu 2025 ) . In some cases, more than one misleader affects the same visualization ( Lo et al. 2022 ) . Figure 1 shows 12 real-world examples. Figure 1: Examples of the 12 types of misleaders included in Misviz. Appendix A explains how these visualizations misrepresent their underlying data table. Automatically classifying whether a visualization is misleading, and identifying which misleaders affect it, if any, can enable timely warnings to chart designers and readers and help prevent the spread of misinformation. This task is framed as a multi-label classification problem. While early work relied on rule-based systems called linters ( Hopkins et al. 2020 ; Fan et al. 2022 ) , recent studies have explored the use of MLLMs ( Lo and Qu 2025 ; Alexander et al. 2024 ) . However, these approaches were evaluated on distinct datasets, which are either small or closed-access, limiting comparability and hindering progress. In this work, we introduce Misviz, a large, diverse, and open benchmark comprising 2,604 real-world visualizations spanning 12 types of misleaders. It reflects scenarios in which vulnerable readers encounter misleading visualizations online. In Misviz, 70% of the visualizations contain up to three misleaders, while the remainder are non-misleading. To support model training, we also release Misviz-synth, a synthetic dataset generated with Matplotlib ( Hunter 2007 ) using real-world data tables. Misviz-synth simulates scenarios where chart designers seek assistance in identifying misleaders unintentionally introduced in their visualizations. It includes not only the visualizations but also their data tables and axis metadata, enabling the training of chart de-rendering models. We conduct extensive experiments with three approaches: (a) state-of-the-art MLLMs, (b) a new rule-based linter that inspect axis metadata for design rule violations, and (c) new classifiers that take the visualization alone or in combination with the axis metadata as input. For (b) and (c), we fine-tune DePlot ( Liu et al. 2023 ) to extract axis metadata. Our experiments address the following research questions (RQs). RQ1 : Which type of model performs best on real-world or synthetic instances? RQ2 : Can models trained on synthetic instances generalize to real-world cases? RQ3 : Can models fine-tuned for axis extraction on synthetic instances generalize to real-world cases? Our results show that MLLMs perform best on real-world visualizations, while linters and fine-tuned classifiers outperform them on synthetic ones, benefiting from the availability of training data for both axis extraction and misleader detection. While the fine-tuned DePlot can extract axes from Misviz-synth, it does not generalize well to Misviz, reducing the performance of the linter and classifier. In summary, our contributions are as follows: (1) we introduce Misviz and Misviz-synth, the first large scale open datasets for misleading visualization detection, (2) we propose a new linter and a new classification method that combines image and extracted axis metadata as input, (3) we conduct comprehensive evaluation and error analysis. 2 Related work Dataset Instances Misleader types Chart types % non-misleading Open access Real-world Multi-label Axis, table MISCHA-QA ( Arif et al. 2024 ) 8,201 4 3 49 ✓ ✗ ✗ ✗ DCDM ( Maciborski et al. 2025 ) 24,480 5 3 51 ✓ ✗ ✗ ✗ Alexander et al. 2024 - design misleaders 1,460 7 > 5 50 ✗ ✓ ✗ ✗ Lo and Qu 2025 150 21 > 5 16 ✓ ✓ ✗ ✗ Misvisfix ( Das and Mueller 2025 ) 450 74 > 5 20 ✓ ✓ ✓ ✗ Misviz-synth (ours) 81,814 11 3 22 ✓ ✗ ✗ ✓ Misviz (ours) 2,604 12 > 5 31 ✓ ✓ ✓ ✗ Table 1: Existing datasets for misleading visualization detection. The first attempts to detect misleading visualizations relied on rule-based systems called linters ( McNutt and Kindlmann 2018 ; Hopkins et al. 2020 ; Chen et al. 2022 ) . These linters assume the availability of the underlying data table or the chart code, which restricts their applicability to real-world scenarios. Fan et al. 2022 and Biselli et al. 2025 overcome these limitations by extracting tables with OCR tools from real-world visualizations before applying rule checks. However, the accuracy of these real-world linters depends heavily on the quality of the intermediate OCR step, which can vary widely ( Biselli et al. 2025 ) . Real-world linters have previously been evaluated in small-scale user studies, with human-in-the-loop to correct OCR errors. Others have explored the potential of MLLMs for the task. Lo and Qu 2025 evaluated four MLLMs with different prompts on a dataset of 150 real-world visualizations, sourced from the corpus of Lo et al. 2022 for misleading cases. They found that detection accuracy decreased as more

Read the complete text