Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language Models

Alexa R. Tartaglini, Satchel Grant, Daniel Wurgaft, Christopher Potts, Judith E. Fan · 2025

Read Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language Models by Alexa R. Tartaglini, Satchel Grant, Daniel Wurgaft, Christopher Potts, Judith E. Fan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21740.txt.

Introduction

Alexa R. Tartaglini and four co-authors dissect why vision-language models misread charts, a skill science and journalism lean on daily. A wrong chart answer could stem from the eyes, from the handoff between vision and language parts, or from the language reasoning itself. The team built a task suite, FUGU, to separate those suspects cleanly.

Probing three widely used models with activation patching and linear readouts, they find an early culprit: models often generate wrong coordinates for individual data points. Those seed errors poison the final answer, even though the correct coordinates can still be read from the vision encoder's inner activity. Feeding correct coordinates rescues simple tasks but hurts statistics across many points, and fine-tuning on the suite still falls short of the ceiling, pointing at architectural limits in today's designs.

Catalog record

CREATORAlexa R. Tartaglini, Satchel Grant, Daniel Wurgaft, Christopher Potts, Judith E. Fan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21740

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language Models Alexa R. Tartaglini, Satchel Grant, Daniel Wurgaft, Christopher Potts, Judith E. Fan Abstract Data visualizations are vital components of many scientific articles and news stories. Current vision-language models (VLMs) still struggle on basic data visualization understanding tasks, but the causes of failure remain unclear. Are VLM failures attributable to limitations in how visual information in the data visualization is encoded, how information is transferred between the vision and language modules, or how information is processed within the language module? We developed FUGU , a suite of data visualization understanding tasks, to precisely characterize potential sources of difficulty (e.g., extracting the position of data points, distances between them, and other summary statistics). We used FUGU to investigate three widely used VLMs ( LLaMA-3.2 , LLaVA-OneVision , and InternVL3 ). To diagnose the sources of errors produced by these models, we used activation patching and linear probes to trace information flow through models across a variety of prompting strategies. We found that some models fail to generate the coordinates of individual data points correctly, and these initial errors often lead to erroneous final responses. When these models are provided with the correct coordinates, performance improves substantially, suggesting that the downstream mathematical reasoning steps performed in the language module are sound. Moreover, even when the model generates an incorrect response, the correct coordinates can be successfully read out from the latent representations in the vision encoder, suggesting that the source of these errors lies in the vision-language handoff. We further found that while providing correct coordinates helps with tasks involving one or a small number of data points, it generally worsens performance for tasks that require extracting statistical relationships across many data points (e.g., correlations, clusters). Finetuning models on FUGU also fails to yield ceiling performance. These findings point to fundamental architectural constraints in current VLMs that might pose significant challenges for reliable data visualization understanding. 1 1 1 Dataset & code are available at https://github.com/cogtoolslab/fugu . 1 Introduction Data visualizations — also known as graphs, charts, and plots — encode quantitative information using a combination of graphical elements, numerals, and words. They are powerful ways of helping people to rapidly grasp quantitative patterns and trends across many fields, including science, journalism, and business ( Franconeri et al. 2021 ) . To build AI systems that can operate effectively in all of these contexts will require them to be able to parse a knowledge base that blends graphics and text. However, current vision-language models (VLMs) have not yet reached human-level performance on even basic data visualization understanding tasks ( Verma et al. 2024 ; Pandey & Ottley 2025 ) . This performance gap raises the question motivating the current work: when and why do VLMs fail to parse data visualizations? Many current VLMs consist of multiple modules that each operate over text and image inputs separately, though different VLMs embody different architectural commitments as to how to combine latent information from visual and linguistic inputs ( Grattafiori et al. 2024 ; Deitke et al. 2024 ; Team 2024 ; Chen et al. 2024 ; Laurençon et al. 2024 ; Lu et al. 2024 ) . For any given VLM, the failures might thus stem from multiple potential sources: the visual component (e.g., failure to encode the graph), the language component (e.g., failure to reason over the visual information), or the transfer of information between the vision and language components (e.g., failure to align visual and linguistic latent spaces). However, distinguishing between these different failure modes is not trivial given the complexity of many multimodal reasoning tasks and many prevalent model architectures. To more precisely identify these potential points of failure in data visualization understanding, we developed FUGU ( Fundamentals of Graph Understanding ). FUGU is a novel benchmark consisting of 3,968 questions paired with scatter plots designed to probe the foundational spatial reasoning and mathematical skills that are needed to parse any data visualization, including the ability to report the values of data points, estimate distances between them, and estimate summary statistics, including the minimum, maximum, and mean. FUGU consists of entirely novel, synthetically generated visualizations that enable more targeted investigation of these foundational skills, allowing us to diagnose where a model’s understanding breaks down. Towards this end, we leveraged multiple mechanistic interpretability techniques to locate the information bottlenecks that hinder model performance. We use activation patching (also known as causal interventions; Vig et al. 2020 ; Finlayson et al. 2021 ; Geiger et al. 2021 ; Geiger et al. 2024 ; Meng et al. 2022 ; Wang et al. 2023 ) to test whether a specific component of a model encodes behaviorally relevant information. In addition, we use linear probes — lightweight classifiers on internal model representations — to evaluate the presence of task-relevant information (e.g., the location of a data point). While this approach could be applied to any VLM architecture, in this work we conducted a thorough investigation of three models: LLaMA-3.2B 11B ( Grattafiori et al. 2024 ) , LLaVA-OneVision 7B ( Li et al. 2024 ) , and InternVL3 14B ( Zhu et al. 2025 ) . These VLMs are widely used, performant, and vary along key design choices: architecture type (cross-attention based vs. decoder-only), vision encoder, LLM backbone, and pretraining data. Our findings serve to identify the limitations of existing VLMs and provide insight towards building better VLMs. 2 Related Work Figure 1: A) The Fundamentals of Graph Understanding ( FUGU ) task suite and dataset. The top row (task suite) displays example prompts for each of our five tasks. The bottom row (sample size) shows example scatter plots from our dataset for each number of data points ( n ). FUGU pairs the five tasks with all applicable scatter plots, resulting in 3,968 unique <task, image> pairs. B) Visualization of the causal intervention method. We perform causal interventions on a counterfactual pair of scatter plots by passing each separately through the model with the same prompt, extracting embeddings from the vision encoder, and swapping the visual embeddings corresponding to a meaningful subset of image tokens (like a data point) from the source to the target run. If this successfully replaces model output behavior on target with the source prediction, then we have located a causally important part of the representation for that task. A growing body of work at the intersection of computer vision and visualization aims to develop AI systems that understand data visualizations as well as humans can. Our paper builds most directly on prior work benchmarking data visualization understanding in AI systems, as well as recent advances in mechanistic interpretability methods applied to large language models and vision-language models. Machine data visualization understanding Several visual question answering benchmarks currently exist to assess progress towards this goal, including FigureQA ( Kahou et al. 2017 ) , DVQA ( Kafle et al. 2018 ) , PlotQA ( Methani et al. 2020 ) , ChartQA ( Masry et al. 2022 ) , and ChartQA-Pro ( Masry et al. 2025 ) . While early efforts like FigureQA found initial success on synthetic data visualization understanding tasks, evaluations were conducted exclusively with models only capable of drawing from a

Read the complete text