Do LLMs Have Visualization Literacy? An Evaluation on Modified Visualizations to Test Generalization in Data Interpretation
\authororcid Jiayi Hong0000-0002-1332-5045, \authororcid Christian Seto0009-0009-0808-2470, \authororcid Arlen Fan0000-0002-8376-2881, and \authororcid Ross Maciejewski0000-0001-8803-6355 · 2025
Read Do LLMs Have Visualization Literacy? An Evaluation on Modified Visualizations to Test Generalization in Data Interpretation by \authororcid Jiayi Hong0000-0002-1332-5045, \authororcid Christian Seto0009-0009-0808-2470, \authororcid Arlen Fan0000-0002-8376-2881, and \authororcid Ross Maciejewski0000-0001-8803-6355 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16277.txt.
Introduction
Jiayi Hong and co-authors test whether large language models can read charts as well as people do. Their paper from 2025 asks if models could one day help grade data graphics, a task where human test data is slow and costly to gather. Chart reading skill, called visualization literacy, is the trait under test.
Two models, GPT-4 and Gemini, took a changed form of a fifty three item human test. Both fell short of the scores the general public reached on the same test. The models also leaned on stored knowledge over what the chart in front of them showed, a habit that skews grading. The authors set base marks for model chart skill and warn against quick trust.
Catalog record
| CREATOR | \authororcid Jiayi Hong0000-0002-1332-5045, \authororcid Christian Seto0009-0009-0808-2470, \authororcid Arlen Fan0000-0002-8376-2881, and \authororcid Ross Maciejewski0000-0001-8803-6355 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 112 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-16277
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Do LLMs Have Visualization Literacy? An Evaluation on Modified Visualizations to Test Generalization in Data Interpretation \authororcid Jiayi Hong0000-0002-1332-5045, \authororcid Christian Seto0009-0009-0808-2470, \authororcid Arlen Fan0000-0002-8376-2881, and \authororcid Ross Maciejewski0000-0001-8803-6355 Abstract In this paper, we assess the visualization literacy of two prominent Large Language Models (LLMs): OpenAI’s Generative Pretrained Transformers (GPT), the backend of ChatGPT, and Google’s Gemini, previously known as Bard, to establish benchmarks for assessing their visualization capabilities. While LLMs have shown promise in generating chart descriptions, captions, and design suggestions, their potential for evaluating visualizations remains under-explored. Collecting data from humans for evaluations has been a bottleneck for visualization research in terms of both time and money, and if LLMs were able to serve, even in some limited role, as evaluators, they could be a significant resource. To investigate the feasibility of using LLMs in the visualization evaluation process, we explore the extent to which LLMs possess visualization literacy—a crucial factor for their effective utility in the field. We conducted a series of experiments using a modified 53-item Visualization Literacy Assessment Test (VLAT) for and . Our findings indicate that the LLMs we explored currently fail to achieve the same levels of visualization literacy when compared to data from the general public reported in VLAT, and LLMs heavily relied on their pre-existing knowledge to answer questions instead of utilizing the information provided by the visualization when answering questions. \onlineid 0 \vgtccategory Research \vgtcpapertype Theoretical & Empirical \authorfooter Jiayi Hong, Christian Seto, Arlen Fan, and Ross Maciejewski are with Arizona State University, USA. E-mail: {jhong76 | cseto | afan5 | rmacieje}@asu.edu. Introduction keywords Large Language Model, Visualization Literacy, Evaluation Study As Generative Pretrained Transformers (GPT) and other Large Language Models (LLMs) continue to advance [ 35 ] , their impact on the field of visualization is becoming increasingly profound. Previous research has explored implementing LLMs in diverse visualization tasks, e. g., visualization creation [ 34 , 16 ] , data encoding [ 36 ] , caption generation [ 26 ] , design advice [ 23 ] , and visualization education [ 11 ] . Moreover, scientists have begun postulating the use of LLMs in evaluation studies (e. g., [ 1 ] ). Such evaluations could also be useful for visualization sanity checks, providing feedback on visualization readability [ 8 ] . Typically, evaluating visualizations incurs high costs related to human labor. If LLMs could support visualization experts in evaluating visual representations and visual systems to a similar level of proficiency as the general public, they could offer significant cost savings [ 18 ] . However, before experts can use LLMs to evaluate visualizations, it is crucial to ensure that these models possess adequate visualization literacy . In this paper, we propose a template to evaluate whether current LLMs have visualization literacy, which encompasses their ability to read, understand, and interpret visual representations, a crucial skill for evaluating visualizations. Such a template is useful in determining when such technology has progressed to the point that we can consider its use in key tasks, such as evaluation. To investigate the practicability of employing LLMs in the visualization assessment process, we systematically explored their visualization literacy. We assumed that an LLM is qualified to assist with visualization evaluation if it can meet the benchmark, indicating that its visualization literacy (VL) is comparable with that of the general public. Researchers have developed benchmarks for evaluating humans’ VL such as the Visualization Literacy Assessment Test (VLAT) [ 24 ] and its abridged version, Mini-VLAT [ 31 ] to gauge individuals’ ability to interpret visual representations. Yet, these static assessments cannot be directly adopted for LLMs, as these models may have already learned from such benchmarks. To this end, we developed a comprehensive evaluation template, incorporating a modified version of the 53-item VLAT to evaluate the visualization literacy of LLMs, specifically focusing on 1 1 1 https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo and 2 2 2 https://ai.google.dev/models in this work. This test covered 12 visualizations and 8 visual tasks. In a series of studies, we employed this modified test to evaluate the performance of LLMs with a primary focus on the accuracy rate as a metric. Additionally, we investigated the limitations of LLMs in reading and understanding visualizations, as well as the comparative costs between LLMs and humans. Our primary research questions are as follows: • RQ1 : To what extent do LLMs have visualization literacy? • RQ2 : What are LLMs’ limitations in interpreting visualizations? • RQ3 : What are the cost differences between LLMs and humans in interpreting visualizations and answering related questions? We addressed RQ1 by investigating the performances across different visualizations, tasks, and models. We also explored the extent to which LLMs’ responses were influenced by their knowledge base versus their ability to extract and reason about data within visualizations. Moreover, we measured the costs associated with using LLMs in the experiments and compared them with the results of previously tested humans . For RQ2 , we examined the necessity of providing answer choices for LLMs and the impact of contextual information when it is kept in or removed from visualizations. Additionally, for RQ3 , we measured the costs in terms of time and money we spent in the experiments to provide practitioners with an estimate of the costs for future usage. Our results indicate that compared to humans, both and lack visualization literacy. Their performance varied across specific tasks and visualizations, and they heavily relied on their pre-existing knowledge base rather than the information present in the visualizations. Moreover, current LLMs seemed to perform poorly in open-ended visualization-related questions, and interestingly, decontextualization tended to enhance their performances, with showing more pronounced improvement compared to . Both LLMs are cost-effective compared with humans considering time and money, with showing greater cost-effectiveness than . In summary, our contributions are • A template and methodology for evaluating the visualization literacy of LLMs; • A series of studies to examine the visualization literacy of LLMs; • An exploration of the limitations of LLMs’ in reading and interpreting visualizations; • A discussion of insights on the future potential of leveraging LLMs in evaluating data visualizations. 1 Related Work This work is grounded in theoretical foundations from visualization literacy, current applications of LLMs for visualization, and evaluation methods for LLMs. 1.1 Visualization Literacy There has been a growing focus on exploring visualization literacy (VL) in society [ 6 ] due to the widespread adoption of visualization across many domains [ 2 ] . Visualization literacy is vital, as it denotes the ability to read and interpret visual representations [ 31 ] . It enables individuals to comprehend unfamiliar charting techniques more easily [ 5 ] , which is essential in evaluating new visualizations. To measure visualization literacy quantitatively, researchers have been creating evaluation methods. For example, Boy et al. [ 7 ] spearheaded this endeavor by conceptualizing visualization literacy and introducing a method based on item response theory (IRT) [ 3 ] for evaluating an individual’s proficiency. Lee et al. [ 24 ] developed and refined the Visualization Literacy Assessment Test (VLAT), which consists of 53 multiple-choice questions to assess visualization literacy among non-expert users. They highlighted a positive correlation between visualization literacy and the ability to interpret unfamiliar visualizations, suggesting