Charts-of-Thought: Enhancing LLM Visualization Literacy Through Structured Data Extraction

\authororcid Amit Kumar Das0000-0002-2600-8321, \authororcid Mohammad Tarun0009-0001-1708-9924, \authororcid Klaus Mueller0000-0002-0996-8590 · 2025

Read Charts-of-Thought: Enhancing LLM Visualization Literacy Through Structured Data Extraction by \authororcid Amit Kumar Das0000-0002-2600-8321, \authororcid Mohammad Tarun0009-0001-1708-9924, \authororcid Klaus Mueller0000-0002-0996-8590 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04842.txt.

Introduction

Amit Kumar Das, Mohammad Tarun, and Klaus Mueller test how well large language models read charts, in a paper from 2025. Reading charts is a skill schools test in people, called visualization literacy. The team gave three top models a standard chart test. They tried both plain prompts and a new staged prompt method of their own.

The method, Charts-of-Thought, makes the model pull out the data, check it, and only then answer. Scores rose for all three models under the method. The best model beat the human average by a wide margin. It aced some chart types that had tripped up models before. The authors say staged prompts unlock skill the models already hold. The trick could also help people who find charts hard to read.

Catalog record

CREATOR\authororcid Amit Kumar Das0000-0002-2600-8321, \authororcid Mohammad Tarun0009-0001-1708-9924, \authororcid Klaus Mueller0000-0002-0996-8590
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04842

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Charts-of-Thought: Enhancing LLM Visualization Literacy Through Structured Data Extraction \authororcid Amit Kumar Das0000-0002-2600-8321, \authororcid Mohammad Tarun0009-0001-1708-9924, \authororcid Klaus Mueller0000-0002-0996-8590 Abstract This paper evaluates the visualization literacy of modern Large Language Models (LLMs) and introduces a novel prompting technique called Charts-of-Thought. We tested three state-of-the-art LLMs (Claude-3.7-sonnet, GPT-4.5-preview, and Gemini-2.0-pro) on the Visualization Literacy Assessment Test (VLAT) using standard prompts and our structured approach. The Charts-of-Thought method guides LLMs through a systematic data extraction, verification, and analysis process before answering visualization questions. Our results show Claude-3.7-sonnet achieved a score of 50.17 using this method, far exceeding the human baseline of 28.82. This approach improved performance across all models, with score increases of 21.8% for GPT-4.5, 9.4% for Gemini-2.0, and 13.5% for Claude-3.7 compared to standard prompting. The performance gains were consistent across original and modified VLAT charts, with Claude correctly answering 100% of questions for several chart types that previously challenged LLMs. Our study reveals that modern multimodal LLMs can surpass human performance on visualization literacy tasks when given the proper analytical framework. These findings establish a new benchmark for LLM visualization literacy and demonstrate the importance of structured prompting strategies for complex visual interpretation tasks. Beyond improving LLM visualization literacy, Charts-of-Thought could also enhance the accessibility of visualizations, potentially benefiting individuals with visual impairments or lower visualization literacy. \onlineid 0 \vgtccategory Research \vgtcpapertype please specify \authorfooter Amit Kumar Das and Klaus Mueller are with the Computer Science Department, Stony Brook University, USA (E-mails: [email protected], [email protected]). Mohammad Tarun is with East West University, Bangladesh (e-mail: [email protected]). Introduction keywords Visualization Literacy, Large Language Models, Charts-of-Thoughts, Data Extraction As Generative Pre-trained Transformers (GPT) and other Large Language Models (LLMs) continue to advance [ 40 ] , their impact on the field of visualization is becoming increasingly significant. Previous research has explored implementing LLMs in diverse visualization tasks, including visualization creation [ 3 , 39 ] , data encoding [ 44 ] , caption generation [ 30 ] , design advice [ 23 ] , and visualization education [ 58 ] . Researchers have also begun investigating whether LLMs can evaluate visualizations [ 2 ] . Such evaluations could support visualization sanity checks and provide feedback on visualization readability [ 10 ] . Traditionally, evaluating visualizations requires significant human effort. If LLMs could support visualization experts in evaluating visual representations at a level comparable to humans, they could offer substantial time and cost savings [ 18 ] . However, before we can reliably use LLMs to evaluate visualizations, we must ensure these models possess adequate visualization literacy—the ability to read, understand, and interpret visual representations [ 25 ] . Recent studies have raised important questions about LLMs’ visualization literacy capabilities. Hong et al. [ 20 ] found that both GPT-4 and Gemini failed to achieve the same levels of visualization literacy as the general public when tested on a modified Visualization Literacy Assessment Test (VLAT). Similarly, Bendeck and Stasko [ 5 ] reported that GPT-4 scored in the 16th percentile of humans on the VLAT, demonstrating particular weakness in value retrieval tasks. These findings contrast the increasing capabilities of multimodal LLMs in other domains [ 7 , 20 , 36 , 37 ] . We hypothesized that the underperformance in visualization literacy tasks might stem not from inherent limitations in the models but from how they approach the task. When humans interpret visualizations, they often engage in a structured analytical process: first identifying axes and data points, then extracting relevant values, and finally performing calculations or comparisons to answer questions [ 25 , 9 ] . Standard prompting techniques for LLMs typically do not guide models through such a systematic approach. To address this issue, we introduce a novel prompting strategy called "Charts-of-Thought." This approach is inspired by the Chain-of-Thought prompting technique [ 52 ] , which has proven effective in improving LLMs’ reasoning capabilities for complex mathematical and logical tasks. Our Charts-of-Thought method guides LLMs through a structured data extraction, verification, and analysis process before answering visualization-related questions. Our Charts-of-Thought approach builds on established theories of how humans process visualizations. Bertin’s work on "reading levels" [ 6 ] describes how viewers progress from identifying elements to extracting meaning—a process our structured prompts mirror. Cleveland and McGill [ 12 ] showed that humans follow a sequence when decoding visualizations: they extract values before making comparisons. Carpenter and Shah [ 11 ] proposed that graph comprehension involves cycles of pattern recognition, interpretation, and integration with prior knowledge. The four tasks in our method (data extraction, sorting, verification, and analysis) align with how people process visualizations step by step rather than all at once, as documented by Hegarty [ 19 ] . By structuring prompts to follow this natural cognitive progression, we help LLMs overcome the visualization literacy challenges identified in previous research. Using this approach, we evaluate three state-of-the-art multimodal LLMs—Claude-3.7-sonnet, GPT-4.5-preview, and Gemini-2.0-pro—on the Modified VLAT and VLAT [ 25 ] , a standardized test for assessing visualization literacy. We compare their performance using both standard prompting techniques and our Charts-of-Thought method and examine how well these models perform across different visualization types, task categories, and question difficulty levels. Through this comprehensive evaluation framework, we aim to address several key questions about the capabilities and limitations of modern LLMs in visualization interpretation tasks: • RQ1: Can modern multimodal LLMs achieve human-level visualization literacy when guided through a structured analytical process? • RQ2: How does the Charts-of-Thought prompting strategy affect LLM performance on visualization literacy tasks compared to standard prompting? • RQ3: How do different LLMs perform across various visualization types and analytical tasks? • RQ4: To what extent do LLMs rely on their prior knowledge versus information in the visualization when answering questions? Our findings reveal that when properly guided, LLMs can not only match but exceed human performance on standardized visualization literacy assessments. These results challenge current assumptions about LLMs’ capabilities and suggest new possibilities for their application in visualization contexts. Beyond raw performance metrics, our analysis uncovers patterns in how different models handle various visualization types and analytical tasks, providing practical insights for researchers and practitioners looking to leverage these technologies. In summary, the main contributions of this work are as follows: • We introduce Charts-of-Thought, a structured prompting technique that guides LLMs through systematic data extraction, verification, and analysis processes for visualization interpretation. • We demonstrate that Claude-3.7-sonnet, when using our approach, achieves a VLAT score of 50.17, substantially exceeding the human mean of 28.82 and challenging the view that current LLMs lack sufficient visualization literacy. • We show that our prompting strategy consistently improves performance across all tested models, with score increases of 21.8% for GPT-4.5-preview, 9.4% for Gemini-2.0-pro, and 13.5% for Claude-3.7-sonnet compared to standard prompting. • We provide a detailed analysis of LLM performance across different visualization types and tasks, revealing patterns of strength and weakness to inform future applications. • We propose a framework to test and enhance LLM visualization literacy for future research and applications. In the following, Section 2 presents related work, Section 3 presents our Charts-of-Thought structured prompting approach and contrasts it with conventional prompting, Sections 4-6 present evaluations with a modified VLAT, the original VLAT, and a chart question answering system, and Section

Read the complete text