Chart Question Answering from Real-World Analytical Narratives

Maeve Hutchinson, Radu Jianu, Aidan Slingsby, Jo Wood, Pranava Madhyastha · 2025

Read Chart Question Answering from Real-World Analytical Narratives by Maeve Hutchinson, Radu Jianu, Aidan Slingsby, Jo Wood, Pranava Madhyastha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01627.txt.

Introduction

Maeve Hutchinson and co-authors build a dataset for answering questions about charts in a paper from 2025. Their charts come from real visualization notebooks, not made-up examples. Each chart set pairs with natural questions drawn from the analyst's own written narrative.

Past benchmarks used cleaner, simpler charts that poorly match real work. The new data reflects true, messy reasoning across several linked views. Top multimodal language models were tested on the set. The best, GPT-4.1, answered under seventy percent correctly. That gap shows how hard real chart reading still is for AI. The dataset gives the field a more honest target to aim at.

Catalog record

CREATORMaeve Hutchinson, Radu Jianu, Aidan Slingsby, Jo Wood, Pranava Madhyastha
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-01627

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Chart Question Answering from Real-World Analytical Narratives Maeve Hutchinson, Radu Jianu, Aidan Slingsby, Jo Wood, Pranava Madhyastha Abstract We present a new dataset for chart question answering (CQA) constructed from visualization notebooks. The dataset features real-world, multi-view charts paired with natural language questions grounded in analytical narratives. Unlike prior benchmarks, our data reflects ecologically valid reasoning workflows. Benchmarking state-of-the-art multimodal large language models reveals a significant performance gap, with GPT-4.1 achieving an accuracy of 69.3%, underscoring the challenges posed by this more authentic CQA setting. 1 Introduction Data visualizations are an essential modality for communicating complex information about data. Alongside natural language, they serve as a key medium for communication across domains. As such, the ability to interpret and reason about visualizations is a crucial skill. As multimodal large language models (MLLMs) evolve beyond simple perception tasks towards becoming visual assistants, there is growing interest in their ability to perform visual reasoning over structured data, including charts and other forms of data visualization. Tasks such as Chart Question Answering (CQA) have emerged for benchmarking a model’s visualization reasoning capabilities. In this work, we introduce a new dataset for CQA that aims to reflect the complexity of real-world data analysis. 1 1 1 Dataset available at: https://huggingface.co/datasets/maevehutch/realworld-chartqa The dataset is constructed from student authored visualization notebooks, which combine explanatory analytical narrative with custom visualizations. Unlike existing CQA datasets, our dataset is grounded in ecologically valid analytical workflows. To situate this contribution, we first review prior work on visualization literacy and CQA. We then detail our data collection and question generation process, describing the structure and composition of the dataset. Finally, we report some initial benchmarking results using state-of-the-art MLLMs. 2 Related Work Visualization Literacy datasets such as the visualization literacy assessment test (VLAT) ( Lee et al. 2017 ) were initially created to assess human understanding of data visualizations. Recently, they have also been applied to probe the visualization literacy of MLLMs ( Bendeck and Stasko 2024 ) . These manually curated datasets present small sets of charts paired with multiple-choice questions that probe the ability to perform specific analytic tasks such as retrieving values, identifying trends, or making comparisons. Whilst these tasks seem to mimic real-world analytical workflows Amar et al. 2005 , the hand-crafted design of these datasets limits their ability to accurately reflect the complexity of real-world visualization reasoning. Chart Question Answering (CQA) is the task of answering a natural language question about a visualization image. CQA datasets are designed to benchmark the chart understanding capabilities of models. Early CQA benchmarks such as FigureQA ( Kahou et al. 2018 ) , DVQA ( Kafle et al. 2018 ) , and LEAF-QA ( Chaudhry et al. 2020 ) used template-based questions and synthetically generated tasks. Again, these controlled settings are limited. More recently, CQA datasets have moved toward real-world visualization images. Kim et al. 2020 and ChartQA ( Masry et al. 2022 ) introduced chart images scraped from real-world reports and online sources. However, these datasets still only have questions that refer to a single chart, and do not include visualizations with multiple views or interactive elements. These datasets begin to reflect more realistic evaluation settings, but still do not completely capture visualization as done in-practice, where users often engage with visualizations that have multiple views, such as dashboards or linked visualizations. Some newer datasets begin to address this. CharXiv ( Wang et al. 2024 ) includes charts composed of multiple sub views, although its questions still focus on one image. MultiChartQA ( Zhu et al. 2025 ) allows questions to target multiple related visualizations, moving closer to the kinds of cross-chart reasoning analysts perform in practice. However, these datasets are still composed solely of static visualizations. Another important distinction lies in how questions are generated. Some datasets, such as VLAT ( Lee et al. 2017 ) and MultiChartQA ( Zhu et al. 2025 ) , rely exclusively on human-authored questions. While this approach ensures high-quality queries aligned with human reasoning, the scalability of dataset construction is limited. Conversely, other datasets like ChartQA ( Masry et al. 2022 ) and CharXiv ( Wang et al. 2024 ) adopt semi-automatic approaches, using models to produce questions alongside human validation, enabling larger datasets across more images. Notably, previous datasets, whether template, human or machine-authored, are generated from the visualization image, caption, or from post hoc chart summaries. This often as a result of data collection processes that extract chart images in isolation, often scraped from online sources, removed from the surrounding analytical narrative. Due to the nature of source materials, this analytical context often does not exist at all and is left entirely implicit, available only from the visual context. The nature of these online sources may also raise copyright concerns due to the use of third-party images without explicit permission. 3 Methods 3.1 Data Collection Our dataset is derived from literate visualization (litvis) notebooks, structured markdown documents that combine narrative analysis, code, embedded datasets, and inline visualizations Wood et al. 2019 . The notebooks were authored by undergraduate and postgraduate students as part of their final coursework for a 10-week data visualization module. These notebooks offer an ecologically valid window into real-world analytical practice: students independently selected datasets to analyze, posed research questions, and designed custom visualizations to explore those questions. These notebooks surface articulations of analytical reasoning that are typically left implicit in other sources of visualizations, providing a rich basis for question generation. See appendix D for an example notebook. We applied several filtering steps to ensure data quality. Submissions were excluded if they lacked visualizations, included personally identifiable information, lacked sufficient narrative, or otherwise failed to meet basic quality thresholds. After filtering, we retained 22 notebooks for further processing. From each retained notebook, we extracted two primary sources of data: the analytical narrative written by the student, and the corresponding visualizations. Visualizations were captured by rendering each notebook in HTML and using a headless browser to take screenshots of the embedded figures. Interactive visualizations were present in many of the notebooks, a feature missing from many sources of visualizations in CQA. To partially capture these interactive dynamics, we developed a method for capturing some interactive views statically. For visualizations with discrete interactive controls, such as radio buttons or drop-down menus, we systematically enumerated all categorical options and recorded screenshots of each interactive view. This allowed us to collect multiple views of the same visualization, reflecting user-driven analytical actions that are absent in existing datasets. To prepare the narrative for question generation, we segmented the extracted content into chunks of at most 200 words. Dataset Visualizations Questions Real-World # Chart Types Multi/Interactive Unanswerable Narrative Context LeafQA ( Chaudhry et al. 2020 ) ✗ 6 ✗/✗ ✗ ✗ Kim et al. 2020 \sim 2 ✗/✗ ✗ ✗ ChartQA ( Masry et al. 2022 ) ✓ 3 ✗/✗ ✗ ✗ CharXiv ( Wang et al. 2024 ) ✓ unbounded ✓/✗ ✓ ✗ MultiChartQA ( Zhu et al. 2025 ) ✓ unbounded ✓/✗ ✓ ✗ Ours ✓ unbounded ✓/✓ ✓ ✓ Table 1: Comparison between our dataset and existing chart question-answering datasets, grouped by visualization and question characteristics. 3.2 Question Generation We structured our dataset according to established analytical task taxonomies

Read the complete text