See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMs

Zhimin Li Vanderbilt University, Haichao Miao Lawrence Livermore National Laboratory, Xinyuan yan University of Utah, Valerio Pascucci University of Utah, Matthew Berger Vanderbilt University, Shusen Liu Lawrence Livermore National Laboratory · 2025

Read See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMs by Zhimin Li Vanderbilt University, Haichao Miao Lawrence Livermore National Laboratory, Xinyuan yan University of Utah, Valerio Pascucci University of Utah, Matthew Berger Vanderbilt University, Shusen Liu Lawrence Livermore National Laboratory (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09809.txt.

Introduction

Zhimin Li and five co-authors ran a sanity check in twenty twenty-five on whether multimodal models truly read charts when answering questions about them. Visualization question answering is used to grade chart understanding, by analogy with human literacy tests. The authors found models answer many test questions correctly with no chart shown at all, options or not.

Their explanation: the language model's stored knowledge lets it recall answers, bypassing the visual signal entirely. That undermines the test's meaning. They propose a sanity check framework — a rule-based decision tree plus a check table — that separates seeing from recalling in a model's responses. It can validate data sets and flag where vision helps, hurts, or is simply ignored.

Catalog record

CREATORZhimin Li Vanderbilt University, Haichao Miao Lawrence Livermore National Laboratory, Xinyuan yan University of Utah, Valerio Pascucci University of Utah, Matthew Berger Vanderbilt University, Shusen Liu Lawrence Livermore National Laboratory
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-09809

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMs Zhimin Li Vanderbilt University, Haichao Miao Lawrence Livermore National Laboratory, Xinyuan yan University of Utah, Valerio Pascucci University of Utah, Matthew Berger Vanderbilt University, Shusen Liu Lawrence Livermore National Laboratory Abstract Recent developments in multimodal large language models (MLLM) have equipped language models to reason about vision and language jointly. This permits MLLMs to both perceive and answer questions about data visualization across a variety of designs and tasks. Applying MLLMs to a broad range of visualization tasks requires us to properly evaluate their capabilities, and the most common way to conduct evaluation is through measuring a model’s visualization reasoning capability, analogous to how we would evaluate human understanding of visualizations (e.g., visualization literacy). However, we found that in the context of visualization question answering (VisQA), how an MLLM perceives and reasons about visualizations can be fundamentally different from how humans approach the same problem. During the evaluation, even without visualization, the model could correctly answer a substantial portion of the visualization test questions, regardless of whether any selection options were provided. We hypothesize that the vast amount of knowledge encoded in the language model permits factual recall that supersedes the need to seek information from the visual signal. It raises concerns that the current VisQA evaluation may not fully capture the models’ visualization reasoning capabilities. To address this, we propose a comprehensive sanity check framework that integrates a rule-based decision tree and a sanity check table to disentangle the effects of ”seeing” (visual processing) and ”recall” (reliance on prior knowledge). This validates VisQA datasets for evaluation, highlighting where models are truly ”seeing”, positively or negatively affected by the factual recall, or relying on inductive biases for question answering. Our study underscores the need for careful consideration in designing future visualization understanding studies when utilizing MLLMs. \vgtcinsertpkg keywords Multimodal Large Language Model, Visualization Question and Answer, Visualization Literacy 1 Introduction Recent advancements in multimodal large language models (MLLM) have facilitated their ability to perceive and understand images, charts, and diagrams [ 46 ] , effectively allowing them to see . The capability of processing multimodal inputs has been explored in many general-purpose vision-related benchmarks [ 24 , 42 , 40 ] . This new visual understanding capability of language models has also inspired exciting research directions in the visualization community, such as investigating an LLM’s visualization reasoning abilities [ 3 , 9 , 25 ] or developing autonomous visualization agents [ 27 ] . Specifically, the visualization understanding capability has been increasingly studied [ 3 , 25 , 18 ] in the context of visualization literacy [ 23 , 33 , 32 ] . These evaluations typically assume that MLLMs behave like humans, requiring them to first read and understand the given visualization, and then answer relevant questions based on their interpretation of the visual information provided in the prompt. However, this assumption does not consider the possibility that MLLMs might not need to interpret a visualization to answer questions correctly. Instead, they might rely on factual recall from their training data in deciding on an answer. This implies the model might discard the visual encoding of data, and rather attend to the question/answer text, if not the rendered text of the visualization in the form of titles, labels, and axes. For example, Figure 1 presents a question from the popular Visualization Literacy Assessment Test (VLAT) dataset designed to evaluate human visualization literacy. When the same question is posed to the GPT-4 model without providing the accompanying visualization, the model still answers correctly and provides an explanation: “The correct answer is: 4) $37.04 - $60.95. In 2015, oil prices fluctuated between approximately $37 and $61 per barrel due to various market conditions, including oversupply and reduced demand.” Such an explanation implies that the model contains prior knowledge on economic data, and thus, the line plot might not be necessary for the model to give a correct answer. Figure 1: An example from the VLAT dataset where the highlighted text ”oil price in 2015” provides context for MLLM that can trigger factual recall, allowing the model to answer correctly even without visual input. This behavior raises critical questions: Are MLLMs actually “Seeing” the visualization to answer the question or simply “Recalling” the knowledge from training data to complete the task? What role does visual input play in visualization question-answering (VisQA) tasks when evaluating MLLMs? Instead of assuming that MLLMs process visual information like humans, our study aims to reveal the different factors influencing the MLLM’s decision-making processes and to understand the role of vision in these tests. We propose a guideline for more robust and rigorous evaluation practices. To understand these behaviors, we investigate the MLLM model’s responses to the different VisQA datasets. We observed that the MLLM can correctly answer a significant portion of questions without ever seeing the corresponding visualizations, indicating that the surprising example of Figure 1 is not the exception. This observation raises the concern of what we are measuring in VisQA evaluation. Do we have a correct interpretation of the evaluation results? How do we know whether there is a chance the model may simply obtain the answer through factual recall instead of correctly interpreting the visualization as the test intends? To gain a better understanding of these questions, we design a comprehensive framework of sanity checks for disentangling the impact of “Recalling” and “Seeing” for MLLM evaluation. The framework starts with a decision-tree-based diagram on a single question to cover all outcomes of different ablation scenarios and categorize them into failure/success cases. The result of the decision tree is summarized into three cases of failing the sanity check. The design is further extended to a probability sanity check table for general validation of a dataset. The evaluation of our methodology goes through each sanity check failing case with VLAT [ 23 ] dataset, and then studies how our framework reveals the problem of counterfactual no-factual VisQA in the VLATforge dataset. We strengthen our with much larger datasets VILA [ 11 ] and ChartQA [ 29 ] . Our experiment demonstrates that “Recalling” can potentially lead to overestimating or underestimating a model’s performance on visual tasks. Meanwhile, this problem cannot be easily fixed through prompt engineering. Our study presents a cautionary tale for researchers examining the visualization understanding capabilities of MLLMs and provides general guidelines for improving the robustness of such evaluations. The findings also highlight a broader issue in MLLM’s vision evaluation and raise fundamental questions about how to assess their vision capabilities properly. Overall, we summarize our main contributions as follows: • We demonstrate the widespread critical issue wherein MLLMs can answer visualization reasoning questions correctly without actually interpreting the visualizations, instead relying on factual recall from their extensive training data in the visualization reasoning context with multiple datasets ( Sections 3 ). • We design a comprehensive and novel sanity check framework that utilizes a decision-tree-base abstraction to distill three validated cases and a sanity check table to verify the effects of “recalling” and “seeing” in MLLMs’ decision-making processes ( Section 4 ). • We experiment with the proposed sanity check framework that

Read the complete text