Evaluating ‘Graphical Perception’ with Multimodal LLMs

Rami Huu Nguyen University of Massachusetts Boston, Kenichi Maeda University of Massachusetts Boston, Mahsa Geshvadi University of Massachusetts Boston, Daniel Haehn University of Massachusetts Boston · 2025

Read Evaluating ‘Graphical Perception’ with Multimodal LLMs by Rami Huu Nguyen University of Massachusetts Boston, Kenichi Maeda University of Massachusetts Boston, Mahsa Geshvadi University of Massachusetts Boston, Daniel Haehn University of Massachusetts Boston (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04221.txt.

Introduction

Rami Huu Nguyen, Kenichi Maeda, Mahsa Geshvadi, and Daniel Haehn test how well AI models read charts in a paper from 2025. Multimodal models now describe images fluently. Pulling exact values from a chart is a different, harder skill. It is the core of graphical perception.

The team reran a classic 1984 experiment by Cleveland and McGill. Humans and models judged the same chart types under the same tasks. Fine-tuned, pretrained, and zero-shot models were all tried. Models beat human accuracy in some tasks and lagged in others. The split map shows where machine chart reading is trustworthy today. Visualization tools that embed AI need exactly this kind of audit.

Catalog record

CREATORRami Huu Nguyen University of Massachusetts Boston, Kenichi Maeda University of Massachusetts Boston, Mahsa Geshvadi University of Massachusetts Boston, Daniel Haehn University of Massachusetts Boston
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-04221

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating ‘Graphical Perception’ with Multimodal LLMs Rami Huu Nguyen University of Massachusetts Boston, Kenichi Maeda University of Massachusetts Boston, Mahsa Geshvadi University of Massachusetts Boston, Daniel Haehn University of Massachusetts Boston Abstract Multimodal Large Language Models (MLLMs) have remarkably progressed in analyzing and understanding images. Despite these advancements, accurately regressing values in charts remains an underexplored area for MLLMs. For visualization, how do MLLMs perform when applied to graphical perception tasks? Our paper investigates this question by reproducing Cleveland and McGill’s seminal 1984 experiment and comparing it against human task performance. Our study primarily evaluates fine-tuned and pretrained models and zero-shot prompting to determine if they closely match human graphical perception. Our findings highlight that MLLMs outperform human task performance in some cases but not in others. We highlight the results of all experiments to foster an understanding of where MLLMs succeed and fail when applied to data visualization. \onlineid 0 \vgtccategory Research \vgtcinsertpkg \teaser Computing Cleveland and McGill’s Position-Angle Experiment using Multimodal Large Language Models . We replicate the original experiment by asking MLLMs to interpret values in pie and bar charts using zero-shot prompting, where models follow instructions without prior examples. Results highlight that MLLMs predict values more accurately from bar charts (mean squared error (MSE) in green). Introduction keywords Multimodal Large Language Models, Graphical Perception, Machine Perception, Deep Learning Nowadays, data visualization has become increasingly important in our lives [ 21 , 11 ] . There has been a rising research focus on computational techniques for studying charts, and graphs. [ 2 , 21 , 26 ] , which are applied in several applications, including data extraction, classification, visual Q&A (e.g., “computer, which section is greater?”), and design evaluation or synthesis. MLLMs have made significant progress in analyzing and understanding images [ 9 , 22 , 1 , 20 , 14 , 27 , 8 ] . Although MLLMs perform well in understanding charts, they struggle in generalization and face difficulties accurately answering chart-related questions [ 1 , 11 ] . This requires the MLLMs to understand both language and information derived in charts and apply reasoning skills to provide correct answers [ 12 , 10 ] . Most current MLLMs are pre-trained vision and knowledge, which means those models are trained before with general knowledge, and they might struggle with new application [ 13 , 15 , 17 , 16 ] , which potentially lead to incorrect visual understanding. Understanding images (computer vision) poses unique challenges as compared to understanding language [ 16 ] [ 19 ] . Language often relies on structured syntax and grammar, while chart data depends on spatial relationships, patterns, and context [ 18 ] . Hence, analyzing chart data might be more challenging for the MLLMs. What’s more, the limitation of MLLMs also persists: MLLMs find it difficult to recognize small objects or tiny details in pictures [ 24 , 22 , 23 ] . Additionally, MLLMs currently have difficulty pinpointing the important details in the images that are unclear or absent in the images [ 25 ] . Also, humans use senses such as sight and language to understand the world and recognize new objects based on their knowledge. [ 17 , 27 ] . Zero-shot prompting follows similar principles as human abilities, with its main purpose being to improve MLLMs using the zero-shot prompts to make them perform better without the need for additional training. Cleveland and McGill introduced the concept of graphical perception, explaining how humans visually interpret information from graphs [ 3 , 4 ] . Cleveland and McGill defined elementary perceptual tasks as mental-visual processes and ranked how complex those tasks are for humans. Building on their work, our research is to compare fine-tuned MLLMs, trained for specific low-level graphical perception tasks, with pretrained MLLMs models using zero-shot prompting. Once those MLLMs models perform well on those tasks, they will establish a strong foundation for interpreting more complex visualizations. 0.1 Related Work To study this graphical perception concept, Cleveland and McGill performed the position-angle experiment (comparing pie charts and bar charts) and the position-length experiment (where participants were asked to compare values in groups and divided charts [ 3 , 4 ] . Then, the authors use this to redesign a statistical map via bars, framed rectangles, and Weber’s law [ 5 ] , using the proportional relation between an initial distribution density and perceivable change. Heer and Bostock later reproduced Cleveland and McGill’s experiments by crowdsourcing participants on Amazon Mechanical Turk with similar findings [ 6 ] . Harrison et al. also repeated the studies while studying viewer emotions, again with similar results [ 5 ] . Talbot et al. explored how variation in bar charts affects human prediction [ 7 ] . Cleveland and McGill’s idea of graphical perception does not rely on human-specific traits, and it targets the process of decoding information visually. Thus, it allows machines to perform similar tasks, such as MLLMs. Nonetheless, machines must match humans’ graphical perception levels to function effectively in practice. Our research paper is inspired by Cleveland and McGill’s, and Haehn et al. [ 2 ] also use CNNs for similar stimuli to investigate where machines perceive and reason visual relationships similar to human perception. 1 Experiment Setup We compare MLLMs to human baselines across five experiments . E1 estimates quantities from visual features based on Cleveland and McGill’s elementary perceptual tasks. E2 replicates their position-angle experiment, comparing pie and bar charts. E3 reproduces their position-length experiment, analyzing grouped versus divided bar charts. E4 evaluates bars and framed rectangles using their visual cue framework. E5 conducts a Weber’s law point cloud experiment. 1.1 Networks and Processes As a starting point, we used three latest closed-source pretrained MLLMs, including GPT-4o with 1.8 trillion parameters, Gemini 1.5 Flash with 8 billion parameters, Gemini 1.0 Vision Pro (unavailable parameter data), and one open-source Llama 3.2-Vision with 11 billion parameters. For fine-tuned MLLMs , we used Llama 3.2-Vision , with 6.0 billion parameters, of which 94.4 million were trainable. We also produced 15 fine-tuned MLLMs (each experiment has three fine-tuned MLLMs) for this study. View our fine-tuning details in our supplementary material. Each experiment generated 5000 unique training images , 1000 for unique validation , and 55 for unique testing for each task , add added 5% noise to each image, and ensure no data leakage. Detailed dataset generation and preprocessing are discussed in our supplementary material. 1.2 Measurements Our research calculates models’ performances using the midmean logistic absolute error metric (MLAE). Our research paper is inspired by Cleveland and McGill’s methodology in their 1984 study [ 3 ] and defines our calculation as follow: {\text{MLAE}=\frac{1}{N}\sum_{i=1}^{N}\log_{2}\left(\lvert\text{predicted}_{i}-\text{true }_{i}\rvert+0.125\right)} We also include standard error metrics such as mean squared error (MSE) and mean absolute error (MAE). Our fine-tuned models use Cross-Entropy loss instead of MLAE, applying the logarithm before averaging to ensure fair evaluation of small and large errors. d 1.3 Stimuli We employed the stimuli generator designed by Haehn et al. [ 2 ] for each perceptual task, and the number of parameter values varied depending on each experiment. 1.4 Human Baselines We gather human baseline measurements for the position-angle (E2) and position-length (E3) experiments from [ 3 ] , with

Read the complete text