ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinement

Ali Salamatian, Amirhossein Abaskohi, Wan-Cyuan Fan, Mir Rayat Imtiaz Hossain, Leonid Sigal, Giuseppe Carenini · 2025

Read ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinement by Ali Salamatian, Amirhossein Abaskohi, Wan-Cyuan Fan, Mir Rayat Imtiaz Hossain, Leonid Sigal, Giuseppe Carenini (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13282.txt.

Introduction

Ali Salamatian, Amirhossein Abaskohi and four co-authors align chart reading in vision-language models with human gaze in 2025. Charts carry much public data, yet models stare at wrong zones and err. Human gaze during chart tasks was not mapped at scale.

ChartGaze records eye tracks on chart reasoning tasks. Model focus diverges from human focus in set ways, which hurts scores and clarity. A gaze-guided fix aligns cross focus with fixations. Scores rise by up to a few points across models. Focus match also improves. Human gaze proves a cheap guide for chart models.

Catalog record

CREATORAli Salamatian, Amirhossein Abaskohi, Wan-Cyuan Fan, Mir Rayat Imtiaz Hossain, Leonid Sigal, Giuseppe Carenini
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-13282

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinement Ali Salamatian, Amirhossein Abaskohi, Wan-Cyuan Fan, Mir Rayat Imtiaz Hossain, Leonid Sigal, Giuseppe Carenini Abstract Charts are a crucial visual medium for communicating and representing information. While Large Vision-Language Models (LVLMs) have made progress on chart question answering (CQA), the task remains challenging, particularly when models attend to irrelevant regions of the chart. In this work, we present ChartGaze, a new eye-tracking dataset that captures human gaze patterns during chart reasoning tasks. Through a systematic comparison of human and model attention, we find that LVLMs often diverge from human gaze, leading to reduced interpretability and accuracy. To address this, we propose a gaze-guided attention refinement that aligns image-text attention with human fixations. Our approach improves both answer accuracy and attention alignment, yielding gains of up to 2.56 percentage points across multiple models. These results demonstrate the promise of incorporating human gaze to enhance both the reasoning quality and interpretability of chart-focused LVLMs 1 1 1 Code and dataset are publicly available at ChartGazeCode and ChartGazeData , respectively. . Figure 1: Motivation. Example of the model’s attention maps. The left model, trained with language loss only, attends inconsistently and gives the wrong answer. The right model, trained with our proposed attention refinement loss, focuses more meaningfully on relevant regions and answers correctly. 1 Introduction Charts are a common visual medium for communicating structured information and supporting analysis, comparison, and decision-making across domains. With the advancement of large language models (LLMs) Brown et al. 2020 ; Achiam et al. 2023 ; Touvron et al. 2023 ; Chiang et al. 2023 ; Grattafiori et al. 2024 ; Anil et al. 2023 ; Liu et al. 2024a and large vision-language models (LVLMs) Liu et al. 2023 ; Team et al. 2023 ; Zhou et al. 2024 ; Chen et al. 2024c ; Chen et al. 2024b ; Masry et al. 2025b , Chart Question Answering (CQA) has emerged as a key research challenge at the intersection of language, vision and data understanding Masry et al. 2022 . Early CQA approaches often converted charts into structured data or templates, but recent efforts have increasingly focused on leveraging LVLMs directly on chart images Han et al. 2023 ; Masry et al. 2025b ; Tian et al. 2025 . Automated chart understanding can significantly aid evidence-based decision-making, by supporting fact-checking Akhtar et al. 2023 or improving accessibility for visually impaired users through textual or spoken descriptions Choi et al. 2019 . As these models continue to advance and find increasing deployment, there is an increasing interest in understanding their internal mechanisms, especially their attention dynamics. Attention between visual and textual input modalities plays a vital role in how LVLMs form inductive biases and make an inference. Additionally, image-text attention serves as a window into the model’s reasoning process, offering a potential avenue for interpretability and transparency. This is particularly important in high-stakes domains such as finance, medicine, and scientific research, where the ability to interpret model decisions is critical. However, recent studies have shown that image-text attention maps in LVLMs often fail to align with the regions humans focus on when answering questions, a phenomenon known as attention misalignment Li et al. 2024 ; An et al. 2025 ; Han et al. 2025 ; Shu et al. 2025 . This misalignment can cause models to fixate on irrelevant text or visual elements, leading to incorrect predictions and reduced interpretability. In chart understanding, this issue can be even more problematic, as key information, such as a specific bar, point, or legend entry, is often small or densely packed. As a result, the decision-making process of LVLM-based CQA systems becomes untrustworthy, limiting their use where interpretability is critical. Hence, in this paper, we analyze the attention misalignment in the context of CQA. We observe that LVLMs, even those instruction tuned on charts, frequently attend to irrelevant chart elements, leading to incorrect responses and a reduced interpretability. We draw inspiration from human visual attention, as prior work Gao et al. 2022 ; Yan et al. 2024 has shown that human gaze tends to align with perceived importance. We leverage eye-tracking data as explicit supervision to guide the attention maps of LVLMs. Specifically, we collect eye-tracking data from human participants responding to chart reasoning questions. Using the human gaze data, we train models for CQA to focus on regions where humans typically fixate, thereby improving both alignment and interpretability. As shown in Figure 1 , models trained with our approach produce more interpretable and human-aligned attention maps, leading to more accurate answers. Empirical results show that aligning LVLM attention with human gaze improves CQA accuracy by up to 2.56 percentage points compared to fine-tuning with language loss alone. To summarize, our key contributions are: • Eye-Tracking Chart Dataset: We introduce a new eye-tracking dataset for CQA, capturing regions users look at while answering chart-related questions, serving as the ground-truth. • Analysis of LVLM Attention on Charts: To the best of our knowledge, we are the first to conduct a systematic study of attention patterns of LVLMs on chart understanding and analyze how they compare with human gaze. • Gaze-Guided Attention Refinement: We develop a training approach that aligns LVLM attention with human gaze, using a gaze-supervised loss. 2 Related Work Chart Question Answering Datasets: Document understanding, particularly scientific chart understanding, has gained significant attention in the machine learning community. As a result, various datasets and benchmarks have been developed to accelerate progress and evaluate models in chart understanding, including summarization ( Kantharaj et al. 2022b ) , question answering ( Masry et al. 2022 ) , explanation generation ( Kantharaj et al. 2022a ) , and fact-checking ( Akhtar et al. 2023 ) . Among these, CQA has become a focal point, driven by the rapid advancements of LVLMs. Early benchmarks such as STL-CQA ( Singh and Shekhar 2020 ) , LEAF-QA ( Chaudhry et al. 2020 ) , FigureQA ( Kahou et al. 2018 ) , and DVQA ( Kafle et al. 2018 ) relied on synthetic charts or templated questions. Later efforts like PlotQA ( Methani et al. 2020 ) and ChartQA ( Masry et al. 2022 ) introduced charts from real-world sources, improving the diversity and realism of the data. Recent benchmarks pushed the evaluation into open domains ( Masry et al. 2025a ; Wang et al. 2024b ; Liu et al. 2024b ) and more complex, reasoning-intensive understanding tasks ( Fan et al. 2024 ; Xia et al. 2024 ; Xu et al. 2023 ) . Different from prior work, our benchmark uses eye-tracking annotations alongside chart question pairs to measure how well the LVLMs’ attention aligns with that of humans. This human-centric design is crucial for bridging the gap between model performance and interpretability. Gaze Datasets : Several studies have collected human gaze data to understand visual attention on charts. Borkin et al. 2016 collected a dataset of 393 visualizations along with participants’ fixation locations during encoding and recognition of the visualizations. Polatsek et al. 2018 analyzed human visual attention during task-solving on 30 charts. More recently, Shin et al. 2022 gathered human attention on 10,960 chart images for the task of chart type recognition

Read the complete text