Chart-RL: Generalized Chart Comprehension via Reinforcement Learning with Verifiable Rewards

Xin Zhang, Xingyu Li, Rongguang Wang, Ruizhong Miao · 2026

Read Chart-RL: Generalized Chart Comprehension via Reinforcement Learning with Verifiable Rewards by Xin Zhang, Xingyu Li, Rongguang Wang, Ruizhong Miao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06958.txt.

Introduction

Zhang and colleagues present Chart-RL, a reward learning method for chart reading with proven rewards. Chart reading challenges multi-mode systems, since much detail is packed into visual forms. Current vision-language models often fail to transfer on unseen charts, which demand abstract, sign-based, and number thinking over visuals. Chart-RL uses proven rewards to improve chart question answers in vision-language models. Tests show it beats guided fine-tuning across chart benchmarks, with gains on MultiChartQA and ChartInsights and better scores in most changed chart kinds.

Catalog record

CREATORXin Zhang, Xingyu Li, Rongguang Wang, Ruizhong Miao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30310
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06958

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Chart-RL: Generalized Chart Comprehension via Reinforcement Learning with Verifiable Rewards Xin Zhang, Xingyu Li, Rongguang Wang, Ruizhong Miao Abstract Accurate chart comprehension represents a critical challenge in advancing multimodal learning systems, as extensive information is compressed into structured visual representations. However, existing vision-language models (VLMs) frequently struggle to generalize on unseen charts because it requires abstract, symbolic, and quantitative reasoning over structured visual representations. In this work, we introduce Chart-RL, an effective reinforcement learning (RL) method that employs mathematically verifiable rewards to enhance chart question answering in VLMs. Our experiments demonstrate that Chart-RL consistently outperforms supervised fine-tuning (SFT) across different chart understanding benchmarks, achieving relative improvements of 16.7% on MutlChartQA, and 11.5% on ChartInsights. We conduct robustness analysis, where Chart-RL achieves enhanced performance in 18 of 25 perturbed chart categories, demonstrating strong consistency and reasoning capability across visual variations. Furthermore, we demonstrate that task difficulty and inherent complexity are more critical than data quantity in RL training. For instance, Chart-RL trained on merely 10 complex chart-query examples significantly outperforms models trained on over 6,000 simple examples. Additionally, training on challenging reasoning tasks not only improves in-domain generalization relative to simpler tasks, but also facilitate strong transfer to out-of-domain visual mathematical problems. 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has demonstrated remarkable success in enhancing the reasoning capabilities of LLMs ( 5 ; 12 ; 33 ) . However, one critical area that remains relatively underexplored is chart understanding in VLMs, whose challenges such as symbolic reasoning and precise data extraction distinguish it fundamentally from traditional vision-language tasks. Unlike natural images, charts exhibit extreme diversity in types (bar charts, pie charts, scatter plots, etc.) and structural organizations, with performance heavily dependent on the training data distribution ( 8 ) . Current VLMs often struggle to bridge the gap between extracting descriptive information and deriving correct answers that require multi-step reasoning ( 15 ) . Figure 1: Comparison of Chart-RL vs SFT to reason on complex chart question answering. Recent studies to improve chart question answering have explored various strategies, such as converting charts into tabular formats suitable for prompting LLMs ( 19 ) , instruction tuning on multiple chart-related objectives ( 29 ) , and generating synthetic chart data to augment VLM’s capability in real-world chart comprehension ( 10 ) . Nonetheless, the majority of chart comprehension methods rely on either larger curated datasets that cover broader chart and question types, or SFT using domain-specific datasets. While large-scale datasets such as ChartQA ( 23 ) , PlotQA ( 25 ) , and FigureQA ( 13 ) have significantly expanded coverage of chart types and reasoning scenarios, they often focus on specific chart formats with template-based questions that may not fully capture real-world complexity. Although domain-specific SFT can enhance model performance on targeted task categories, it frequently leads to catastrophic forgetting and performance degradation on untrained tasks due to data distribution shifts ( 21 ; 14 ) . This phenomenon fundamentally limits the capability of current approaches to generalize across diverse chart types and reasoning scenarios in real-world applications ( 30 ; 9 ) . To address these limitations, we propose Chart-RL , an efficient reinforcement fine-tuning framework that employs R1-style reinforcement learning to enhance chart understanding capabilities in VLMs. A key insight motivating our approach is that chart-based questions frequently correspond to mathematically verifiable ground truths, making them well-suited for rule-based reward functions in RL training. More importantly, many charts encapsulate sufficiently complex visual and semantic information that necessitate iterative reasoning processes. We find that such reasoning complexity is of central importance because it enables models to develop advanced reasoning capabilities that can’t be acquired through simple chart interpretation tasks ( 27 ) . Leveraging group relative policy optimization (GRPO) with chart-aligned accuracy and format rewards ( 31 ; 32 ) , we train VLMs to develop robust reasoning capabilities that generalize across different chart types and complexity levels. Our approach circumvents the limitation of conventional SFT methods, where task-specific adaptation results in narrow specialization rather than transferable reasoning skills ( 21 ) . As illustrated in Figure 1 , we demonstrate that Chart-RL improves performance across different chart understanding tasks, ranging from simple data extraction to complex multi-step reasoning, while achieving remarkable data efficiency. By training on complex chart problems, Chart-RL develops proficiency in low-level component tasks without requiring explicit instruction tuning. The key contributions of this paper include: • First Adoption of RLVR for Chart Comprehension in VLMs: We introduce Chart-RL, a reinforcement fine-tuning strategy that uses reinforcement learning with verifiable rewards to enhance chart comprehension capabilities in VLMs, achieving significant improvements over SFT approaches. • Efficient Generalization with Superior Training Data Efficiency: We demonstrate, for the first time, that RL training on complex chart reasoning tasks enables robust generalization, achieving consistent improvements across diverse benchmarks without task-specific optimization. • Comprehensive Experimental Validation of Robustness and Generalization: Our extensive empirical evaluation reveals a critical insight: task complexity , rather than the quantity of training data, is pivotal in developing generalizable chart understanding capabilities. Figure 2: Schematic diagram of Chart-RL framework with GRPO optimization for chart comprehension. 2 Related Works 2.1 Chart-Based Question Answering Chart understanding has evolved from multi-stage pipelines ( 23 ) to end-to-end models. This transformation was pioneered by OCR-free architectures like Donut ( 16 ) , which paved the way for powerful VLMs. Recent progress has been predominantly achieved through SFT approaches, with models such as Pix2Struct 17 and UniChart ( 22 ) being adapted for direct chart-to-answer generation. Concurrently, advanced models including LLaVA 20 , PaliGemma ( 2 ) , and ChartLlama ( 6 ) have been fine-tuned to address more sophisticated and diverse reasoning challenges. Despite its success in task-specific adaptation, SFT faces critical limitations in the domain of chart understanding. The need for large, meticulously annotated datasets makes the process costly and prone to overfitting, where models learn to exploit data artifacts rather than acquiring robust reasoning principles. Furthermore, SFT-based models often exhibit poor generalization ability: performance gains on one type of task do not reliably transfer to related tasks or different data distributions 30 . 2.2 Reinforcement Fine-Tuning in VLMs The success of R1-style reinforcement learning in enhancing LLM reasoning capabilities ( 12 ; 5 ) has inspired extensive research into its application toVLMs. For example, R1-OneVision ( 38 ) and Vision-R1 ( 11 ) construct multimodal reasoning datasets by converting visual information into textual format before RL training, while other studies focus on specific applications such as object counting ( 3 ) or employ staged training strategies with progressive difficulty rewards ( 28 ; 4 ) . Notably, several studies have observed the “visual aha moment” in VLMs ( 40 ; 24 ) , demonstrating emergent reasoning capabilities. The VLM-R1 framework ( 32 ) provides a comprehensive investigation into applying RL to general visual understanding tasks, including Referring Expression Comprehension (REC) and Open-Vocabulary Object Detection (OVD). Despite these advances, the reasoning capabilities cultivated by existing R1-style VLMs are primarily tailored for understanding of natural images. Since charts present highly condensed information in diverse structural formats, the unique challenges of chart understanding have yet to be systematically addressed. These tasks require interpreting relationships, performing calculations, and synthesizing information from graphical elements

Read the complete text