CharTool : Tool-Integrated Visual Reasoning for Chart Understanding
Situo Zhang, Yifan Zhang, Zichen Zhu, Da Ma, Lei Pan, Danyang Zhang, Zihan Zhao, Lu Chen, Kai Yu · 2026
Read CharTool : Tool-Integrated Visual Reasoning for Chart Understanding by Situo Zhang, Yifan Zhang, Zichen Zhu, Da Ma, Lei Pan, Danyang Zhang, Zihan Zhao, Lu Chen, Kai Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02794.txt.
Introduction
Zhang and eight colleagues present CharTool, a tool-integrated system for chart reasoning in multimodal language models. Charts demand fine-grained visual grounding and exact numerical computation, yet training data is scarce. Their DuoChart pipeline mixes synthesized and real charts for training data, while CharTool adds image cropping for local perception and code-based computation for arithmetic. CharTool-7B beats its base model by 8.0 percent on CharXiv reasoning.
Catalog record
| CREATOR | Situo Zhang, Yifan Zhang, Zichen Zhu, Da Ma, Lei Pan, Danyang Zhang, Zihan Zhao, Lu Chen, Kai Yu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52051 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.02794
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CharTool : Tool-Integrated Visual Reasoning for Chart Understanding Situo Zhang, Yifan Zhang, Zichen Zhu, Da Ma, Lei Pan, Danyang Zhang, Zihan Zhao, Lu Chen, Kai Yu Abstract Charts are ubiquitous in scientific and financial literature for presenting structured data. However, chart reasoning remains challenging for multimodal large language models (MLLMs) due to the lack of high-quality training data, as well as the need for fine-grained visual grounding and precise numerical computation. To address these challenges, we first propose DuoChart , a scalable dual-source data pipeline that combines synthesized charts with real-world charts to construct diverse, high-quality chart training data. We then introduce CharTool , which equips MLLMs with external tools, including image cropping for localized visual perception and code-based computation for accurate numerical reasoning. Through agentic reinforcement learning on DuoChart , CharTool learns tool-integrated reasoning grounded in chart content. Extensive experiments on six chart benchmarks show that our method consistently improves over strong MLLM baselines across model scales. Notably, CharTool -7B outperforms the base model by +8.0% on CharXiv (Reasoning) and +9.78% on ChartQAPro, while achieving competitive performance with substantially larger or proprietary models. Moreover, CharTool demonstrates positive generalization to out-of-domain visual math reasoning benchmarks. † † affiliationnotice: Equal contribution Figure 1 : Motivation for our method. (Left) Chart reasoning requires fine-grained visual perception and numerical reasoning. (Middle) Synthetic charts often lack diversity and visual quality. (Right) Purely textual reasoning leads to errors on complex layouts, while explicit tool grounding enables accurate, localized analysis. (Only cropping is illustrated; see Appendix F for more examples.) 1 Introduction Charts serve as a fundamental medium for presenting structured data in scientific papers and financial reports, combining textual annotations with structured visual elements to convey quantitative relationships. Although recent multimodal large language models (MLLMs) have achieved strong performance in general visual understanding and reasoning ( OpenAI 2023 ; Liu et al. 2023c ; Comanici et al. 2025 ; Bai et al. 2025a ; Huang et al. 2025 ) , they continue to struggle with chart reasoning, which requires fine-grained visual perception as well as precise relational and numerical reasoning over densely structured information ( Masry et al. 2022 ; Wang et al. 2024 ; Xu et al. 2023 ) . To tackle chart reasoning, existing research has primarily focused on either scaling up training data through synthesis or improving models’ reasoning capabilities for chart question answering ( Liu et al. 2023b ; Han et al. 2023 ; Liu et al. 2024b ; Meng et al. 2024 ) . However, current works suffer from two key limitations. 1) Existing synthesized chart datasets often lack sufficient diversity and complexity ( Wang et al. 2024 ) . Early approaches rely on fixed templates and cover only a limited set of chart types ( Han et al. 2023 ; Kantharaj et al. 2022 ) . More recent code-driven pipelines are more scalable, but they still struggle to capture real-world, long-tail chart patterns and often produce degraded layouts (e.g., overlapping text) as chart complexity increases ( He et al. 2025 ; Yang et al. 2025b ) . 2) Current chart reasoning methods are constrained by the limitations of chain-of-thought text-based reasoning . Unlike general visual question answering, chart reasoning requires fine-grained visual and numerical analysis to interpret complex elements such as overlapping data points, intersecting trend lines, and dense numerical annotations, while simultaneously modeling their spatial arrangements and semantic meanings. However, existing methods typically generate answers either directly or through textual reasoning ( Chen et al. 2025 ; Masry et al. 2025b ; Wei et al. 2022 ) ( Figure 1 -Right), focusing on textual logical inference rather than grounded visual processing. As a result, they are often reliant on language priors, making them prone to hallucination ( Bai et al. 2024 ) and limiting their ability to perform precise numerical computation or capture complex relationships encoded in charts ( Zheng et al. 2025 ; Zhang et al. 2025b ) . To address these limitations and further improve the chart reasoning capabilities of MLLMs, we propose DuoChart , a dual-source chart synthesis pipeline (§ 2.1.1 ). The two data sources offer complementary strengths: real-world charts from scientific literature provide broad coverage and high visual quality, while code-driven synthetic charts provide precise underlying data for answer generation and verification. To combine these advantages, we sample diverse and fine-grained specifications, including chart types, layouts, and difficulty levels, to generate varied chart-rendering code. We further incorporate high-quality charts from scientific literature to better capture real-world chart distributions. In addition, we design a dedicated question-answer synthesis pipeline with multi-stage checking and scoring to construct high-quality, challenging QA training data grounded in the generated charts (§ 2.1.2 ). Using this pipeline, we construct DuoChart -100k, which contains high-quality charts and 100k challenging QA samples. Second, to address the limitations of text-based reasoning for fine-grained visual and numerical chart understanding, we introduce CharTool , which equips MLLMs with tool-integrated reasoning capabilities (§ 2.2 ). In particular, CharTool introduces an image-cropping tool for capturing localized visual cues, together with code-based computation tools for accurate numerical calculation. We train the model using end-to-end reinforcement learning ( Jin et al. 2025 ; Li et al. 2025b ; Qian et al. 2025 ; Song et al. 2025 ) within an isolated sandbox environment, with tool-aware reward to guide effective tool usage. This enables the model to learn how to invoke tools, generate executable tool actions, and incorporate returned observations into subsequent reasoning, thereby supporting effective multi-step, tool-integrated reasoning. Experimental results on six chart reasoning benchmarks show that CharTool , trained on DuoChart and equipped with tool-grounded reasoning, consistently improves performance across different model scales (§ 3.3 ). Notably, on the challenging real-world benchmark CharXiv ( Wang et al. 2024 ) , CharTool -7B achieves an accuracy of 50.5%, corresponding to an 8.0% absolute improvement . Moreover, CharTool -7B yields a 9.78% absolute improvement on ChartQAPro ( Masry et al. 2025a ) . Furthermore, CharTool demonstrates strong generalization to three out-of-domain visual mathematical reasoning benchmarks (§ 3.4 ), indicating that the tool-integrated reasoning ability learned from charts can transfer beyond the chart domain. Our main contributions are summarized as follows: • We develop a multi-agent chart synthesis and question–answer generation pipeline that produces diverse, high-quality charts and challenging QA pairs. • We enable MLLMs to leverage external tools for precise numerical computation and fine-grained visual reasoning, with agentic reinforcement learning used to train effective tool-integrated chart reasoning. • We demonstrate the effectiveness of our approach across six chart reasoning benchmarks, achieving substantial gains on challenging real-world datasets and showing strong generalization to out-of-domain visual mathematical reasoning tasks. Figure 2 : Data synthesis pipeline of DuoChart . (A). Chart images are constructed from two sources prior to quality filtering: a scalable LLM-based code synthesis pipeline and real-world chart mining. (B). High-quality QAs, named DuoChart , are generated by metadata-guided QA generation followed by rigorous four-stage quality validation. (C). Cold-start trajectories are synthesized by an advanced MLLM-powered Tool Agent. 2 Method In this section, we first describe DuoChart , our multi-agent chart data synthesis pipeline, with a focus on chart image construction and question-answer generation. We then introduce CharTool , our tool-integrated reasoning framework for chart understanding. 2.1 Multi-Agent Chart Data Synthesis