ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding
Muye Huang, Lingling Zhang, Jie Ma, Han Lai, Fangzhi Xu, Yifei Li, Wenjun Wu, Yaqiang Wu, Jun Liu · 2025
Read ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding by Muye Huang, Lingling Zhang, Jie Ma, Han Lai, Fangzhi Xu, Yifei Li, Wenjun Wu, Yaqiang Wu, Jun Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19076.txt.
Introduction
Charts pack dense data into small spaces, and AI models often misread them when they reason in words alone. Muye Huang and eight co-authors present ChartSketcher in 2025, a method that lets a model draw on the chart as it thinks. Using a sketching library, the model marks its steps right on the image and reads those marks back in.
Training came in two stages: first the model learned the sketching style, then a form of practice learning sharpened its self checking. The team report strong scores on chart tests and on general vision tasks. Because each step leaves a visible mark, people can follow how the model reached its answer. The authors offer it as a more open way to read charts.
Catalog record
| CREATOR | Muye Huang, Lingling Zhang, Jie Ma, Han Lai, Fangzhi Xu, Yifei Li, Wenjun Wu, Yaqiang Wu, Jun Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19076
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding Muye Huang, Lingling Zhang, Jie Ma, Han Lai, Fangzhi Xu, Yifei Li, Wenjun Wu, Yaqiang Wu, Jun Liu Abstract Charts are high-density visualization carriers for complex data, serving as a crucial medium for information extraction and analysis. Automated chart understanding poses significant challenges to existing multimodal large language models (MLLMs) due to the need for precise and complex visual reasoning. Current step-by-step reasoning models primarily focus on text-based logical reasoning for chart understanding. However, they struggle to refine or correct their reasoning when errors stem from flawed visual understanding, as they lack the ability to leverage multimodal interaction for deeper comprehension. Inspired by human cognitive behavior, we propose ChartSketcher, a multimodal feedback-driven step-by-step reasoning method designed to address these limitations. ChartSketcher is a chart understanding model that employs Sketch-CoT, enabling MLLMs to annotate intermediate reasoning steps directly onto charts using a programmatic sketching library, iteratively feeding these visual annotations back into the reasoning process. This mechanism enables the model to visually ground its reasoning and refine its understanding over multiple steps. We employ a two-stage training strategy: a cold start phase to learn sketch-based reasoning patterns, followed by off-policy reinforcement learning to enhance reflection and generalization. Experiments demonstrate that ChartSketcher achieves promising performance on chart understanding benchmarks and general vision tasks, providing an interactive and interpretable approach to chart comprehension. 1 Introduction Charts are widely used as data visualization methods in scientific papers and business reports. Automated chart understanding is a key step in achieving automated data analysis. Recent advances in MLLMs [ 1 , 22 , 35 , 42 , 47 ] have shown substantial progress in chart understanding tasks. These include proprietary models like GPT-4o [ 35 ] and Gemini-2.0 [ 41 ] , as well as open-source models such as Qwen-2VL [ 47 ] and InternVL-2.5 [ 4 ] . The use of MLLMs has become a mainstream approach for chart understanding. However, existing MLLMs face significant challenges in chart understanding, which involves the systematic interpretation and analysis of visual data representations. Chart understanding requires high-precision visual reasoning capabilities to process complex elements such as overlapping data points, multiple intersecting trend lines, and dense numerical information, demanding simultaneous comprehension of both spatial relationships and their semantic meanings. For example, in Figure 1 , answering “What is the value of ‘Good’ in 2015?” requires identifying that 2015 lies between the marked years 2014 and 2016. Models must precisely locate and identify these visual elements while understanding their quantitative relationships to correctly determine that the ‘Good’ value in 2015 is 57. This visual reasoning process requires analysis of visual dependencies and precise numerical understanding at each step. Such complex visual reasoning tasks pose significant challenges to existing approaches. MLLMs [ 3 , 5 , 44 ] have attempted to achieve fine-grained visual reasoning through long chains of thought. For example, multimodal reasoning models like QvQ [ 44 ] demonstrate the capability to generate long-chain reasoning text. However, their effectiveness remains limited in visually intensive scenarios like charts. This limitation stems from their predominant focus on textual logical processes rather than visual information processing, causing reduced interpretability for users and an inability to correct errors originating from flawed multimodal understanding. Recent attempts, such as VisualCoT [ 38 ] , propose image cropping techniques to enhance visual understanding by focusing on key regions. However, the inherent constraints of cropping mechanisms prevent simultaneous analysis of multiple regions, thereby limiting the model’s capacity for complex visual reasoning. As illustrated in Appendix A , this represents a challenging case in existing MLLMs. The development of visual reasoning models that focus on processing complex elements requires urgent exploration. Interestingly, when humans encounter complex visual information, they often create sketches to mark and organize key details. This process helps them break down problems and focus on critical areas within the image. For example, when determining the value of ‘Good’ , humans typically start by locating the relevant position on the x-axis, then trace vertically upward to identify the corresponding value on the colored line: a natural way of decomposing the visual reasoning process. This behavior reflects a subconscious strategy humans use to enhance visual focus and understanding. Drawing inspiration from natural human behaviors, we propose ChartSketcher, a multimodal feedback-driven step-by-step reasoning method that addresses these visual reasoning limitations in chart understanding. Specifically, ChartSketcher employs Sketch-CoT, enabling MLLMs to explicitly annotate their intermediate reasoning processes on images and feed the visual annotations back to themselves, achieving stable step-by-step multimodal reasoning. Moreover, by incorporating reflection processes between steps and leveraging reinforcement learning, we endow MLLMs with human-like reflection capabilities. The model not only marks the visual reasoning process on images but can also identify reasoning errors and promptly correct mistakes from previous steps. As illustrated in Figure 1 , our approach demonstrates powerful visual reasoning capabilities across diverse scenarios. The implementation of ChartSketcher follows a two-stage training pipeline: a cold start phase and an RL phase. In the cold start phase, we transfer reasoning and reflection patterns from LLM to MLLM through cross-modal distillation, creating 300K fine-grained annotated chart understanding samples. The subsequent RL phase employs MCTS and diverse data sampling techniques with over 50K step-by-step reasoning examples to enhance the model’s capabilities through off-policy reinforcement learning. Our main contributions can be summarized in three aspects: • We propose ChartSketcher, a novel multimodal feedback reasoning approach that enhances MLLMs’ visual reasoning capabilities through iterative Sketch-CoT and self-reflection mechanisms. Code and data available at https://github.com/MuyeHuang/ChartSketcher . • We construct a comprehensive dataset of 300K annotated samples for cold start training and 50K curated samples for reinforcement learning. The dataset is designed to support chart step-by-step reasoning. • We conduct extensive experiments across multiple datasets to demonstrate the effectiveness of ChartSketcher. Through comprehensive ablation studies, we investigate the importance of each training stage and validate the contribution of key components in our work. 2 Related Work Chart Understanding. Chart Understanding aims to comprehend the visual context of charts to address specific tasks, such as QA or summarization. FigureQA [ 16 ] stands as a pioneering work, introducing a chart understanding pipeline capable of handling binary classification tasks for chart-related questions. Subsequent works [ 26 , 20 , 24 , 21 ] further enhanced chart understanding capabilities by employing multi-component pipelines. For instance, DePlot [ 58 ] leveraged multiple components combined with the mathematical capabilities of LLMs to achieve performance improvements on PlotQA [ 33 ] . With the advent of MLLMs, approaches utilizing MLLMs as the primary component have become mainstream in the field [ 28 , 29 , 2 , 13 ] . ChartLlama [ 11 ] , through clever data construction and fine-tuning of LLaVA [ 22 ] , built a robust chart-expert model. Leveraging the powerful language capabilities inherent in MLLMs, recent studies have employed multi-task training methodologies to bolster chart understanding across a variety of tasks. ChartAssistant [ 32 ] utilized unified multi-task training to improve overall performance. TinyChart [ 57 ] utilizes the Program-of-Thoughts technique to enhance numerical reasoning capabilities in chart QA tasks. ChartMoE [ 52 ] employed a Mixture of Experts approach to model different chart types effectively, thereby enabling understanding across diverse