Visual Programmability: A Guide for Code-as-Thought in Chart Understanding

Bohao Tang, Fei Zhang, Jiadi Su, Ethan Chern, Zhulin Hu, Zhixin Wang, Pengfei Liu, Ya Zhang, [0.6em] Shanghai Jiao Tong University · 2025

Read Visual Programmability: A Guide for Code-as-Thought in Chart Understanding by Bohao Tang, Fei Zhang, Jiadi Su, Ethan Chern, Zhulin Hu, Zhixin Wang, Pengfei Liu, Ya Zhang, [0.6em] Shanghai Jiao Tong University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-09286.txt.

Introduction

Bohao Tang and 8 co-authors propose Visual Programmability for chart understanding in a 2025 paper. Charts test the reasoning of vision language models. Code as thought represents chart information in verifiable symbolic form, but a fixed code only approach fails on complex charts.

Visual programmability is a learnable property that decides if a chart question suits a code based route or direct looking. A model learns to choose between the two pathways. Reinforcement learning uses a dual reward: one for data accuracy, one for choosing the right strategy. That prevents defaulting to a single mode. Experiments show strong, robust performance across diverse chart benchmarks.

Catalog record

CREATORBohao Tang, Fei Zhang, Jiadi Su, Ethan Chern, Zhulin Hu, Zhixin Wang, Pengfei Liu, Ya Zhang, [0.6em] Shanghai Jiao Tong University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-09286

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Visual Programmability: A Guide for Code-as-Thought in Chart Understanding Bohao Tang, Fei Zhang, Jiadi Su, Ethan Chern, Zhulin Hu, Zhixin Wang, Pengfei Liu, Ya Zhang, [0.6em] Shanghai Jiao Tong University Abstract Chart understanding presents a critical test to the reasoning capabilities of Vision-Language Models (VLMs). Prior approaches face critical limitations: some rely on external tools, making them brittle and constrained by a predefined toolkit, while others fine-tune specialist models that often adopt a single reasoning strategy, such as text-based chain-of-thought (CoT). The intermediate steps of text-based reasoning are difficult to verify, which complicates the use of reinforcement-learning signals that reward factual accuracy. To address this, we propose a Code-as-Thought (CaT) approach to represent the visual information of a chart in a verifiable, symbolic format. Our key insight is that this strategy must be adaptive: a fixed, code-only implementation consistently fails on complex charts where symbolic representation is unsuitable. This finding leads us to introduce Visual Programmability : a learnable property that determines if a chart–question pair is better solved with code or direct visual analysis. We implement this concept in an adaptive framework where a VLM learns to choose between the CaT pathway and a direct visual reasoning pathway. The selection policy of the model is trained with reinforcement learning using a novel dual-reward system. This system combines a data-accuracy reward to ground the model in facts and prevent numerical hallucination, with a decision reward that teaches the model when to use each strategy, preventing it from defaulting to a single reasoning mode. Experiments demonstrate strong and robust performance across diverse chart-understanding benchmarks. Our work shows that VLMs can be taught not only to reason but also how to reason, dynamically selecting the optimal reasoning pathway for each task. 2 2 footnotetext: Open-source implementation are available at github.com/Aphelios-Tang/Code-as-Thought . 1 Introduction Figure 1: Adaptive Reasoning vs. Fixed Strategies for Chart Understanding. Prevailing approaches are limited by their rigid strategies. (a) Tool-Use Models are constrained by a predefined toolkit and fail on novel tasks. (b) Specialized Models employ a single reasoning pattern (e.g., text-only or code-only), which limits their generalization. In contrast, our (c) Adaptive Framework first assesses a task’s Visual Programmability. It then dynamically selects the precise Code-as-Thought pathway for programmable tasks or the robust Direct Visual Reasoning pathway for complex ones, achieving superior performance across all chart types. The capabilities of Vision-Language Models (VLMs), built upon powerful Large Language Models brown2020language ; touvron2023llama , have rapidly advanced multimodal understanding (e.g., radford2021learning ; liu2023visual ; achiam2023gpt ; comanici2025gemini ; bai2025qwen2 ). Among the many applications, chart understanding stands out as a critical benchmark huang2024pixels , testing an AI’s ability to connect low-level visual perception lee2023pix2struct with high-level logical inference. Despite significant progress with specialized models cheng2023chartreader ; masry2023unichart ; meng2024chartassisstant , a fundamental generalization problem remains: even state-of-the-art VLMs show a stark performance decline on the complex, "in-the-wild" charts found in real-world contexts islam2024large ; wang2024charxiv . Prevailing efforts to overcome this generalization challenge have largely followed two dominant strategies, each with distinct drawbacks. The first approach treats the VLM as a controller for external tools and APIs huang2025chartsketcher ; gupta2023visual ; suris2023vipergpt (see Figure 1 a). While powerful, their reliance on a predefined toolkit makes them brittle when encountering charts that require capabilities beyond their predefined functions schick2023toolformer ; yao2023react ; patil2024gorilla ; parisi2022talm . The second strategy involves fine-tuning specialized models on chart-specific data cheng2023chartreader ; masry2023unichart ; meng2024chartassisstant (see Figure 1 b). These models typically rely on a monolithic reasoning pattern —that is, they exclusively use a single mode of thought, such as text-based Chain-of-Thought or code-based reasoning. This lack of flexibility hinders their ability to generalize to out-of-distribution (OOD) visualizations, as no single reasoning style is optimal for all chart types wang2024charxiv ; xu2023chartbench . The limitations of predefined toolkits highlight the appeal of a more universal and flexible tool: code. Unlike a fixed API, code can be dynamically generated to create novel tools tailored to the specific visual complexities of any chart, a concept explored in recent agentic vision systems zhao2025pyvision . However, the shared failure of rigid approaches motivates our core insight: the optimal reasoning strategy depends on the task itself. Some charts are easily broken down into programmable elements dai2024msg , while others require a holistic visual analysis that code cannot capture. This requires moving beyond refining a single reasoning chain wei2022chain to mastering strategy selection—a shift that reflects a broader trend in AI towards deliberate problem-solving wang2022self ; shinn2023reflexion ; yao2023tree and adaptive computation graves2016adaptive . This principle is also central to the design of frontier models like GPT-5 openai2025gpt5systemcard , which aim to integrate similar adaptive capabilities. To address these challenges, we propose the concept of Visual Programmability : a learnable, task-dependent property that indicates whether a given chart-question pair is best solved through programmatic reasoning or direct visual analysis. We implement this concept in an adaptive framework that enables a VLM to autonomously choose its reasoning pathway. The model’s decision-making policy is trained via reinforcement learning (RL)—specifically, using the Group Relative Policy Optimization (GRPO) algorithm—guided by a novel dual-reward system . This system is carefully designed to foster adaptive behavior: a data-accuracy reward ensures the generated code is factually grounded to the chart’s content, thereby preventing numerical hallucination. In parallel, a dedicated decision reward explicitly teaches the model the boundaries of programmability, preventing the policy from collapsing into a single, suboptimal mode. Our experiments, conducted on the Qwen2.5-VL model bai2025qwen2 across a diverse suite of benchmarks, validate our approach. The resulting adaptive model consistently outperforms both pure visual baselines and rigid code-based methods. It achieves this by dynamically modulating its strategy, heavily employing code-based reasoning ( >60% ) on benchmarks where it is advantageous, while minimizing its use (<10%) where it is detrimental. Ablation studies confirm that our dual-reward system is essential for preventing mode collapse and fostering strategic diversity. Our contributions are threefold: • We introduce Visual Programmability , a novel concept to determine if a chart task is suitable for code-based reasoning, serving as the foundation for adaptive strategy selection. • Building on this concept, we develop an adaptive framework that learns to choose the optimal reasoning path (code or vision). This framework is trained with a specialized dual-reward RL system that promotes both factual accuracy and strategic flexibility. • Our adaptive model demonstrates outstanding performance and generalization , consistently outperforming rigid strategies across diverse benchmarks by intelligently switching between reasoning modes. 2 Related Work 2.1 Programmatic Reasoning for Chart Understanding The field of chart understanding has evolved rapidly, driven by more capable models and challenging benchmarks. Early research established foundational datasets and tasks methani2020plotqa ; masry2022chartqa . The arrival of powerful Vision-Language Models (VLMs) li2022blip ; li2023blip ; dai2023instructblip and frontier systems achiam2023gpt ; comanici2025gemini ; bai2025qwen2 led to strong performance on these benchmarks, prompting the development of specialized, fine-tuned models liu2022deplot ; cheng2023chartreader ; masry2023unichart ; han2023chartllama ; meng2024chartassisstant . However, their success was often misleading, as they tended to learn benchmark-specific shortcuts rather than generalizable reasoning skills. This weakness was exposed by a new wave of diverse and complex benchmarks xia2024chartx ; wang2024charxiv ; xu2023chartbench ; yang2024chartmimic , where even state-of-the-art models showed a significant

Read the complete text