Chain of Functions: A Programmatic Pipeline for Fine-Grained Chart Reasoning Data

Zijian Li, Jingjing Fu, Lei Song, Jiang Bian, Jun Zhang, Rui Wang · 2025

Read Chain of Functions: A Programmatic Pipeline for Fine-Grained Chart Reasoning Data by Zijian Li, Jingjing Fu, Lei Song, Jiang Bian, Jun Zhang, Rui Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16260.txt.

Introduction

Zijian Li, Jingjing Fu, Lei Song, Jiang Bian, Jun Zhang, and Rui Wang tackle a data problem. AI models that reason about charts need good training examples. Asking a big model to write them freely produces sloppy, samey material.

Their pipeline, Chain of Functions, starts from small building-block operations such as finding a maximum or doing arithmetic. Software explores chains of these steps on its own. Only then are the chains turned into worded explanations and questions by a modest open model. Because every step is a real function call, the reasoning stays exact, and the variety of chains yields varied questions. Training on their dataset pushed models to top scores among their peers on standard chart tests.

Catalog record

CREATORZijian Li, Jingjing Fu, Lei Song, Jiang Bian, Jun Zhang, Rui Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16260

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Chain of Functions: A Programmatic Pipeline for Fine-Grained Chart Reasoning Data Zijian Li, Jingjing Fu, Lei Song, Jiang Bian, Jun Zhang, Rui Wang Abstract Visual reasoning is crucial for multimodal large language models (MLLMs) to address complex chart queries, yet high-quality rationale data remains scarce. Existing methods leveraged (M)LLMs for data generation, but direct prompting often yields limited precision and diversity. In this paper, we propose Chain of Functions (CoF) , a novel programmatic reasoning data generation pipeline that utilizes freely-explored reasoning paths as supervision to ensure data precision and diversity. Specifically, it starts with human-free exploration among the atomic functions (e.g., maximum data and arithmetic operations) to generate diverse function chains, which are then translated into linguistic rationales and questions with only a moderate open-sourced LLM. CoF provides multiple benefits: 1) Precision: function-governed generation reduces hallucinations compared to freeform generation; 2) Diversity: enumerating function chains enables varied question taxonomies; 3) Explainability: function chains serve as built-in rationales, allowing fine-grained evaluation beyond overall accuracy; 4) Practicality: eliminating reliance on extremely large models. Employing CoF , we construct the ChartCoF dataset, with 1.4k complex reasoning Q&A for fine-grained analysis and 50k Q&A for reasoning enhancement. The fine-grained evaluation on ChartCoF reveals varying performance across question taxonomies for each MLLM, and the experiments also show that finetuning with ChartCoF achieves state-of-the-art performance among same-scale MLLMs on widely used benchmarks. Furthermore, the novel paradigm of function-governed rationale generation in CoF could inspire broader applications beyond charts. The code and data will be publicly available at https://github.com/microsoft/Chain-of-Functions . Figure 1: Our proposed CoF constructs a high-quality reasoning dataset ChartCoF for the fine-grained evaluation and reasoning enhancement of MLLMs. 1 Introduction Recent advancements in large language models (LLMs) Chowdhery et al. (2023) ; Dubey et al. (2024) ; Guo et al. (2025) have paved the way for the development of multi-modal large language models (MLLMs) Liu et al. (2024b) ; Bai et al. (2023b) , which have demonstrated a remarkable ability to understand visual semantics through the alignment between visual and embedding spaces. Despite this progress, current MLLMs exhibit limitations in their reasoning capabilities and encounter difficulties in accurately interpreting charts in scholarly articles and financial documents Xu et al. (2023) ; Xia et al. (2024) . This is particularly evident when they handle complex reasoning questions that necessitate accurate and step-by-step thought processes Wang et al. (2024b) . The analysis in ChartQA Masry et al. (2022) , as shown in Table 4 , highlights a significant performance discrepancy between complex reasoning questions (Human set) and simpler perceptual questions (Augmented set). For instance, InternVL-2.5-8B Chen et al. (2024b) demonstrates a performance gap of nearly 20%, which underscores the challenges that MLLMs face in bridging the gap between human-like reasoning and current computational capabilities. Training with chain-of-thought (CoT) data has emerged as an effective strategy to enhance the reasoning abilities of MLLMs for chart understanding Wei et al. (2022) ; Zhang et al. (2024c) ; Zhang et al. (2024d) ; He et al. (2024) . Nonetheless, high-caliber CoT data are scarce for chart reasoning, which require complete reasoning processes and accurate chart information (e.g., object values and positions) in the rationales Dong et al. (2024) ; Masry et al. (2024b) ; He et al. (2024) . To generate CoT data, recent investigations have leveraged the capabilities of advanced (M)LLMs to autonomously produce questions, answers, and their corresponding rationales by either directly analyzing the charts or their textual descriptions with well-designed prompts Liu et al. (2024a) ; He et al. (2024) ; Masry et al. (2024b) . Despite these efforts, directly prompting (M)LLMs to generate questions and rationales based only on charts may result in low accuracy and limited diversity. Moreover, relying on extremely large (M)LLMs poses a notable barrier to the data scalability. In addition to the scarcity of CoT data for effective finetuning, the evaluation of MLLMs’ reasoning capabilities remains underexplored. While current benchmarks have incorporated reasoning questions to evaluate the reasoning capabilities of MLLMs Masry et al. (2022) ; Xia et al. (2024) ; Wang et al. (2024b) , these questions often lack complexity and require only short reasoning chains to solve. More importantly, these benchmarks tend to gauge the reasoning performance in a broad sense with an overall accuracy metric, which overlooks the nuanced analysis of MLLMs’ proficiency across questions that require varying reasoning chains. A fine-grained reasoning evaluation of the models’ specific strengths and weaknesses on question taxonomies remains a valuable avenue for research. In response to the scarcity of diverse and high-caliber reasoning datasets for the fine-grained evaluation and enhancement of chart reasoning, as presented in Fig. 1 , we introduce a novel automatic reasoning data synthesis pipeline named Chain of Functions (CoF) . Unlike prior methods that rely on end-to-end LLM prompting, our approach first systematically explores chart elements through a set of atomic functions to ensure correct and diverse reasoning paths and then translate them into language, which greatly reduces hallucinations and enables more precise supervision. Concretely, CoF encompasses two key processes: program-based functional discovery and reverse linguistic CoT data synthesis . In program-based functional discovery, we carefully design atomic functions and their corresponding conditions, which are intelligently combined to form a coherent function chain based on a chart. Then in the reverse linguistic CoT data synthesis process, these function chains are translated into natural language instructions using LLMs in a reverse manner, with rationales first, and then questions. This method ensures the precision of questions, rationales, and answers. Crucially, since the reasoning process is determined by the function chain rather than by generative prompts alone, we can leverage a moderate open-sourced LLM (Qwen2.5-32B-instruct Yang et al. (2024) used in experiments) for the linguistic transfer, greatly lowering dependence on extremely large models. Furthermore, CoF effectively bridges structured reasoning and language modeling, with potential applications beyond charts. Key contributions: 1) Our proposed reasoning data generation pipeline CoF greatly ensures explainability, precision, and diversity of generated reasoning data, thus enabling the fine-grained evaluation and reasoning enhancement for MLLMs. 2) We introduce ChartCoF , which encompasses an extensive variety of over 19 chart types, with a test set comprising 648 charts paired with 1,451 Q&A pairs and a training set featuring 18,349 charts with 50,329 Q&A pairs for fine-grained evaluation and model finetuning. 3) The fine-grained evaluation reveals the weak performance of existing MLLMs on the complex reasoning questions of ChartCoF and provides deep insight into their skilled and unskilled question taxonomies. 4) Extensive experiments demonstrate that after finetuning with ChartCoF , MLLMs achieve state-of-the-art performance among same-scale MLLMs in three widely used benchmarks on charts. Additional analysis reveals the significant improvement of finetuned MLLMs with ChartCoF on out-of-distribution (OOD), model scalability, and data scalability evaluations. 2 Related Works Multimodal large language models (MLLMs) have aligned the vision space with the embedding space of LLMs for visual understanding Vaswani (2017) ; Radford (2018) ; Brown et al. (2020) ; Zhang et al. (2022) ; Chowdhery et al. (2023) ; Dubey et al. (2024) ; Team (2023) ; Bai et al. (2023a) ; Yin et al. (2023) , which is normally achieved via connectors, e.g., Q-Former Li et al. (2023) or MLP Bai et al. (2023b) . With connectors, Mini-GPT4, mPLUG-Owl, and InstructBLIP

Read the complete text