Effective Training Data Synthesis for Improving MLLM Chart Understanding
Yuwei Yang, Zeyu Zhang, Yunzhong Hou, Zhuowan Li, Gaowen Liu, Ali Payani, Yuan-Sen Ting, Liang Zheng · 2025
Read Effective Training Data Synthesis for Improving MLLM Chart Understanding by Yuwei Yang, Zeyu Zhang, Yunzhong Hou, Zhuowan Li, Gaowen Liu, Ali Payani, Yuan-Sen Ting, Liang Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06492.txt.
Introduction
Yuwei Yang and colleagues generated synthetic charts to teach multimodal models to read graphs in 2025. Chart reading is core to scientific assistants, yet open models answer only about a third to half of hard chart questions correctly. Earlier synthetic charts looked too clean and regular, so training on them did not transfer to messy real figures in papers.
Their five-step pipeline separates data from plotting code, builds multi-panel figures step by step, varies visual details widely, filters weak examples, and writes question-answer pairs with a strong model. The resulting dataset holds over ten thousand charts and hundreds of thousands of questions. Fine-tuned models read complex real charts better, showing synthetic data works when it mimics real variety.
Catalog record
| CREATOR | Yuwei Yang, Zeyu Zhang, Yunzhong Hou, Zhuowan Li, Gaowen Liu, Ali Payani, Yuan-Sen Ting, Liang Zheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06492
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Effective Training Data Synthesis for Improving MLLM Chart Understanding Yuwei Yang, Zeyu Zhang, Yunzhong Hou, Zhuowan Li, Gaowen Liu, Ali Payani, Yuan-Sen Ting, Liang Zheng Abstract † † *This work was supported in part by the Australian Research Council under Discovery Project DP210102801 and Future Fellowship FT240100820, as well as by the National Science Foundation under Grant No. AST-2406729. Being able to effectively read scientific plots, or chart understanding, is a central part toward building effective agents for science. However, existing multimodal large language models (MLLMs), especially open-source ones, are still falling behind with a typical success rate of 30%-50% on challenging benchmarks. Previous studies on fine-tuning MLLMs with synthetic charts are often restricted by their inadequate similarity to the real charts, which could compromise model training and performance on complex real-world charts. In this study, we show that modularizing chart generation and diversifying visual details improves chart understanding capabilities. In particular, we design a five-step data synthesis pipeline, where we separate data and function creation for single plot generation, condition the generation of later subplots on earlier ones for multi-subplot figures, visually diversify the generated figures, filter out low quality data, and finally generate the question-answer (QA) pairs with GPT-4o. This approach allows us to streamline the generation of fine-tuning datasets and introduce the effective chart dataset (ECD), which contains 10k+ chart images and 300k+ QA pairs, covering 25 topics and featuring 250+ chart type combinations with high visual complexity. We show that ECD consistently improves the performance of various MLLMs on a range of real-world and synthetic test sets. Code, data and models are available at: https://github.com/yuweiyang-anu/ECD. Figure 1 : Overview of the effective chart dataset (ECD). The QA-referenced chart has been adjusted for readability. The ECD is composed of chart images and question-answer (QA) pairs and is intended as a training set for chart understanding. It includes 29 chart types and over 250 subplot combinations, offering more than 10k images and 300k QA pairs with high quality. 1 Introduction Chart understanding aims to enable MLLMs to accurately answer both descriptive questions ( e.g . , “what is the label of the x -axis?”) that focus on recognizing basic chart elements, and reasoning questions ( e.g . , “which city demonstrates the second-highest GDP growth rate?”) that require analytical capabilities. Several benchmarks extracted from real scientific papers, such as CharXiv [ 44 ] , have been developed to evaluate model performance on chart understanding tasks. But the performance of open-source models on these benchmarks remains mediocre, highlighting the need for more effective data and training approaches. An advantage of chart data compared to natural images is that charts can be accurately synthesized using code, data, and standard libraries like Python’s plotting packages [ 19 ] . Unlike natural images which require extensive collection and annotation efforts and may suffer from low faithfulness when generated synthetically, charts can be created programmatically with high fidelity. This characteristic enables cost-efficient training data synthesis, an approach widely explored in the chart understanding field [ 47 , 15 , 38 , 49 , 14 , 46 , 22 , 23 ] . However, existing synthetic chart training datasets suffer from limited similarity to the real data. Early datasets such as PlotQA [ 39 ] and OpenCQA [ 24 ] , while having some data variations, use fixed generation code templates with restricted modularity, resulting in only a few chart categories with limited style variations and inauthentic styles. ChartBench [ 49 ] has improved data diversity by generating data table separately but its visual diversity is limited by the codes for plotting the charts. The recent ReachQA dataset [ 15 ] , on the contrary, allows the code to be generated alongside the data table for the chart. However, this design limits the data patterns, further hurting the chart complexity. These limitations compromise their similarity to and, therefore, effectiveness on real-world data. Our work considers the above limitations and introduces a data synthesis pipeline that generates high-quality charts and question-answer (QA) pairs to mimic real charts. The pipeline begins with human-defined chart functions and separated data generation, so that GPT-4o [ 4 ] can focus on generating more complex data. These data and functions are fed into Python to render a single plot. To create coherent multi-subplot charts, we use previous subplots as a condition when generating new subplots. We further diversify the charts by randomly adding visual elements such as annotations, area shadings, arrows, zoom-in insets, and subplot titles. Then, we rate the generated charts based on visual clarity and semantic coherence, filtering out low-quality examples before finally generating QA pairs using GPT-4o. This pipeline allows us to create the effective chart dataset (ECD). It has over 10,000 images and 300,000+ QA pairs, belonging to 29 chart types and over 250 combinations of different chart types (Fig. 1 ). By fine-tuning open-source MLLMs with ECD, we show overall improvements on various test sets and that our improvements surpass gains achieved with existing chart training sets. Figure 2 : Our chart generation pipeline consists of five key steps: I. Single plot generation (Sec. 3.1 ) creates individual charts using pre-defined chart functions and a separate chart data generator to produce data, titles, labels, and markers for each plot. II. Combined subplot generation (Sec. 3.2 ) produces multi-plot layouts through conditional sequential generation, where each subplot is informed by previous ones to maintain semantic coherence. III. Chart image diversification (Sec. 3.3 ) enhances visual complexity and realism by adding annotations, area shading, zoom-in insets, and other stylistic variations including font modifications and axis border adjustments. IV. Chart image filtering (Sec. 3.4 ) uses rating metrics based on visual clarity and semantic coherence to filter out low-quality chart generations and ensure chart visual quality. V. QA pair generation & filtering (Sec. 3.5 ) creates both descriptive questions and reasoning questions for the chart images, followed by additional quality filtering. 2 Related Works Real-world chart datasets such as ChartQA [ 34 ] , OpenCQA [ 24 ] , and CharXiv [ 44 ] extract charts from online sources, providing genuine examples of visualizations encountered in practice. Despite their authenticity, such real-world datasets suffer from limited scale due to the effort required for collection and annotation. Synthetic chart datasets have gained popularity due to the programmatic nature of chart generation. Early datasets such as FigureQA [ 23 ] , DVQA [ 22 ] , PlotQA [ 39 ] and SimChart9k [ 46 ] only generate basic chart types ( e.g. , line, pie, and bar charts) using template-based approaches and exhibit limited complexity. More recent synthetic datasets, including ChartLlama [ 14 ] , ChartBench [ 49 ] , ChartAssistant [ 38 ] , NovaChart [ 17 ] and ChartX [ 47 ] have expanded the number of images, chart types, and styles, but still demonstrate insufficient visual complexity compared to authentic scientific visualizations. ReachQA [ 15 ] uses large language models for code augmentation, but its data complexity and scale are limited. This work presents a new data synthesis pipeline and dataset, and extensive analysis demonstrates that our approach holistically address these limitations. Multimodal large language models for chart understanding generally adopt two strategies. The first employs a