Improved Iterative Refinement for Chart-to-Code Generation via Structured Instruction
Chengzhi Xu, Yuyang Wang, Lai Wei, Lichao Sun, Weiran Huang, [0.1cm] MIFA Lab, Shanghai Jiao Tong University · 2025
Read Improved Iterative Refinement for Chart-to-Code Generation via Structured Instruction by Chengzhi Xu, Yuyang Wang, Lai Wei, Lichao Sun, Weiran Huang, [0.1cm] MIFA Lab, Shanghai Jiao Tong University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-14837.txt.
Introduction
Chengzhi Xu and co-authors teach large vision-language models to turn chart images into code, in work from 2025. The code, when run, should redraw the same chart. Models are good at seeing, but their first-pass code for charts is often off in bars, labels, or scale.
Their method, ChartIR, fixes code in rounds. First the model writes what it sees in the chart, in a set form. Then it lists how its redrawn chart differs from the source. Those notes guide a fresh pass at the code, and rounds repeat. Tests on an open model and on GPT-4o beat past methods. The split of the job into seeing, then coding, then checking is the core trick. It keeps each step plain enough for the model to do well.
Catalog record
| CREATOR | Chengzhi Xu, Yuyang Wang, Lai Wei, Lichao Sun, Weiran Huang, [0.1cm] MIFA Lab, Shanghai Jiao Tong University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-14837
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improved Iterative Refinement for Chart-to-Code Generation via Structured Instruction Chengzhi Xu, Yuyang Wang, Lai Wei, Lichao Sun, Weiran Huang, [0.1cm] MIFA Lab, Shanghai Jiao Tong University Abstract Recently, multimodal large language models (MLLMs) have attracted increasing research attention due to their powerful visual understanding capabilities. While they have achieved impressive results on various vision tasks, their performance on chart-to-code generation remains suboptimal. This task requires MLLMs to generate executable code that can reproduce a given chart, demanding not only precise visual understanding but also accurate translation of visual elements into structured code. Directly prompting MLLMs to perform this complex task often yields unsatisfactory results. To address this challenge, we propose ChartIR, an iterative refinement method based on structured instruction. First, we distinguish two tasks: visual understanding and code translation. To accomplish the visual understanding component, we design two types of structured instructions: description and difference. The description instruction captures the visual elements of the reference chart, while the difference instruction characterizes the discrepancies between the reference chart and the generated chart. These instructions effectively transform visual features into language representations, thereby facilitating the subsequent code translation process. Second, we decompose the overall chart generation pipeline into two stages: initial code generation and iterative refinement, enabling progressive enhancement of the final output. Experimental results show that, compared to other method, our method achieves superior performance on both the open-source model Qwen2-VL and the closed-source model GPT-4o. 1 Introduction Recently, Multimodal Large Language Models (MLLMs) bai2023qwenvlversatilevisionlanguagemodel ; lu2024deepseekvlrealworldvisionlanguageunderstanding ; geminiteam2024geminifamilyhighlycapable ; openai2024gpt4ocard ; liu2023visualinstructiontuning , such as GPT-4V, LLaVA, and Qwen-VL, have attracted significant attention due to their impressive capabilities in visual understanding and reasoning. Among the many downstream tasks involving multimodal inputs, chart reasoning has emerged as a important domain xia2025chartxchartvlmversatile ; masry2022chartqabenchmarkquestionanswering ; han2023chartllamamultimodalllmchart ; liu2024mmcadvancingmultimodalchart ; he2024distillvisualchartreasoning , as charts play a crucial role in presenting structured data in a visual format. However, recent studies show that MLLMs still struggle to handle chart-related tasks effectively, including summarization, question answering, and particularly chart-to-code generation. The unique difficulty of chart inputs—such as their high information density and complex layout—renders direct learning from vision-language corpora insufficient. Therefore, existing MLLMs generally perform poorly on chart-to-code generation benchmarks, due to their inability to accurately capture and translate multi-dimensional chart information. To address the limitations of direct generation, some recent efforts have attempted to enhance the chart-to-code capability of MLLMs through more structured approaches. ChartCoder zhao2025chartcoderadvancingmultimodallarge improves performance by decomposing the generation process into step-by-step procedures and applying instruction tuning to open-source models. However, due to its reliance on training, ChartCoder cannot be applied to commercial closed-source models like GPT-4o, and its final performance remains suboptimal. On the other hand, METAL li2025metalmultiagentframeworkchart adopts a training-free, multi-agent system that enables compatibility with both open-source and commercial MLLMs. It iteratively refines the generated code through a pipeline of generation, critique, and revision based on visual evaluation metrics. Nevertheless, METAL suffers from critical limitations: it updates code based only on the lowest-performing single metric, which often leads to unbalanced optimization, and some metrics such as color are rarely used in practice (see Table 1 ). Moreover, optimizing one metric may degrade others, and relying on a single score for update decisions can result in code that is worse overall. These issues hinder METAL’s ability to generate high-quality, visually faithful chart code. Table 1: Total usage counts of three metrics Metric Plot2Code ChartMimic Text 93 694 Color 0 4 Overall 108 554 Total samples 201 1252 To better understand the unique challenges behind chart-to-code generation, we conduct a detailed analysis and identify two core bottlenecks: visual understanding and code translation . Visual understanding refers to the MLLM’s ability to extract accurate and multi-dimensional information—such as structure, text, colors, and layout—from the input chart. Code translation involves converting this rich visual information into executable and semantically faithful code. Motivated by this observation, we propose ChartIR, a training-free iterative framework that explicitly addresses both challenges. To enhance the visual understanding phase, we introduce a structured, multi-dimensional chart description mechanism that provides detailed annotations over several key visual components. To improve code translation, we design an iterative generation scheme that leverages both the original image and the chart description to guide code refinement. In each iteration, the model compares differences between the generated chart and the reference chart across all key dimensions–text, color, layout, and type–enabling more holistic and targeted updates. To validate the effectiveness of ChartIR, we conduct extensive experiments on two widely-used chart-to-code benchmarks: Plot2Code wu2024plot2codecomprehensivebenchmarkevaluating and ChartMimic yang2025chartmimicevaluatinglmmscrossmodal . We evaluate our method on both open-source (Qwen2-VL) and closed-source (GPT-4o) models. Results show that ChartIR significantly outperforms both direct generation and METAL across a comprehensive set of evaluation metrics, including GPT-4o Score, low-level metrics (Text, Layout, Type, Color), and traditional image similarity metrics (SSIM, CLIP, MSE, PSNR). Notably, the improvements in GPT-4o Score—an evaluation metric closely aligned with human judgment—demonstrate that our approach leads to substantial gains in visual and structural fidelity of the generated charts. We further conduct ablation studies to isolate the effects of chart description and iterative difference-based refinement. Results show that removing either component leads to considerable performance degradation, confirming the necessity and effectiveness of each module in the ChartIR framework. In summary, our contributions are threefold: (1) We propose ChartIR, a training-free, model-agnostic framework for enhancing chart-to-code generation in MLLMs; (2) We introduce a structured chart description mechanism that captures multi-dimensional visual features of the reference chart, which is shown to be essential for accurate code generation; and (3) We design an iterative difference-based refinement process that improves code quality across all visual aspects. Our method achieves state-of-the-art results on multiple benchmarks both on open-source and closed-source models and provides a robust blueprint for future research in chart-to-code generation. 2 Related work 2.1 Multimodal Large Language Models Multimodal large language models (MLLMs) extend the capabilities of large language models (LLMs) by incorporating vision inputs, enabling them to perform tasks involving both text and images. Open-source MLLMs such as Qwen-VL bai2023qwenvlversatilevisionlanguagemodel , DeepSeek-VL lu2024deepseekvlrealworldvisionlanguageunderstanding , LLaVA liu2023visualinstructiontuning , and Mini-Gemini geminiteam2024geminifamilyhighlycapable have attracted attention due to their transparency, accessibility, and potential for fine-tuning. These models are often built by aligning pre-trained vision encoders with language models using multimodal instruction tuning. On the other hand, closed-source commercial models such as GPT-4o openai2024gpt4ocard and Claude anthropic2024claude3 integrate proprietary vision-language pipelines and demonstrate superior performance across a wide range of multimodal tasks. In chart-related tasks, including chart-to-code generation, these closed-source models consistently outperform their open-source counterparts. However, their closed nature restricts research access, model interpretability, and downstream adaptability, especially for tasks requiring task-specific customization or iterative optimization. 2.2 Chart-to-Code Generation Chart-to-code generation aims to convert visual charts into executable code that reproduces the original figure. Recent studies xia2025chartxchartvlmversatile ; han2023chartllamamultimodalllmchart ; liu2024mmcadvancingmultimodalchart ; zhang2025enhancingcharttocodegenerationmultimodal have evaluated MLLMs on this task. These works show that both open-source models (e.g., Qwen-VL, DeepSeek-VL) and closed-source models (e.g., GPT-4o, Claude) still struggle with accurately reproducing charts. Among the most recent and advanced approaches, ChartCoder improves performance by decomposing the task into sequential substeps and training dedicated modules, but is limited to open-source models and still underperforms closed-source ones. METAL introduces a training-free multi-agent framework that critiques and revises code iteratively, allowing compatibility