METAL : A Multi-Agent Framework for Chart Generation with Test-Time Scaling

Bingxuan Li † Yiwei Wang Jiuxiang Gu ‡ Kai-Wei Chang † Nanyun Peng † † University of California, Los Angeles · 2025

Read METAL : A Multi-Agent Framework for Chart Generation with Test-Time Scaling by Bingxuan Li † Yiwei Wang Jiuxiang Gu ‡ Kai-Wei Chang † Nanyun Peng † † University of California, Los Angeles (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17651.txt.

Introduction

Bingxuan Li, Yiwei Wang, Jiuxiang Gu, Kai-Wei Chang, and Nanyun Peng automate chart making in this two thousand twenty-five paper. Generating a good chart means writing code that encodes exact texts, layout, colors, and chart type, demanding both visual design sense and precise coding from one model. Direct prompting of vision-language models handles that poorly. METAL instead splits the job among specialized agents that collaborate iteratively, critiquing and correcting each other's work.

The framework improved on the previous best chart generation result by about five percent. It also showed test-time scaling: performance rose steadily as the allowed token budget grew across a wide range. Separating modalities during the critique stage, judging visuals and code apart, strengthened the models' self-correction. The authors target automatic professional report generation in finance, research presentation, education, and healthcare, where charts must be both accurate and presentable.

Catalog record

CREATORBingxuan Li † Yiwei Wang Jiuxiang Gu ‡ Kai-Wei Chang † Nanyun Peng † † University of California, Los Angeles
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-17651

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

METAL : A Multi-Agent Framework for Chart Generation with Test-Time Scaling Bingxuan Li † Yiwei Wang Jiuxiang Gu ‡ Kai-Wei Chang † Nanyun Peng † † University of California, Los Angeles Abstract Chart generation aims to generate code to produce charts satisfying the desired visual properties, e.g., texts, layout, color, and type. It has great potential to empower the automatic professional report generation in financial analysis, research presentation, education, and healthcare. In this work, we build a vision-language model (VLM) based multi-agent framework for effective automatic chart generation. Generating high-quality charts requires both strong visual design skills and precise coding capabilities that embed the desired visual properties into code. Such a complex multi-modal reasoning process is difficult for direct prompting of VLMs. To resolve these challenges, we propose METAL ( M ulti-ag E n T fr A mework with vision L anguage models for chart generation), a multi-agent framework that decomposes the task of chart generation into the iterative collaboration among specialized agents. METAL achieves a 5.2% improvement in the F1 score over the current best result in the chart generation task. Additionally, METAL improves chart generation performance by 11.33% over Direct Prompting with LLaMA 3.2-11b . Furthermore, the METAL framework exhibits the phenomenon of test-time scaling: its performance increases monotonically as the logarithm of computational budget grows from 2 9 2^{9} to 2 13 2^{13} tokens. 1 Introduction Data visualization through charts is an important part of the communication and research life cycle. Well-designed visualizations help distill complex data into digestible insights, allowing researchers, analysts, and stakeholders to identify relationships that might remain hidden in raw data ( Qin et al., 2020 ; Xu et al., 2023 ; Yang et al., 2024 ) . Figure 1: Direct prompting of current VLMs (e.g. GPT-4o ) often fails to generate charts that accurately replicate reference charts, resulting in errors in structure, color, and text alignment. Our proposed approach, METAL , tackle this challenge with iterative refinement through generation, critique, and revision. Our experiments show that increasing the logarithm of test-time compute recurrence and token usage leads to improved performance. Recent advancements in vision language models (VLMs), such as GPT-4V OpenAI (2023) and LLaVA Li et al. (2024) , have expanded the capabilities of language models in tackling complex multimodal problem-solving tasks. These breakthroughs have sparked growing interests in designing intelligent AI assistants to help humans with limited coding expertise create compelling charts, leading to the emergence of a complex multi-modal generation task with crucial practical value - Chart to code generation task ( Wu et al., 2024a ; Han et al., 2023 ; Shi et al., 2024 ) . The chart to code generation task focuses on automatically generating visualization code based on visual references. This task embodies a highly challenging visually-grounded code generation problem that demands robust visual understanding and advanced reasoning. The model must interpret complex visual elements—such as layouts, color schemes, and data relationships—and translate them into syntactically correct, semantically meaningful code. Successfully addressing this challenge not only improves chart replication but also paves the way for advancing the general capabilities of VLMs in multimodal learning and program synthesis. As illustrated in Figure 1 , current state-of-the-art VLMs, such as GPT-4o, often fail to accurately interpret and reproduce the intricate visual elements and relationships embedded in reference charts. Existing solutions, such as Best-of-N and Hint-enhanced Wang et al. (2024) , have not effectively improved upon direct prompting of VLMs. The core challenge in leveraging VLMs for chart generation lies in effectively integrating visual comprehension with code synthesis. This complex task exceeds the capabilities of the single model or single agent. In this paper, we present METAL , a multi-agent framework designed for chart generation. Our framework decomposes this complex multimodal reasoning task into four specialized roles, each handled by a specialized agent: (1) Generation Agent : Responsible for the initial translation of chart images into the corresponding code. (2) Visual Critique Agent : Analyzes and identifies visual differences between the reference chart and the generated output. (3) Code Critique Agent : Reviews the generated code and suggests improvements to better match the reference chart. (4) Revision Agent : Implements code modifications based on the combined feedback from both critique agents. During inference, these agents collaborate iteratively, critiquing and refining the code until the rendered chart achieves the desired quality. In contrast to existing methods, our approach delivers more concrete and targeted feedback, and iteratively refines outputs through the multi-agentic framework, leading to enhanced chart generation performance. Experiment results show that our framework achieves a 5.2% improvement over the current best result in the chart generation task. Additionally, METAL improves chart generation performance by 11.33% over Direct Prompting with LLaMA 3.2-11b , demonstrating its potential to significantly enhance VLMs’ ability to integrate visual understanding with code synthesis. Furthermore, we have two key findings through in-depth analysis: (1) Test-time scaling in the METAL framework : Recent works suggest the potential of scaling the number of tokens (computational budget) to enhance the reasoning performance of LLMs ( Snell et al., 2024 ; Muennighoff et al., 2025 ) . Specifically, utilizing more tokens during inference may lead to improved performance. In this work, we found that there is a near-linear relationship between the performance and the logarithm of the computational budget in experiments. Specifically, the performance of METAL increases monotonically as the logarithm of the computational budget grows from 2^{9} to 2^{13} tokens. (2) Modality-tailored critiques enhance self-correction : We observe that explicitly separating different modalities during the critique process—such as visual evaluation and code analysis—substantially enhances the multimodal self-correction capabilities of VLMs. Ablation study shows that METAL with the separate-critique design achieves a 5.6% improvement over the single-critique baseline. In summary, we present METAL , a VLMs-based multi-agent framework, which achieves significant improvements over the current best methods for chart generation, and our insights into test-time scaling and multi-modal critique offer a promising pathway for enhanced visually-grounded code generation with VLMs. 2 Related Works We discuss three lines of related work: chart-to-code generation, multi-agent framework, and test-time scaling research. 2.1 Chart Generation with VLMs Chart generation, or chart-to-code generation, is an emerging task aimed at automatically translating visual representations of charts into corresponding visualization code Shi et al. (2024) ; Wu et al. (2024a) . This task is inherently challenging as it requires both visual understanding and precise code synthesis, often demanding complex reasoning over visual elements. Recent advances in Vision-Language Models (VLMs) have expanded the capabilities of language models in tackling complex multimodal problem-solving tasks, such as visually-grounded code generation. Leading proprietary models, such as GPT-4V OpenAI (2023) , Gemini Google (2023) , and Claude-3 Anthropic (2024) , have demonstrated impressive capabilities in understanding complex visual patterns. The open-source community has contributed models like LLaVA Xu et al. (2024) ; Li et al. (2024) , Qwen-VL Bai et al. (2023) , and DeepSeek-VL Lu et al. (2024) , which provide researchers with greater flexibility for specific applications like chart generation. Despite these advancements, current VLMs often struggle with accurately interpreting chart structures and faithfully reproducing visualization code. Figure 2: Overview of METAL : A multi-agents system that consists of four specialized agents working in an iterative pipeline: (1) Generation Agent

Read the complete text