Does It Run and Is That Enough ? Revisiting Text-to-Chart Generation with a Multi-Agent Approach

James Ford, Anthony Rios · 2025

Read Does It Run and Is That Enough ? Revisiting Text-to-Chart Generation with a Multi-Agent Approach by James Ford, Anthony Rios (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06175.txt.

Introduction

James Ford and Anthony Rios ask whether making chart code that merely runs is enough, in a 2025 paper. Language models can turn a chart description in words into code that should draw the chart. Yet about fifteen percent of scripts still fail to run, even after fine training and reward learning. The pair ask if the fault lies in the models or in asking one prompt to do it all.

Their light pipeline splits the work among agents: one drafts, one runs, one repairs, and one judges. It uses a standard off-the-shelf model with no extra training. On a known bench, errors fell to under five percent within three repair rounds. That beat the strongest fine-tuned base by a clear margin with far less compute.

Catalog record

CREATORJames Ford, Anthony Rios
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06175

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Does It Run and Is That Enough ? Revisiting Text-to-Chart Generation with a Multi-Agent Approach James Ford, Anthony Rios Abstract Large language models can translate natural-language chart descriptions into runnable code, yet approximately 15% of the generated scripts still fail to execute, even after supervised fine-tuning and reinforcement learning. We investigate whether this persistent error rate stems from model limitations or from reliance on a single-prompt design. To explore this, we propose a lightweight multi-agent pipeline that separates drafting, execution, repair, and judgment, using only an off-the-shelf GPT-4o-mini model. On the Text2Chart31 benchmark, our system reduces execution errors to 4.5% within three repair iterations, outperforming the strongest fine-tuned baseline by nearly 5 percentage points while requiring significantly less compute. Similar performance is observed on the ChartX benchmark, with an error rate of 4.6%, demonstrating strong generalization. Under current benchmarks, execution success appears largely solved. However, manual review reveals that 6 out of 100 sampled charts contain hallucinations, and an LLM-based accessibility audit shows that only 33.3% ( Text2Chart31 ) and 7.2% ( ChartX ) of generated charts satisfy basic colorblindness guidelines. These findings suggest that future work should shift focus from execution reliability toward improving chart aesthetics, semantic fidelity, and accessibility. 1 Introduction Natural language chart generation turns a request such as plot quarterly revenue by region into runnable code and a complete figure. A reliable solution would let non-programmers explore data, shorten the loops of professional analysts, and (potentially) create alternative renderings for low-vision users that are more accessible to screen readers. Two conditions decide whether the community can call the task solved: the generated script must execute without error, and the visual output must match the description. One public benchmark has guided much of the recent progress. The dataset Text2Chart31 Pesaran Zadeh et al. 2024 measures exact failure rates across thirty-one plot types along with various code similarity metrics. Zero-shot prompts for large language models (LLMs) initially failed on about forty percent of inputs. Supervised fine-tuning and reinforcement learning reduced that number below fifteen percent and brought pairwise plots close to six percent. Prompt engineering improved results a little more Zhang et al. 2024b ; Koh et al. 2025 ; Hu et al. 2024 ; Li et al. 2024b . Closed-source models such as GPT-4-turbo and Claude3Opus achieved similar levels, yet stubborn errors remained for complex surfaces and volumetric plots. Work on chart captioning and visual question answering Ford et al. 2025 ; Ye et al. 2024 ; Han et al. 2023 ; Kim et al. 2025 ; Wang et al. 2024b shows that even a small execution failure can break an entire downstream pipeline. Instead of one large prompt, agent-based systems use several small prompts that plan, call tools, and verify interim results Ji and Wang 2025 ; Zhao et al. 2024 . Code-repair agents update faulty scripts after runtime errors and often succeed within a few steps Fan et al. 2023 ; de Fitero-Dominguez et al. 2024 ; Bouzenia et al. 2024 ; Wang et al. 2024a . This study asks two research questions. First, whether chart generation, defined in its present benchmark setting, can be improved using a simple agent-based framework. If the answer is yes, the field would need to revisit its evaluation criteria, shift resources away from training larger models for pure execution, and retire the current benchmarks in favour of more difficult scenarios such as noisy data or multi-step analytic sessions. Second, if using traditional metrics and our approach effectively “solves” what objectives should guide new work when execution errors are already rare? For general image generation, could extend chart evaluation work to move beyond a binary “runs or not” view. Structural similarity (SSIM) scores reflect low-level alignment. At the same time, multimodal LLM judges rate perceptual fidelity and semantic match You et al. 2025 ; Wu et al. 2025 ; Yan et al. 2024 ; Goswami et al. 2025 . Furthermore, a focus on aesthetics, readability, and accessibility could produce charts that follow color-contrast guidelines for color-blind users, place legends to avoid overlap, and include alt-text or tactile representations for screen readers and embossing devices. Such qualities matter for journalism, education, and government dashboards, where the chart must inform a wide audience rather than only run without crashing. To answer these questions, we developed a lightweight multi-agent pipeline that uses an off-the-shelf GPT-4o-mini model without any additional training. A Draft agent generates Python/Matplotlib code from natural-language chart descriptions, and a Repair agent iteratively debugs and rewrites the code—up to three times—when execution fails. On both Text2Chart31 and ChartX , the system reduces execution errors by nearly 5 absolute percentage points, outperforming fine-tuned baselines while preserving image quality as measured by SSIM and multimodal LLM judgment. A manual analysis of 100 sampled outputs found that 83% were accurate, with most remaining issues related to minor stylistic differences rather than semantic or data errors. These findings suggest that agentic pipelines are not only more robust than single-shot prompting but also capable of producing high-quality, visually faithful charts. Overall, we make the following contributions. • We introduce a multi-agent pipeline that achieves state-of-the-art execution success on two public benchmarks. Our system reduces the overall error rate of existing models by nearly 5% with no drop in performance in visual quality. • We provide empirical evidence that execution is largely “solved” in current benchmarks and perform a comprehensive analysis to explore the missing gaps in current evaluation that should be targeted in the future. 2 Related Work Chart Generation and Evaluation. Large language models (LLMs) are increasingly used to automate chart generation from natural language, but major challenges persist in quality control and evaluation. Datasets such as ChartQA , PlotQA , ChartLlama , and Charxiv Masry et al. 2022 ; Methani et al. 2020 ; Han et al. 2023 ; Wang et al. 2024b establish benchmarks for chart reasoning, captioning, and data extraction. These corpora focus primarily on chart understanding and question answering. Ford et al. 2025 and Ye et al. 2024 also examine chart captioning and visual QA. To address text-to-chart generation directly, Text2Chart31 and ChartX pair instructions with both code and image outputs, allowing complete evaluation pipelines Pesaran Zadeh et al. 2024 ; Xia et al. 2025 . Still, these datasets remain small and often require human references. Prompt engineering, CoT prompting Podo et al. 2024a ; Li et al. 2024a , and image-based analysis methods (e.g., SSIM) have been proposed to assess generated charts You et al. 2025 ; Wu et al. 2025 . However, hallucinated or stylistically inconsistent outputs remain common Podo et al. 2024b ; Tian et al. 2025 . Moreover, recent work questions the reliability of multimodal LLM judges for chart evaluation Mukhopadhyay et al. 2024 ; Masry et al. 2024 . This motivates the development of task-specific evaluation protocols combining low-level visual similarity (e.g., SSIM) as well as using LLM-as-a-judge Yan et al. 2024 ; Goswami et al. 2025 . Code Generation and Repair. Chart generation with LLMs relies on producing syntactically and semantically valid code. Yet generated scripts often contain runtime errors, prompting the need for automated debugging frameworks. Several works propose feedback-driven or error-trace-guided repairs Fan et al. 2023 ; de Fitero-Dominguez et al. 2024 ;

Read the complete text