DiagramEval: Evaluating LLM-Generated Diagrams via Graphs
Chumeng Liang, Jiaxuan You · 2025
Read DiagramEval: Evaluating LLM-Generated Diagrams via Graphs by Chumeng Liang, Jiaxuan You (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25761.txt.
Introduction
Chumeng Liang and Jiaxuan You test language model generated diagrams. Their 2025 paper proposes DiagramEval, a graph-based metric. Diagrams convey ideas centrally in research papers. They are notoriously complex and labor-intensive to create. Image models struggle producing clear structured diagrams. Generating diagrams as SVG text is promising direction. Language models can produce such textual diagrams directly.
Discriminative, explainable quality metrics for diagrams are lacking. DiagramEval conceptualizes diagrams as graphs for testing. Text elements become nodes in that graph. Connections become directed edges between those nodes. Two metric groups assess diagram quality. Node matching checks element correspondence between diagrams. Path matching checks connection form preservation. They test diagrams from leading language models. Recent research literature supplied the source papers. Metrics validity is showed quantitatively in tests. Enhanced explainability offers insights into diagram characteristics. Common failure modes become visible.
Catalog record
| CREATOR | Chumeng Liang, Jiaxuan You |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-25761
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DiagramEval: Evaluating LLM-Generated Diagrams via Graphs Chumeng Liang, Jiaxuan You Abstract Diagrams play a central role in research papers for conveying ideas, yet they are often notoriously complex and labor-intensive to create. Although diagrams are presented as images, standard image generative models struggle to produce clear diagrams with well-defined structure. We argue that a promising direction is to generate demonstration diagrams directly in textual form as SVGs, which can leverage recent advances in large language models (LLMs). However, due to the complexity of components and the multimodal nature of diagrams, sufficiently discriminative and explainable metrics for evaluating the quality of LLM-generated diagrams remain lacking. In this paper, we propose DiagramEval, a novel evaluation metric designed to assess demonstration diagrams generated by LLMs. Specifically, DiagramEval conceptualizes diagrams as graphs, treating text elements as nodes and their connections as directed edges, and evaluates diagram quality using two new groups of metrics: node alignment and path alignment. For the first time, we effectively evaluate diagrams produced by state-of-the-art LLMs on recent research literature, quantitatively demonstrating the validity of our metrics. Furthermore, we show how the enhanced explainability of our proposed metrics offers valuable insights into the characteristics of LLM-generated diagrams. Code: https://github.com/ulab-uiuc/diagram-eval . 1 Introduction Diagrams play a central role in research papers for conveying ideas, e.g. , the diagram of Transformer Vaswani et al. 2017 has played a pivotal role in presenting and publicizing the idea, making it one of the most cited deep learning papers. However, generating high-quality diagrams is often notoriously complex and labor-intensive to create. Consequently, automated diagram generation is a central challenge in AI-assisted scientific discovery ( Eger et al. 2025 ) , potentially saving millions of hours for researchers while improving the quality of research publications. Although diagrams are presented as images, standard image generative models struggle to produce clear diagrams with well-defined structure ( Zala et al. 2023 ) . We argue that a promising direction is to generate demonstration diagrams directly in textual form as SVGs, which can leverage recent advances in large language models (LLMs). Existing methods on automated diagram generation with LLMs rely heavily on proprietary LLMs, either through direct planning ( Mondal et al. 2024 ; Zhang et al. 2024 ; Cui et al. 2025 ) or as assistance in diagram generation ( Belouadi et al. 2023 ; Belouadi et al. 2024 ; Cui et al. 2025 ) . Given this reliance, advancements in cutting-edge proprietary LLMs, such as Claude 3.7 Sonnet ( Anthropic 2025 ) and Gemini 2.5 Pro ( Google DeepMind 2025 ) , directly enhance automated diagram generation through SVG code generation capabilities ( Blecher et al. 2023 ) . However, existing evaluation metrics lack the expressiveness needed to differentiate diagrams of varying quality. Most benchmarks employ model-based, diagram-level metrics ( Hessel et al. 2021 ; Fu et al. 2023 ) , which were originally designed for general text-to-image tasks rather than text-within-image generation specific to diagrams ( Rodriguez et al. 2023 ) . These metrics evaluate entire diagrams using general vision models, which limits their explainability and their ability to accurately capture detailed logical correctness, resulting in only moderate correlation with human judgments ( Eger et al. 2025 ) . Thus, there is a critical need for new metrics. Figure 1: DiagramEval framework overview . By considering both research paper context and paper diagrams as directed graphs, we use information precision and recall among nodes and edges of the generated diagram graph and those of the reference graph (from reference diagrams or paper context) to measure the generation quality of paper diagrams. In this paper, we propose DiagramEval, a novel evaluation metric designed to assess demonstration diagrams generated by LLMs. As demonstrated in Figure 1 , DiagramEval conceptualizes diagrams as graphs, treating text elements as nodes and their connections as directed edges, and evaluates diagram quality using two new groups of metrics: node alignment and path alignment. Specifically, we transform diagrams into SVG format, parsing text elements as nodes. Connections between elements are then extracted as edges. Our metrics quantify the node and path (multi-hop edges) alignment between generated and ground-truth diagrams using precision, recall, and F1 scores. For the first time, we effectively evaluate diagrams produced by state-of-the-art LLMs on recent research literature, validating the proper consistency between our metrics and existing metrics. We also demonstrate metric statistics and case studies that reveal some previously unrecognized shortcomings of existing metrics due to their moderate explainability, for example, over-sensitivity to spatial layouts and other visual elements and metric hacking. We furthermore show empirically how the enhanced explainability of our metrics helps overcome these shortcomings and meanwhile offers valuable insights into the characteristics of LLM-generated diagrams. Based on our proposed metrics, we build a benchmark to evaluate the quality of LLM-generated paper diagrams. We implement an automated pipeline to extract context and diagrams from preprint versions of research papers on Arxiv only with their titles. With this highly available data source, we construct a simple agent for diagram generation with Gemini 2.5 Flash Image, the proprietary image generative models from Google. Our metrics are then used to evaluate the quality of generated diagrams. To the best of our knowledge, this is the first benchmark specified for evaluating LLM-generated paper diagrams. Code: https://github.com/ulab-uiuc/diagram-eval . 2 Related Works Extensive research has addressed both the understanding ( Han et al. 2023 ; Liu et al. 2023 ; Wang et al. 2024b ; Hu et al. 2024 ) and generation ( Maddigan and Susnjak 2023 ; Yang et al. 2024 ) of scientific images, with specific efforts targeting diagram generation ( Zala et al. 2023 ; Zhang et al. 2024 ; Mondal et al. 2024 ; Cui et al. 2025 ) . Current automated evaluation metrics fall into two main categories: 1) diagram-to-diagram similarity ( Fu et al. 2023 ) , and 2) caption-to-diagram similarity ( Zhang et al. 2019 ; Hessel et al. 2021 ) . Both metric types primarily measure overall similarity between generated diagrams and references in latent space, neglecting detailed logical accuracy, such as verifying element-wise connections against the paper context. Closely related to our approach is the VPEval metric ( Cho et al. 2023 ) , which evaluates element and connection accuracy via visual question answering (VQA). However, VPEval often struggles to accurately identify connections even in diagrams involving common knowledge, requiring manual annotations ( Zala et al. 2023 ) . Our approach extends the concept of VPEval by leveraging vision-language models (VLMs) and file parsing to reliably extract elements and connections, thus better suited for evaluating diagrams generated from academic papers. 3 Preliminary Problem Definition Following existing research working on diagram generation ( Zala et al. 2023 ; Mondal et al. 2024 ) , we define the task of automated generation of scientific diagrams from academic papers: Given the input corpus \{T,c,\pi\} , which includes original paper context T , original diagram captions c_{orig} , and layout captions c_{layout} , our goal is to generate a diagram \mathcal{D} which demonstrates the overall idea of research paper T . We have a groundtruth diagram \mathcal{D}_{gt} , which we use to generated the layout captions with an independent vision language model. \mathcal{D}_{gt} also serve as reference to evaluate generated