MermaidSeqBench: An Evaluation Benchmark for LLM-to-Mermaid Sequence Diagram Generation
Basel Shbita, Farhan Ahmed, Chad DeLuca · 2025
Read MermaidSeqBench: An Evaluation Benchmark for LLM-to-Mermaid Sequence Diagram Generation by Basel Shbita, Farhan Ahmed, Chad DeLuca (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14967.txt.
Introduction
Basel Shbita, Farhan Ahmed, and Chad DeLuca introduce MermaidSeqBench, a test set for turning plain language into Mermaid sequence diagrams. Software teams use such diagrams to show how parts of a system talk over time, and language models are increasingly asked to draft them. Until now there was no careful benchmark for whether the generated diagrams are actually right.
The set holds a bit over a hundred cases built from human-checked flows, model-based expansion, and rule-based growth. A model judge grades each output on fine-grained traits such as correct actors, message order, and structure. With a shared yardstick, teams can finally compare diagram generation skills across models in a principled way.
Catalog record
| CREATOR | Basel Shbita, Farhan Ahmed, Chad DeLuca |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-14967
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MermaidSeqBench: An Evaluation Benchmark for LLM-to-Mermaid Sequence Diagram Generation Basel Shbita, Farhan Ahmed, Chad DeLuca Abstract Large language models (LLMs) have demonstrated excellent capabilities in generating structured diagrams from natural language descriptions. In particular, they have shown great promise in generating sequence diagrams for software engineering, typically represented in a text-based syntax such as Mermaid. However, systematic evaluations in this space remain underdeveloped as there is a lack of existing benchmarks to assess the LLM’s correctness on this task. To address this shortcoming, we introduce MermaidSeqBench, a human-verified and LLM-synthetically-extended benchmark for assessing an LLM’s capabilities in generating Mermaid sequence diagrams from textual prompts. The benchmark consists of a core set of 132 samples, starting from a small set of manually crafted and verified flows. These were expanded via a hybrid methodology combining human annotation, in-context LLM prompting, and rule-based variation generation. Our benchmark uses an LLM-as-a-judge model to assess Mermaid sequence diagram generation across fine-grained metrics, including syntax correctness, activation handling, error handling, and practical usability. We perform initial evaluations on numerous state-of-the-art LLMs and utilize multiple LLM judge models to demonstrate the effectiveness and flexibility of our benchmark. Our results reveal significant capability gaps across models and evaluation modes. Our proposed benchmark provides a foundation for advancing research in structured diagram generation and for developing more rigorous, fine-grained evaluation methodologies. \workshoptitle Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities in programming tasks such as code generation [ 1 , 2 ] , code documentation [ 3 , 4 ] , and coding assistants [ 5 ] . Furthermore, they show great promise in generating structured diagrams from natural language descriptions [ 6 ] . One type of structured diagram is a sequence diagram: a visual model used in software engineering that shows how objects, components, and processes interact with each other over time. Sequence diagrams are most commonly visualized in Unified Modeling Language (UML) [ 7 ] , but can also be represented in a text-based syntax such as PlantUML [ 8 ] or Mermaid [ 9 ] (see Appendix A for more details). LLMs have proven to be proficient in generating such text-based based sequence diagrams [ 10 , 11 ] . However, systematic evaluations for assessing an LLM’s correctness in producing sequence diagrams remains largely underdeveloped. This has motivated research on developing evaluations for sequence diagram generation such as statistical method [ 12 ] or an LLM-as-a-judge [ 13 ] . Mermaid [ 9 ] is a diagramming tool that uses a Markdown-inspired syntax and is one such text-based method of representing sequence diagrams. Sequence diagrams textually represented in Mermaid are aptly named Mermaid sequence diagrams. Prior work has shown that LLMs are capable of effectively producing Mermaid sequence diagrams [ 14 ] . However, evaluation in this space is even more underexplored as there is a severe lack of existing benchmarks or even public datasets to evaluate an LLM’s Mermaid sequence diagram generation capabilities. This has driven our work to create a reproducible, scalable, and fine-grained evaluation framework for this task. Our key contributions are as follows: • We introduce MermaidSeqBench , a human-verified and LLM-synthetically extended benchmark for assessing an LLM’s capabilities in generating Mermaid sequence diagrams from textual prompts. This benchmarks consists of 132 samples and a systematic evaluation method using an LLM-as-a-judge to assess Mermaid sequence diagram generation across fine-grained metrics such as syntax correctness, activation handling, error handling, and practical usability. • We perform initial evaluations on six models from three families and sizes ( Qwen 2.5 [ 15 ] at 0.5B and 7B, Llama 3.1/3.2 [ 16 ] at 1B and 8B, and Granite 3.3 [ 17 ] at 2B and 8B) on MermaidSeqBench using two large LLM judge models ( DeepSeek-V3 [ 18 ] at 671B and GPT-OSS [ 19 ] at 120B). This demonstrates both the effectiveness and flexibility of our proposed benchmark as our results reveal significant capability gaps across models and evaluation modes. Our benchmark aims to provide a foundation for advancing research, not only in Mermaid sequence diagram generation, but also structured diagram generation as a whole. As a result, we open source MermaidSeqBench 1 1 1 Code available at: https://github.com/IBM/MermaidSeqBench-Eval and make the benchmark dataset publicly accessible 2 2 2 Dataset available at: https://huggingface.co/datasets/ibm-research/MermaidSeqBench in hopes of encouraging further work in developing more rigorous, fine-grained evaluation methodologies in this space. Furthermore, although we focus on LLM-to-Mermaid sequence diagram evaluations, our methodology can be applied to other structural diagrams or even other textual representations such as PlantUML. 2 Related Work Existing work on evaluating LLM generation of Mermaid sequence diagrams remains largely underexplored and very limited. Expanding to other forms of structural diagrams, Saxena et al. 2025 and Guernsey 2025 both explore methods on using LLMs to generate various structural diagrams in Mermaid syntax such as class, flow, and sequence diagrams. However, evaluations remain relatively simple through simple compliance or visualization checks which are not as robust or scalable. In the broader space, most work on using LLMs to generate structural diagrams is typically represented textually in PlantUML [ 8 ] . There are several works on LLM generation of PlantUML class, flow, case, and sequence diagrams [ 6 , 11 , 20 , 21 ] . Evaluations for these tasks provide more robust approaches. Rouabhia and Hadjadj 2025 use statistical methods to validate the syntactic, structural, and behavioral consistency of the LLM generated UML class diagrams in similar way to existing code benchmarks such as HumanEval [ 22 ] or MBPP [ 23 ] . Ferrari et al. 2024 utilize similar statistical methods to evaluate LLM generated UML sequence diagrams on categories such as completeness, correctness, and adherence to standards. Furthermore, Ahmed et al. 2025 explore using an LLM-as-a-judge for evaluating the LLM-to-PlantUML sequence diagram task. To the best our knowledge, no formal benchmark or even public dataset exists for evaluating an LLM’s capabilities on producing Mermaid sequence diagrams. Our work is the first to introduce such a benchmark, public dataset, and systematic evaluation framework to evaluating this task in a similar manner to code generation benchmarks. 3 Methodology Our methodology for constructing MermaidSeqBench proceeds in three stages: (1) initial dataset construction; (2) synthetic expansion using scalable LLM-based generation; and (3) systematic augmentation through rule-based variation. This pipeline enables us to create a benchmark that is both human-verified at its core and systematically extended for scalability and diversity. 3.1 Initial Dataset Construction We begin by constructing a small, high-quality seed set of ten Mermaid sequence diagrams. These were written manually by a subject matter expert (SME) given a set of natural language descriptions of different sequence flows. Each flow was manually verified to confirm syntax, semantic plausibility, and completeness. This curated core serves as the foundation of the benchmark. 3.2 Synthetic Expansion To scale beyond the core set, we employ Scalable Synthetic Data Generation (SDG) [ 24 ] , an open-source general-purpose LLM-wrapper framework designed for producing large quantities of synthetic data. In this stage, we use in-context examples of full flows of valid sequence diagrams in Mermaid syntax to guide the LLM in generating additional diagram flows. This allows us to