Accurate and Consistent Graph Model Generation from Text with Large Language Models

Boqi Chen 1 2 , Ou Wei 3 , Bingzhou Zheng 3 , Gunter Mussbacher 2 · 2025

Read Accurate and Consistent Graph Model Generation from Text with Large Language Models by Boqi Chen 1 2 , Ou Wei 3 , Bingzhou Zheng 3 , Gunter Mussbacher 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00255.txt.

Introduction

Boqi Chen and three co-authors improve graph model generation from text with large language models. Generating models from natural language serves many software engineering tasks. Language model output, though, often violates metamodel syntax, breaks domain constraints, or inserts hallucinated elements, and only the first flaw has standard fixes.

An abstraction concretization framework borrows from self consistency methods. Multiple candidate outputs are aggregated into a probabilistic partial model. That partial model is then refined into the concrete model that best satisfies all constraints. Evaluation spans popular open and closed source models on diverse datasets. The framework raises both accuracy and consistency of the generated graph models.

Catalog record

CREATORBoqi Chen 1 2 , Ou Wei 3 , Bingzhou Zheng 3 , Gunter Mussbacher 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-00255

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Accurate and Consistent Graph Model Generation from Text with Large Language Models Thanks: 1 Work partially done during an internship at Huawei Research Canada. Thanks: We thank Dániel Varró, Zohreh Aghababaeyan, Khaled Ahmed, Ru Ji, and anonymous reviewers for their valuable suggestions and feedback on the paper. Boqi Chen 1 2 , Ou Wei 3 , Bingzhou Zheng 3 , Gunter Mussbacher 2 Abstract Graph model generation from natural language description is an important task with many applications in software engineering. With the rise of large language models (LLMs), there is a growing interest in using LLMs for graph model generation. Nevertheless, LLM-based graph model generation typically produces partially correct models that suffer from three main issues: (1) syntax violations : the generated model may not adhere to the syntax defined by its metamodel, (2) constraint inconsistencies : the structure of the model might not conform to some domain-specific constraints, and (3) inaccuracy : due to the inherent uncertainty in LLMs, the models can include inaccurate, hallucinated elements. While the first issue is often addressed through techniques such as constraint decoding or filtering, the latter two remain largely unaddressed. Motivated by recent self-consistency approaches in LLMs, we propose a novel abstraction-concretization framework that enhances the consistency and quality of generated graph models by considering multiple outputs from an LLM. Our approach first constructs a probabilistic partial model that aggregates all candidate outputs and then refines this partial model into the most appropriate concrete model that satisfies all constraints. We evaluate our framework on several popular open-source and closed-source LLMs using diverse datasets for model generation tasks. The results demonstrate that our approach significantly improves both the consistency and quality of the generated graph models. I Introduction Context. Large language models (LLMs), such as GPT-4 [ 1 ] , Llama3 [ 2 ] , and DeepSeek-R1 [ 3 ] , have demonstrated impressive performance on various natural language tasks. Consequently, their application in model-driven engineering (MDE) has attracted significant interest [ 4 , 5 ] . A common use case in MDE involves generating models from natural language descriptions , for example, from requirements or use case narratives to domain models [ 6 , 7 ] , goal models [ 8 ] , sequence diagrams [ 9 ] , or taxonomies [ 10 ] . However, current approaches rely exclusively on the LLM to generate or iteratively refine outputs, limiting their effectiveness. Traditional model generation aims to produce models that are consistent with predefined well-formedness constraints [ 11 ] . However, they do not account for natural language descriptions that allow domain experts to further specify the instance model. In contrast, LLM-based generation often prioritizes generating models that accurately reflect such textual description, ignoring any constraints. Although syntactical correctness is typically addressed in both cases, ensuring both accuracy and consistency is critical for effectively utilizing generated models in subsequent model-driven automation processes, such as code generation [ 12 ] and model-based testing [ 13 ] . Therefore, jointly considering both the accuracy and consistency of models generated by LLMs is essential. Problem statement. Due to the intrinsic limitations of LLMs, the models they generate are typically partially correct and may exhibit issues in all three aspects: (1) syntax : the textual representation of the model may violate the syntax specified by the metamodel; (2) consistency : the structure of the graph model may conflict with domain-specific well-formedness constraints [ 11 ] ; and (3) quality : the generated graph models may be inaccurate with respect to the given descriptions due to hallucinations from LLMs [ 6 , 8 ] . Taking UML activity diagrams as an example, the first issue occurs when generated models violate modeling language syntax (e.g., PlantUML [ 14 ] ). Models affected by the second issue may contain errors like a decision node with only one outgoing edge, which is not valid in activity diagrams. Finally, the third issue occurs when an activity diagram includes inaccurate information compared to the description such as incorrect ordering of activities. The first issue can typically be mitigated by incorporating a filtering mechanism for the outputs or by applying a constraint decoding method to enforce syntax rules [ 15 , 16 ] . However, approaches to address the second and third issues have not yet been thoroughly investigated. Self-consistency leverages the inherent uncertainty of LLMs by synthesizing a more accurate result from multiple outputs generated with the same input [ 17 ] . This approach improves the performance of LLMs on a variety of tasks, particularly when combined with the chain-of-thought (CoT) approach [ 18 ] . With foundational CoT-based reasoning LLMs, such as OpenAI o1 [ 19 ] and DeepSeek R1 [ 3 ] , the adoption of self-consistency is expected to become more common across various applications. The core hypothesis of self-consistency is that if an LLM is capable of solving a task, it should produce the correct answer more frequently than incorrect alternatives [ 17 ] . However, this assumption does not necessarily hold for model generation. Since graph models are composite structures, even if an LLM has the potential to solve a task, it may only predominantly output partially correct graphs . Objective. Our paper aims to simultaneously address the consistency and quality issues when applying LLMs for graph model generation. Motivated by LLM self-consistency, we combine multiple LLM-generated models given a problem description, a metamodel, and a set of well-formedness constraints, such that the final graph adheres to the metamodel and constraints while accurately reflecting the input description. We assume that if LLMs are capable of solving a model generation task, they will produce the correct graph elements more frequently. By aggregating multiple outputs and applying constraints on the graph structure, our approach derives improved solutions. More generally, we aim to demonstrate that leveraging the intrinsic uncertainty of LLMs by combining their generative capability with formal constraints in modeling techniques can enhance both the consistency (issue 2) and quality (issue 3) of the generated models. Contribution. We present AbsCon : an Abs traction- Con cretization framework motivated by self-consistency to derive an accurate and consistent graph from multiple generation results based on partial modeling. Multiple LLM-generated graphs are first abstracted into a probabilistic partial model and then concretized into a final model taking the metamodel and constraints into consideration. Specifically, this paper makes the following contributions: 1. We propose AbsCon : a general framework to create an accurate and consistent model from a pool of candidates. 2. We propose a specific abstraction and concretization method that guarantees the consistency of generated models while improving their quality. 3. We apply AbsCon to three different model generation tasks at different levels of complexity and domains. 4. Through systematic evaluation, we demonstrate the effectiveness of AbsCon in generating consistent models while also improving model quality. Moreover, we explore the influence of the number of candidates on the final model and provide insights on further improvements. Added value. The AbsCon approach guarantees consistency in the generated models, which in turn enhances the overall quality of the outputs. Since it treats LLMs as a black box and requires no additional training or modification to their structure, AbsCon can be easily applied

Read the complete text