Reliability of Large Language Models for Design Synthesis: An Empirical Study of Variance, Prompt Sensitivity, and Method Scaffolding

Rabia Iftikhar, Andreas Rausch · 2026

Read Reliability of Large Language Models for Design Synthesis: An Empirical Study of Variance, Prompt Sensitivity, and Method Scaffolding by Rabia Iftikhar, Andreas Rausch (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00851.txt.

Introduction

Iftikhar and Rausch test how reliably large language models build UML class diagrams from natural language, moving beyond correct syntax to design quality. Their prompting, which favors preferred designs with a few examples, steers outputs toward object-oriented principles and pattern-consistent structures across 540 experiments on three models. Preference alignment improves fit to design intent but does not remove run-to-run variation, and model-level behavior strongly shapes reliability.

Catalog record

CREATORRabia Iftikhar, Andreas Rausch
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38036
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.00851

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reliability of Large Language Models for Design Synthesis: An Empirical Study of Variance, Prompt Sensitivity, and Method Scaffolding Rabia Iftikhar, Andreas Rausch Abstract Large Language Models (LLMs) are increasingly applied to automate software engineering tasks, including the generation of UML class diagrams from natural language descriptions. While prior work demonstrates that LLMs can produce syntactically valid diagrams, syntactic correctness alone does not guarantee meaningful design. This study investigates whether LLMs can move beyond diagram translation to perform design synthesis, and how reliably they maintain design-oriented reasoning under variation. We introduce a preference-based few-shot prompting approach that biases LLM outputs toward designs satisfying object-oriented principles and pattern-consistent structures. Two design-intent benchmarks, each with three domain-only, paraphrased prompts and 10 repeated runs, are used to evaluate three LLMs (ChatGPT 4o-mini, Claude 3.5 Sonnet, Gemini 2.5 Flash) across three modeling strategies: standard prompting, rule-injection prompting, and preference-based prompting, totaling 540 experiments (i.e. 2x3x10x3x3). Results indicate that while preference-based alignment improves adherence to design intent it does not eliminate non-determinism, and model-level behavior strongly influences design reliability. These findings highlight that achieving dependable LLM-assisted software design requires not only effective prompting but also careful consideration of model behavior and robustness. 1 1 1 For the reproducibility of the experiments conducted in this research, the data is available at [ 1 ] I Introduction Large Language Models (LLMs) are increasingly used to automate software engineering tasks [ 2 , 3 , 4 , 5 ] , including the generation of design artifacts such as UML class diagrams [ 6 , 7 ] . Recent tools and studies show that LLMs can produce syntactically valid diagrams from natural language descriptions, however, syntactic validity alone does not constitute good software design that ensures maintainability, extensibility, and architectural soundness. In practice, many LLM-generated diagrams remain structurally shallow: they mirror surface-level entities mentioned in text but fail to capture the design intent: the design principles (i.e. abstraction, encapsulation, separation of concern), and design knowledge (i.e. GoF design patterns) the experienced designers apply to promote extensibility and reusability [ 8 , 6 , 7 ] . When this knowledge is absent, the modeling process risks becoming "garbage in, garbage out,” where small changes or omissions in the input lead to inconsistent diagrams that cannot be trusted for downstream implementation [ 4 , 9 , 10 ] . This limitation highlights a fundamental distinction between diagram translation and design synthesis. Many current AI-based modeling approaches behave primarily as translators, mapping textual nouns and verbs to classes, attributes, and associations. Yet high-quality architectural modeling requires more than element extraction—it requires inferring implicit design patterns and principles from requirements. Importantly, real-world requirements rarely specify which design pattern or principles to apply. Instead, they describe domain rules and invariants (e.g., dynamic user roles or policy-dependent behavior), from which engineers derive appropriate architectural structures. Therefore, an LLM-based modeling assistant should be evaluated on whether it can implicitly infer and apply design principles and knowledge rather than merely reproducing patterns and principles named in prompts [ 11 , 12 ] . Beyond design quality, an equally critical but underexplored challenge is behavioral reliability. Generative models are inherently non-deterministic, yet architectural workflows demand stability and repeatability. Practitioners must be able to expect structurally consistent designs under realistic variations such as prompt paraphrasing or repeated executions. If outputs fluctuate significantly, their practical utility diminishes regardless of average performance. Despite growing adoption of LLM-assisted development, limited empirical work has systematically examined how reliably frontier models sustain design-level reasoning [ 13 ] . Existing studies largely emphasize capability while overlooking behavioral stability across runs, sensitivity to prompt variation, and differences across model families [ 12 , 13 , 14 ] . For architecture-centric tasks, where inconsistencies may propagate into implementation defects, this omission is particularly consequential. This leads to the following research questions in this study. RQ1: To what extent can LLMs generate design-level UML class diagrams that satisfy domain constraints and design principles when provided with domain-only descriptions? RQ2: To what extent do LLM-generated diagrams exhibit pattern-consistent structures implicitly, and how does the proposed approach affect the emergence of such structures compared to standard prompting and rule-injection baselines? RQ3: How robust are the generated designs across repeated runs and prompt paraphrases, and what behavioral stability differences emerge across models? RQ4: How does increasing domain complexity affect adherence to design principles, pattern emergence, and behavioral stability? This paper investigates not only whether LLM behavior can be guided from translation toward design synthesis, but also how reliably different models maintain design-oriented reasoning under controlled sources of variation. We propose a preference-based few-shot prompting approach that exposes models to contrasting modeling solutions and biases generation toward designs that satisfy object-oriented principles and pattern-consistent structures 2 2 2 A preliminary version of the preference-based alignment idea was previously presented as an abstract-only contribution at a workshop venue [ 15 ] ; the present paper substantially extends that early concept by providing a concrete implementation, a controlled experimental protocol, and a comprehensive empirical evaluation across multiple models, scenarios, and sources of variation. . To avoid trivializing the task we construct two varied domain, design-intent benchmarks in which textual problem statements describe domain behavior and constraints without naming principles and patterns. For each design task, we compare three LLMs (i.e. ChatGPT 4o-mini, Claude 3.5 Sonnet, Gemini 2.5 Flash), on three paraphrased prompts, using three different modeling approaches (i) standard prompting baselines, (ii) a rule-injection baseline that explicitly encodes design principles, and (iii) preference-based prompting, for 10 repeated runs (i.e. 2x3x3x3x10=540 runs). From our results a clear pattern emerges: performance differences cannot be attributed solely to prompting strategy when applied to design synthesis tasks. Instead, model-level behavioral characteristics strongly influence reliability in design synthesis tasks. While the proposed preference-based alignment improves the likelihood of generating models that reflect architectural intent, we observe distinct stability regimes across the evaluated models—Claude exhibiting high decoding stability across repeated runs, ChatGPT demonstrating predictable behavior once design scaffolding is defined, and Gemini showing substantial variability across paraphrases and executions. These findings suggest that model choice may be a dominant factor in achieving dependable software designs, reframing LLM-based modeling as not only a capability problem but also a reliability challenge. Paper Structure: The remainder of the paper is structured as follows: Section 2 reviews related work. Section 3 outlines the experimental design. Section 4 describes the technical implementation of the prompting strategies used within that protocol. Section 5 reports evaluation results and discusses findings with respect to the research questions. Section 6 outlines study limitations, and section 7 concludes the paper and identifies directions for future research. II Background and Related Work LLMs for Software Design Early studies demonstrated that models like Codex and GPT-3 can generate syntactically correct code from natural language prompts, highlighting their potential as intelligent programming assistants [ 16 ] . Subsequent work extended these capabilities to software design, investigating whether LLMs can produce UML diagrams from textual descriptions [ 17 , 6 , 18 , 7 ] , suggesting that natural language-based design automation may support early-stage modeling activities. Despite these promising capabilities, researchers caution that syntactic correctness does not guarantee design quality. LLM-generated artifacts often reflect surface-level textual correlations rather than underlying architectural reasoning [ 13

Read the complete text