Leveraging LLMs for Multi-File DSL Code Generation: An Industrial Case Study Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 9–12 June, 2026; Glasgow, Scotland, United Kingdom CCS: Software and its engineering Domain specific languages

Sivajeet Chand, Kevin Nguyen, Peter Kuntz, Alexander Pretschner · 2026

Read Leveraging LLMs for Multi-File DSL Code Generation: An Industrial Case Study Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 9–12 June, 2026; Glasgow, Scotland, United Kingdom CCS: Software and its engineering Domain specific languages by Sivajeet Chand, Kevin Nguyen, Peter Kuntz, Alexander Pretschner (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24678.txt.

Introduction

Chand, Nguyen, Kuntz and Pretschner report an industrial case study at BMW on using language models for enterprise domain-specific languages. They adapt code models to generate and modify multi-file DSL artifacts from a single natural-language instruction, encoding folder hierarchies as path-preserving JSON. They test two instruction-tuned code models under baseline prompting, one-shot learning and parameter-efficient fine-tuning. Fine-tuning gives the largest gains, with high exact-match accuracy and strong edit similarity.

Catalog record

CREATORSivajeet Chand, Kevin Nguyen, Peter Kuntz, Alexander Pretschner
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44687
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.24678

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging LLMs for Multi-File DSL Code Generation: An Industrial Case Study Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 9–12 June, 2026; Glasgow, Scotland, United Kingdom CCS: Software and its engineering Domain specific languages Sivajeet Chand, Kevin Nguyen, Peter Kuntz, Alexander Pretschner Abstract . Large language models (LLMs) perform strongly on general-purpose code generation, yet their applicability to enterprise domain-specific languages (DSLs) remains underexplored, especially for repository-scale change generation spanning multiple files and folder structures from a single natural-language (NL) instruction. We report an industrial case study at BMW that adapts code-oriented LLMs to generate and modify project-root DSL artifacts for an Xtext-based DSL that drives downstream Java/TypeScript code generation. We develop an end-to-end pipeline for dataset construction, multi-file task representation, model adaptation, and evaluation. We encode DSL folder hierarchies as structured, path-preserving JSON, allowing single-response generation at repository scale and learning cross-file dependencies. We evaluate two instruction-tuned code LLMs (Qwen2.5-Coder and DeepSeek-Coder, 7B) under three configurations: baseline prompting, one-shot in-context learning, and parameter-efficient fine-tuning (QLoRA). Beyond standard similarity metrics, we introduce task-specific measures that assess edit correctness and repository structural fidelity. Fine-tuning yields the most significant gains across models and metrics, achieving high exact-match accuracy, substantial edit similarity, and structural fidelity of 1.00 on our held-out set for multi-file outputs. At the same time, one-shot in-context learning provides smaller but consistent improvements over baseline prompting. We further validate practical utility via an expert developer survey and an execution-based check using the existing code generator. 1. Introduction Transformer-based LLMs have become effective assistants for code-related tasks in general-purpose languages (GPLs) ( 37 ; 13 ; 29 ) , but there is still limited empirical evidence on their effectiveness for enterprise DSLs ( 16 ) , particularly for repository-scale generation involving multiple files and folder structures from a single NL instruction. In industrial settings, DSLs are frequently used as the authoritative source from which generators produce downstream artifacts (e.g., Java/TypeScript), meaning that small errors in the DSL can break compilation or generation and directly impact engineering workflows. This paper reports an industrial case study with BMW Financial Services, where a custom Xtext-based DSL ( 14 ) is used to generate boilerplate and configuration artifacts across multiple regional markets. The DSL serves as an abstraction layer that accelerates development, but authoring and evolving the DSL remains manual and requires substantial expertise—particularly when creating or adapting market configurations that involve coordinated changes across multiple folders and files. Our goal is to let developers (and potentially non-technical stakeholders) specify high-level changes in NL and automatically obtain a consistent, project-root set of DSL artifacts consumable by the existing code generator. Figure 1. Overview of current workflow vs the proposed workflow A key challenge is repository-scale generation: one instruction can require coordinated edits across multiple DSL files with cross-file dependencies (e.g., adding an attribute in one layer implies updates in related layers). To fit within context-window limits and enable objective evaluation, we linearize the entire DSL folder hierarchy into a single JSON document, using file paths as keys and file contents as values. This lets the model operate at project-root level in one generation step and supports systematic comparison of original context, target state, and model output. Figure 1 illustrates the industrial workflow and how the proposed approach changes it: in the current (“as-is”) process, developers manually create/modify DSL files file-by-file; in the proposed (“to-be”) process, a NL instruction is translated into a complete, consistent set of DSL files, which is then passed to the existing DSL generator to produce downstream artifacts. The critical success criterion is therefore not only textual similarity, but that the generated DSL artifacts preserve repository structure and satisfy the DSL toolchain requirements. To investigate feasibility and cost–benefit trade-offs of adapting LLMs, we evaluate two instruction-tuned LLMs (Qwen2.5-Coder and DeepSeek-Coder, 7B) under three configurations: (i) baseline zero-shot prompting, (ii) one-shot in-context learning (ICL) with DSL rules and an example ( 3 ) , and (iii) parameter-efficient fine-tuning using QLoRA ( 18 ) . Because generic similarity metrics alone are insufficient for repository-scale DSL changes, we combine standard measures (e.g., exact match, BLEU, JSON validity) with task-specific metrics that focus on edit correctness and structural fidelity of the generated folder/file tree. We further validate practical utility through execution-oriented checks using an existing DSL generator and an expert review by senior developers familiar with the DSL. This paper makes the following contributions : • An industrial BMW case study on repository-scale NL-driven generation and modification of Xtext-based DSL artifacts. • A structured linearization that encodes multi-file DSL repositories into a single input/output for project-root generation and consistent cross-file updates. • A task-specific evaluation framework that complements standard metrics with change-focused similarity and structural fidelity measures for multi-file DSL changes. • An empirical comparison of zero-shot, one-shot ICL, and parameter-efficient fine-tuning (QLoRA) on this industrial DSL task, including execution- and expert-based validation. We evaluate these contributions via three research questions: RQ1: How accurately can LLMs generate repository-scale DSL change-sets (files, folders, and contents) from a natural-language instruction? This research question establishes the feasibility of applying LLMs to the proposed task. RQ2: To what extent do prompting and fine-tuning improve LLM performance on this task? This question assesses the incremental gains of adaptation techniques over the baseline, weighing prompt engineering and fine-tuning effort against improvements across validity, structural correctness, and semantic similarity. RQ3: How useful are the generated DSL change-sets to developers who maintain these DSL repositories in practice? This question examines whether LLM outputs provide workflow value by reducing authoring time and correction effort while maintaining acceptable quality. 2. Background DSLs raise the level of abstraction by providing constructs tailored to a particular domain, enabling more concise specifications than GPLs and supporting automation via generators ( 41 ; 38 ; 26 ; 15 ) . In model-driven settings, DSL artifacts are commonly transformed into GPL code, which reduces boilerplate editing effort and can improve consistency and maintainability. Listing 1: Anonymized DSL snippet for server-side configuration ⬇ entity abstract FinanceProductBase extends FinanceProduct { attribute1 : AttributeType16 attribute2 : AttributeType17 attribute3 : AttributeType8 attribute4 : AttributeType9 attribute5 : AttributeType9 attribute6 : AttributeType6 } entity ProductTypeA extends FinanceProductBase { attribute7 : AttributeType18 attribute8 = FinanceProductTypeModule :: type_a attribute9 = CategoryType :: loan } As shown in Figure 1 , in our industrial setting, the DSL is implemented with Eclipse Xtext ( 14 ; 48 ) . Xtext grammars define the DSL syntax and allow generation of parsers and abstract syntax representations, while a custom generator translates DSL files into Java and TypeScript artifacts for different subsystems. This workflow is effective, but it still relies on developers to manually author and update many DSL files, often across nested project structures. Our work targets automating this root-folder / multi-file DSL authoring step from NL instructions while maintaining generator compatibility. Listing 2: Anonymized Java Code Snippet for Server Configuration ⬇ @RegionSpecific ( Region . MARKET ) public abstract class FinanceProductBase extends FinanceProduct { public static final String PROPERTY_ATTRIBUTE_1 = "attribute1" ; public static final String PROPERTY_ATTRIBUTE_2 = "attribute2" ; public static final String PROPERTY_ATTRIBUTE_3 = "attribute3" ; public static final String PROPERTY_ATTRIBUTE_4 = "attribute4" ; public static final String PROPERTY_ATTRIBUTE_5 = "attribute5" ; public static final String

Read the complete text