Optimizing Retrieval Augmented Generation for Object Constraint Language

Kevin Chenhao Li, Vahid Zolfaghari, Nenad Petrovic, Fengjunjie Pan, Prof. Alois Knoll · 2025

Read Optimizing Retrieval Augmented Generation for Object Constraint Language by Kevin Chenhao Li, Vahid Zolfaghari, Nenad Petrovic, Fengjunjie Pan, Prof. Alois Knoll (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13129.txt.

Introduction

Kevin Chenhao Li and four co-authors tune retrieval-augmented generation for writing rules in a formal constraint language. The Object Constraint Language states precise conditions inside model-based engineering projects. Writing those rules by hand is slow and tricky, so automation has real value.

Their 2025 study compares three ways of fetching helpful examples for a language model. A word-matching method called BM25 actually did worse than no retrieval. Two meaning-based methods, BERT retrieval and SPLADE, did better, with SPLADE strongest when few passages were fetched. Fetching too many passages hurt, as off-topic text crept in. The team also benchmarked against PathOCL, a leading graph-based tool, in one shared test frame.

Catalog record

CREATORKevin Chenhao Li, Vahid Zolfaghari, Nenad Petrovic, Fengjunjie Pan, Prof. Alois Knoll
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13129

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimizing Retrieval Augmented Generation for Object Constraint Language Kevin Chenhao Li, Vahid Zolfaghari, Nenad Petrovic, Fengjunjie Pan, Prof. Alois Knoll Abstract The Object Constraint Language (OCL) is essential for defining precise constraints within Model-Based Systems Engineering (MBSE). However, manually writing OCL rules is complex and time-consuming. This study explores the optimization of Retrieval-Augmented Generation (RAG) for automating OCL rule generation, focusing on the impact of different retrieval strategies. We evaluate three retrieval approaches—BM25 (lexical-based), BERT-based (semantic retrieval), and SPLADE (sparse-vector retrieval)—analyzing their effectiveness in providing relevant context for a large language model. To further assess our approach, we compare and benchmark our retrieval-optimized generation results against PathOCL, a state-of-the-art graph-based method. We directly compare BM25, BERT, and SPLADE retrieval methods with PathOCL to understand how different retrieval methods perform for a unified evaluation framework. Our experimental results, focusing on retrieval-augmented generation, indicate that while retrieval can enhance generation accuracy, its effectiveness depends on the retrieval method and the number of retrieved chunks (k). BM25 underperforms the baseline, whereas semantic approaches (BERT and SPLADE) achieve better results, with SPLADE performing best at lower k values. However, excessive retrieval with high k parameter can lead to retrieving irrelevant chunks which degrades model performance. Our findings highlight the importance of optimizing retrieval configurations to balance context relevance and output consistency. This research provides insights into improving OCL rule generation using RAG and underscores the need for tailoring retrieval. [[email protected], ] *1 [[email protected], ] [[email protected], ] [[email protected], ] [[email protected], ] keywords Retrieval-Augmented Generation ,Object Constraint Language ,Model-Based Systems Engineering ,Large Language Models ,Information Retrieval † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: First Workshop on Large Language Models For Generative Software Engineering (LLM4SE 2025) † † address: Technical University of Munich (TUM), Arcisstraße 21 D-80333 Munich, Germany † † corresp: Corresponding author. 1 Introduction Object Constraint Language Object Management Group 2014 plays an important role in Model-Based Systems Engineering (MBSE) by enabling precise constraint definition within meta-models. OCL is used to ensure the integrity of system designs by specifying conditions that must be held within a model. It is widely used in Unified Modeling Language (UML) Pilone and Pitman 2005 and Eclipse Modeling Framework (EMF) Steinberg et al. 2008 by defining invariants, preconditions, postconditions, and derived attributes, which cannot be specified by the model itself and thereby enhancing model expressiveness. However, manually writing OCL rules is complex and time-consuming, requiring a deep understanding of both the system model and OCL syntax. Natural language is often the starting point for defining system constraints, such as those given in the requirements and specifications of the system. This makes an automated approach that translates natural language specifications into OCL rules highly attractive and could significantly improve efficiency and accessibility. Recent advances in Large Language Models (LLMs) have revolutionized automated code and rule generation. Models such as GPT-4 OpenAI et al. 2024 , DeepSeek DeepSeek-AI et al. 2025 , and Meta-Llama-3 Grattafiori et al. 2024 have demonstrated remarkable capabilities in understanding and generating structured text, including programming languages and domain-specific rule sets. These models leverage pre-trained knowledge from large text corpora, enabling them to generalize across different programming languages, formal notations, and syntactic structures. In particular, LLMs have been thoroughly applied in natural language-to-code translation Guo et al. 2024 Ni et al. 2023 , showing strong performance in converting natural language instructions into executable code in languages such as Python, Java, and SQL. Similarly, they can be adapted to translate natural language specifications into OCL rules Pan et al. 2024 Abukhalaf et al. 2023 , reducing the manual effort required by human engineers. However, LLMs often struggle with domain-specific knowledge Gu et al. 2025 , especially when dealing with complex specifications and extensive meta-models. A meta-model defines the structure and rules for how models are built in MBSE. It includes elements such as classes, associations, enumerations, and attributes. This is where RAG becomes essential. Large Language Models have a limited context window, making it challenging to include large and complex meta-models entirely in a prompt. RAG allows relevant parts of the meta-model to be retrieved and injected dynamically, helping the LLM generate precise rules even when the whole meta-model does not fit into the context window. Figure 1: Example meta-model from Pan et al. 2024 Retrieval-Augmented Generation is an approach that enhances LLMs by integrating external knowledge retrieval into the generation process Zhao et al. 2024 Gao et al. 2024 . This idea originated from the question answering domain Sharma et al. 2024 . Instead of relying solely on a model’s pre-trained knowledge, RAG retrieves relevant information from an external knowledge base and incorporates it into the model’s input before generating the final output. This method has shown promise in improving accuracy, reducing hallucinations Li et al. 2024 , and ensuring that generated content aligns with the domain Gao et al. 2024 . Since OCL rules are tightly coupled with the underlying meta-model structure, a standard LLM may not have sufficient context to generate the correct rule. RAG allows us to retrieve relevant meta-model elements (e.g., classes, associations, enumerations) from a retrievable knowledge base and include them in the input prompt. The desired result is that RAG helps the LLM generate rules that adhere to proper OCL syntax and semantics. However, optimizing retrieval strategies for OCL generation has not been extensively studied, particularly in the context of balancing retrieval efficiency and generation accuracy. While fine-tuning can adapt LLMs to specific tasks, such as OCL rule generation, it is resource-intensive and may not generalize well across new or unseen meta-models. Recent research has increasingly highlighted the advantages of Retrieval-Augmented Generation over fine-tuning for knowledge injection in large language models. RAG consistently outperforms fine-tuning across multiple datasets, even when dealing with previously known and entirely new knowledge. Fine-tuning struggles with learning new factual information and requires extensive training data Ovadia et al. 2024 . Similarly, another study Soudani et al. 2024 found that RAG is more effective in handling less popular or low-frequency knowledge, including domain-specific knowledge. The study emphasizes that while smaller language models may still benefit from fine-tuning, larger models gain little additional advantage. Additionally, fine-tuning remains resource-intensive. Combining RAG with fine-tuning can lead to further improvements to performance in specialized domains as shown in Balaguer et al. 2024 . Current research in OCL rule generation has focused on other approaches, such as fine-tuning Pan et al. 2024 . This study uses RAG but does not evaluate the impact of it or experiment with different retrieval configurations. Other current work about OCL rule generation using LLMs include Abukhalaf et al. 2023 and Abukhalaf et al. 2024 , which either fully inject the meta-model without retrieval or use a path-based approach. However, the impact of retrieval-based methods such as RAG on OCL rule generation remains underexplored. This gap in the literature highlights the necessity of further investigating retrieval-based strategies to enhance the accuracy and efficiency of OCL rule generation. By evaluating different retrieval configurations and assessing their impact, this study aims to contribute a novel perspective to the field. This study explores the optimization of RAG for OCL

Read the complete text