Synthline: A Product Line Approach for Synthetic Requirements Engineering Data Generation using Large Language Models

Abdelkarim El-Hajjami, Camille Salinesi · 2025

Read Synthline: A Product Line Approach for Synthetic Requirements Engineering Data Generation using Large Language Models by Abdelkarim El-Hajjami, Camille Salinesi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03265.txt.

Introduction

Abdelkarim El-Hajjami and Camille Salinesi introduced Synthline in 2025, a method for making synthetic training data for requirements engineering. This field studies how software needs are written and checked, and it relies on language and learning tools. Those tools are held back by a lack of good data sets.

Synthline treats data generation like a product line, using large language models to produce varied data in a systematic way. The team tested it on the task of spotting defects in requirements documents. Synthetic data alone was less varied than real data but good enough to train on. Mixing synthetic with real data raised precision a lot and doubled recall. Code and data are shared openly.

Catalog record

CREATORAbdelkarim El-Hajjami, Camille Salinesi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03265

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Synthline: A Product Line Approach for Synthetic Requirements Engineering Data Generation using Large Language Models Abdelkarim El-Hajjami, Camille Salinesi Abstract While modern Requirements Engineering (RE) heavily relies on natural language processing and Machine Learning (ML) techniques, their effectiveness is limited by the scarcity of high-quality datasets. This paper introduces Synthline, a Product Line (PL) approach that leverages Large Language Models to systematically generate synthetic RE data for classification-based use cases. Through an empirical evaluation conducted in the context of using ML for the identification of requirements specification defects, we investigated both the diversity of the generated data and its utility for training downstream models. Our analysis reveals that while synthetic datasets exhibit less diversity than real data, they are good enough to serve as viable training resources. Moreover, our evaluation shows that combining synthetic and real data leads to substantial performance improvements. Specifically, hybrid approaches achieve up to 85% improvement in precision and a 2× increase in recall compared to models trained exclusively on real data. These findings demonstrate the potential of PL-based synthetic data generation to address data scarcity in RE. We make both our implementation and generated datasets publicly available to support reproducibility and advancement in the field. Product Line 1 Introduction Requirements Engineering (RE) largely operates through Natural Language (NL), which is the primary medium for capturing, specifying, and validating all types of requirements [ 1 ] . Although this reliance on NL has enabled significant advancements in the use of Natural Language Processing (NLP) techniques and Artificial Intelligence (AI) models in RE [ 9 , 10 ] , their effectiveness remains tied to the availability of diverse high-quality datasets. Even with the paradigm shift introduced by in-context learning in Large Language Models (LLMs) [ 7 ] , achieving optimal performance still requires sufficient high-quality training data and additional fine-tuning [ 8 ] . Therefore, the shortage of freely available RE-specific datasets remains a major bottleneck [ 9 ] , [ 17 ] , slowing progress and limiting the effective use of AI models in RE. However, the acquisition of high-quality data presents significant challenges. It is both costly and time-consuming, with the data annotation phase often being laborious and error-prone [ 12 ] . In addition, companies are often reluctant to share sensitive and private data, restricting the availability and use of such data for research purposes outside the organizations that produce it. Consequently, synthetic data emerges as a promising solution to address these issues. Data synthesis may take several forms: anonymization of sensitive data [ 16 ] ; annotation for supervised learning [ 12 ] ; augmentation through text transformations (e.g., word deletion or synonym replacement) [ 4 , 6 ] ; and the generation of entirely new samples from scratch [ 19 ] . Our research follows this last approach, leveraging LLMs’ advanced text generation and multilingual capabilities [ 2 ] to address data scarcity in RE. This approach not only enables cost-effective, large-scale data generation but also provides systematic control over data properties, supporting varied RE use cases and addressing specific challenges such as data imbalance [ 15 ] and the scarcity of non-English resources [ 9 ] . To enable systematic controllability, we propose a generative approach that formalizes the variability of RE use cases and their specific requirements using a Feature Model (FM). This study represents the first iteration of a broader design science cycle [ 3 ] , where we design and evaluate a minimal Product Line (PL) of synthetic datasets designed for classification-based RE use cases. Within this scope, we examine the following research questions: • RQ1. How does the diversity of the synthetic data compare to that of real data? • RQ2. How does training on synthetic data affect model performance relative to training exclusively on real data? Contributions. \raisebox{-0.8pt}{1}⃝ We propose Synthline , a minimal and configurable approach for systematically controlling LLM-based data generation. \raisebox{-0.8pt}{2}⃝ We evaluate our approach on the use case of identifying requirements specification defects. The evaluation was conducted along two dimensions: data diversity and downstream model performance. \raisebox{-0.8pt}{3}⃝ We provide both the generated datasets and our implementation code as open-access resources to support broader adoption and reproducibility in the RE community 1 1 1 https://github.com/abdelkarim-elhajjami/synthline/tree/v0.0.0 . The paper is organized as follows: Section 2 reviews the related work relevant to our study. Section 3 introduces the methodology, detailing our PL-based approach and its implementation through Synthline. Section 4 presents our evaluation approach for assessing data diversity and utility. Section 5 discusses the results of our experiments. Section 6 examines threats to validity. Finally, Section 7 concludes the paper and outlines future research directions. 2 Related Work LLMs have motivated numerous studies to explore different approaches for synthetic data generation. The first methods relied on simple context-informed prompting, either in a zero-shot setting using only task specification [ 13 ] or in a few-shot setting, guided by a small number of examples [ 11 ] . Recent works have explored attributed prompting, incorporating explicit attributes to control generation and address issues of low informativeness and redundancy [ 19 ] . Furthermore, multi-step generation approaches decompose the workflow into sequential steps, generating multiple data series to ensure greater diversity [ 18 ] . These studies underscore the importance of controlled generation for both data diversity and quality. In the RE domain, synthetic data research has primarily focused on data augmentation. Malik et al. [ 4 ] use techniques like back translation, paraphrasing, and RE-specific substitutions (e.g., Noun-Verb or Actor-Action) to improve conflict and duplicate detection in imbalanced datasets. Similarly, Majidzadeh et al. [ 5 ] introduce code-based augmentation methods such as variable renaming and operand swapping, to enhance requirements traceability, yielding improved precision, recall, and F1-score. To the best of our knowledge, Cheng et al. [ 6 ] are the only researchers to employ LLMs (GPT-J) for RE data augmentation; they demonstrated performance gains in classification across three industry case studies. However, these augmentation approaches have several key limitations. First, they rely on existing datasets, making them ineffective in cases of severe data scarcity. Second, their transformations often produce data that remains close to the original domain and distribution, restricting the potential for genuinely novel data. Third, they typically offer limited control over data attributes such as linguistic, or domain variability. By contrast, our research focuses on generating entirely new synthetic datasets with LLMs, guided by an FM to offer fine-grained controllability. 3 Overview of the Synthline approach This study constitutes the first iteration of a broader design science cycle, where design problems and knowledge questions are iteratively addressed through cycles of improvement design and empirical investigation [ 3 ] . We chose the PL approach to systematically control and configure the generation of synthetic datasets according to various RE use cases. The PL approach provides a form of systematic customization of a common platform from which multiple variants can be derived [ 21 ] . This paradigm offers two important benefits in our synthetic data generation context. First, it enables systematic customization, ensuring each variant aligns with the specific needs and constraints of its RE use case. Second, it helps improve the quality of the generated data as the platform commonalities can be systematically

Read the complete text