How Good Are Synthetic Requirements ? Evaluating LLM-Generated Datasets for AI4RE

Abdelkarim El-Hajjami, Camille Salinesi · 2025

Read How Good Are Synthetic Requirements ? Evaluating LLM-Generated Datasets for AI4RE by Abdelkarim El-Hajjami, Camille Salinesi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-21138.txt.

Introduction

El-Hajjami and Salinesi test in twenty twenty-five two tricks to make language-model-generated training data less repetitive for requirements engineering tasks. Fake data is cheap where real labelled data is scarce or private, but model output tends to echo itself, which weakens the models trained on it. Their configurable generator combines asking for many samples per prompt with an actor-critic loop that keeps editing prompts for variety.

Many-sample prompting raised both variety and downstream scores across four classification tasks, sometimes by wide margins. The editing loop raised word variety everywhere but helped accuracy only on some tasks, a warning against chasing variety alone. Strikingly, fake data matched or beat human-written data where real data was thinnest.

Catalog record

CREATORAbdelkarim El-Hajjami, Camille Salinesi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-21138

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Good Are Synthetic Requirements ? Evaluating LLM-Generated Datasets for AI4RE Abdelkarim El-Hajjami, Camille Salinesi Abstract The shortage of publicly available, labeled requirements datasets represents a major barrier to advancing Artificial Intelligence for Requirements Engineering (AI4RE) research. While Large Language Models offer promising capabilities for synthetic data generation, systematic approaches for controlling and optimizing the quality of generated requirements remain underexplored. This paper presents Synthline v1, an enhanced Product Line approach for generating synthetic requirements data that extends our earlier v0 version with advanced generation strategies and curation techniques. We investigate four research questions examining how prompting strategies, automated prompt optimization, and post-generation curation affect synthetic data quality across four requirements classification tasks: defects detection, functional vs. non-functional classification, quality vs. non-quality classification, and security vs. non-security classification. Our experimental evaluation demonstrates that multi-sample prompting significantly improves both utility and diversity compared to single-sample generation, with F1-score improvements ranging from 6-44 percentage points. The application of PACE (Prompt Actor-Critic Editing) for automated prompt optimization shows task-dependent results, substantially improving functional requirements classification (+32.5 percentage points) while degrading performance on other tasks. Counterintuitively, similarity-based curation consistently improves diversity metrics but often hurts classification performance, revealing that not all redundancy is detrimental to Machine Learning effectiveness. Most significantly, our results establish that synthetic requirements can match or surpass human-authored requirements for specific classification tasks, with synthetic data achieving superior performance for security (+7.8 percentage points) and defects classification (+15.4 percentage points). These findings provide actionable insights for the AI4RE community and demonstrate a viable path toward addressing dataset scarcity through systematic synthetic data generation. 1 Introduction Artificial Intelligence for Requirements Engineering (AI4RE) has emerged as a rapidly evolving research domain, leveraging machine learning (ML) and natural language processing (NLP) techniques to automate and enhance RE tasks [ 5 ] . The effectiveness of these AI-based methods is fundamentally dependent on the availability of high-quality, task-specific datasets that serve as the foundation for training and evaluation [ 6 ] . Although generative AI and Large Language Models (LLMs) with in-context learning have reduced reliance on extensive training datasets, achieving optimal performance still requires sufficient high-quality data for fine-tuning [ 11 ] . Additionally, evaluating these new approaches requires adequate benchmarks specific to RE [ 10 ] . However, a fundamental challenge persists within the RE community, as the shortage of publicly available, labeled requirements datasets continues to represent a major barrier to advancing automated RE research [ 6 , 10 , 12 , 13 ] . Consequently, addressing this data scarcity has become essential for advancing AI4RE research and enabling effective automated solutions in RE. Over the past decade, surveys of AI4RE have reported roughly forty open dataset entries [ 14 , 15 ] . However, a close inspection of the supplementary material attached to [ 14 ] —cross-checked with the corpus descriptions summarized in [ 15 ] —shows that the number of truly distinct, openly accessible, and ML-ready datasets is far smaller. Many of the cited resources correspond to different versions or repackagings of the same underlying data. The most notable case is the PROMISE family, which includes the original PROMISE corpus, its extended version PROMISE_exp, and the Tera-PROMISE redistribution—all derived from the same base dataset. Once such duplications are consolidated, and after excluding entries linked to dead URLs, restricted-access sites, or raw unannotated sources, the number of genuinely distinct, labeled, and openly reusable datasets falls below twenty. This small core—centering on PROMISE, iTrust, PURE, NASA CM-1 and MODIS, and a small number of security or NFR corpora—accounts for the vast majority of dataset use in AI4RE research. These datasets share several limitations: most are small in scale, typically containing only a few hundred requirements; they commonly have class imbalance, offer limited domain and linguistic diversity as nearly all originate from information system projects and are written exclusively in English, and provide annotations that can be inconsistent—as even trained annotators frequently produce divergent labels when classifying requirements [ 13 ] —which makes reliable benchmarking difficult. The proprietary nature of most industrial requirements adds further barriers, as organizations are generally unwilling to release specifications they consider strategic assets. These limitations point to an urgent need for the AI4RE community to take a data-centric direction, where building high-quality datasets becomes the key to real progress. Such datasets should be large enough, with balanced class distributions, and truly diverse — covering different domains, languages, specification formats, levels, requirement types, and stakeholder perspectives, while staying faithful to the semantics of real requirements. They must also provide consistent, high-quality annotations to enable reliable benchmarking, while minimizing unintended biases that could distort model behavior. Prioritizing data quality over marginal improvements in modeling techniques is essential to enable reproducible, trustworthy, and practically useful AI4RE systems. Synthetic data generation using Large Language Models (LLMs) emerges as a viable solution to mitigate this data scarcity in AI4RE. Recent advances in LLMs enable the automated creation of large volumes of requirements-like text that can be tailored to specific tasks or domains through prompt engineering. This approach promises not only to alleviate the dependence on proprietary industrial datasets but also to improve coverage of underrepresented domains, requirement types, and linguistic properties. To the best of our knowledge, our prior work was the first to systematically apply LLM-based synthetic data generation in the context of RE [ 1 ] . We introduced the first version of Synthline (v0), a Product Line (PL) approach for controlling synthetic requirements data generation. Synthline v0 applied systematic variability management to configure key properties of generated data via a Feature Model (FM). Our empirical evaluation demonstrated that models trained exclusively on Synthline-generated data consistently outperformed models trained solely on human-authored data in both precision and recall. Specifically, synthetic data enabled improvements of up to +37% in precision and +30% in recall compared to the real-data-only baseline. The best results were obtained through hybrid training strategies that combined real and synthetic data, achieving precision gains of up to +85% and a doubling of recall relative to models trained exclusively on real data. However, this initial study also revealed important challenges: the synthetic data exhibited lower diversity than real data, and combining synthetic sources indiscriminately sometimes degraded classifier performance. These findings highlighted the need for more advanced generation control techniques and systematic post-generation curation to improve the quality and utility of synthetic data. This paper presents the second iteration within our Design Science Research (DSR) methodology [ 2 ] , extending Synthline with enhanced generation and curation strategies aimed at optimizing synthetic data quality. Building upon our initial findings, this iteration focuses on optimizing the generation process itself. We investigate how different prompting strategies including Prompt Actor-Critic Editing (PACE) [ 3 ] and post-generation curation influence the quality—defined by diversity and utility—of synthetic requirements data. Specifically, we address the following research questions: • RQ1: How does the quantity of samples requested in each prompt impact the quality of the resulting synthetic data? • RQ2: How does the application of PACE-based prompt optimization influence the quality of the resulting synthetic data? • RQ3: How does a similarity-based curation affect the quality of the resulting synthetic data? • RQ4: To what extent can synthetic requirements replace or surpass human-authored requirements for classification tasks?

Read the complete text