How Effective are Generative Large Language Models in Performing Requirements Classification?

Waad Alhoshan, Alessio Ferrari, Liping Zhao · 2025

Read How Effective are Generative Large Language Models in Performing Requirements Classification? by Waad Alhoshan, Alessio Ferrari, Liping Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16768.txt.

Introduction

Waad Alhoshan, Alessio Ferrari, and Liping Zhao test how well generative language models sort software requirements in this 2025 paper. Requirements are the written statements of what a system should do, and sorting them by type is a routine engineering task. Earlier models of the fill-in style did the job well, but text-generating models had barely been tried.

The team ran three generative models, Bloom, Gemma, and Llama, through more than four hundred experiments. They used three known datasets and both simple yes-no sorting and multi-class sorting. Prompt design and model architecture mattered in every setting. The dataset itself mattered only sometimes, depending on how hard the sorting task was. The authors say builders should tune prompts and match the model to the task rather than expect one setup to fit all. The study maps where these models stand on a bread-and-butter engineering chore.

Catalog record

CREATORWaad Alhoshan, Alessio Ferrari, Liping Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE133 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16768

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Effective are Generative Large Language Models in Performing Requirements Classification? Waad Alhoshan, Alessio Ferrari, Liping Zhao Abstract . Context: In recent years, transformer-based large language models (LLMs) have revolutionised natural language processing (NLP), with generative models opening new possibilities for tasks that require context-aware text generation. Requirements engineering (RE) has also seen a surge in the experimentation of LLMs for different tasks, including trace-link detection, regulatory compliance, and others. Problem: Requirements classification is a common task in RE. While non-generative LLMs like BERT have been successfully applied to this task, there has been limited exploration of generative LLMs. This gap raises an important question: how well can generative LLMs, which produce context-aware outputs, perform in requirements classification? Solution: In this study, we explore the effectiveness of three generative LLMs—Bloom, Gemma, and Llama—in performing both binary and multi-class requirements classification. We design an extensive experimental study involving over 400 experiments across three widely used datasets (PROMISE NFR, Functional-Quality, and SecReq). Our study focuses on the impact of four core components: models, datasets, prompts, and tasks. Additionally, we compare the best-performing generative LLMs from this study with the top non-generative models from previous works to identify which model type achieves superior performance in requirements classification. Results: Our findings show that model selection significantly influences performance, with Bloom excelling in precision, achieving a maximum weighted precision score of approximately 0.77 0.77 . Gemma excelled in binary classification, particularly in recall, with an average weighted recall rate of 0.51 0.51 . Llama offered balanced performance across both precision and recall, making it well suited for both binary and multi-class classification tasks. Prompt formulation plays an essential role, with assertion-based prompts being the most effective. We also observe that generative models demonstrate robustness to dataset variations, maintaining consistent performance despite changes in input text. In comparison to non-generative models (e.g., All-Mini and SBERT), generative models generally performed less effectively in multi-class classification, while non-generative models, particularly All-Mini, outperformed them in binary classification tasks. Contributions: This paper presents the first comprehensive experimental study of generative LLMs for requirements classification, offering key insights into their strengths and limitations. We highlight the critical role of four core components—models, datasets, prompts, and tasks—on classification performance. Our study concludes that while factors like prompt design and LLM architecture are universally important, others-—such as dataset variations—-have a more situational impact, depending on the complexity of the classification task. This insight can guide future model development and deployment strategies, focusing on optimising prompt structures and aligning model architectures with task-specific needs for improved performance. DOI: XXXXXXX.XXXXXXX Journal: TOSEM Volume: 00 00 000 0 1. Introduction In recent years, the transformer architecture ( Vaswani et al., 2017 ) has revolutionised NLP and reshaped the landscape of the field. Known for its unparalleled efficiency in processing and generating text, the transformer has become the foundational architecture behind the majority of state-of-the-art LLMs , including BERT ( Devlin et al., 2018 ) , ChatGPT ( Achiam et al., 2023 ) , Llama ( Touvron et al., 2023 ) , Bloom ( Le Scao et al., 2023 ) , and GPT-3 ( Dale, 2021 ) , among others. At the heart of its power is the self-attention mechanism, which enables each word or token in an input sequence to interact with every other word, capturing complex contextual relationships irrespective of their distance in the sequence. This attention-driven connectivity empowers transformer-based models to tackle a wide array of real-world applications, ranging from chatbots and virtual assistants to content creation, language translation, text summarization, sentiment analysis, education, and medical diagnostics ( Alto, 2024 ; Raiaan et al., 2024 ) . With the rapid advancement of LLMs, their capabilities continue to grow at an unprecedented pace. To fully harness the potential of LLMs, our previous study ( Alhoshan et al., 2023 ) explored the effectiveness of BERT-based models for requirements classification tasks. Specifically, we applied four BERT-based models (SBERT 1 1 1 https://sbert.net/ , all-MiniLM-L12-v2 (All-Mini) 2 2 2 https://docs.pinecone.io/models/all-MiniLM-L12-v2 , SObert ( Tabassum et al., 2020 ) , and BERT4RE ( Ajagbe and Zhao, 2022 ) ) with zero-shot learning approaches to both binary and multi-class classification tasks, achieving F1 scores ranging from 0.66 to 0.80 . While these non-generative models, such as BERT-related models, have shown promising performance, their reliance on high-dimensional word embeddings to make predictions may limit their ability to fully adapt to varying contexts. Although non-generative LLMs can sometimes be applied to generate text, their primary objective is not oriented toward text generation. Instead, they primarily rely on tasks such as predicting masked tokens or next sentence prediction during pre-training, which reflects their focus on understanding and encoding context rather than generating human-like, contextually relevant text. This potential limitation highlights the need to explore how generative LLMs, which can dynamically produce text tailored to the input context, could enhance performance in requirements classification. In this paper, we aim to bridge this gap by investigating the performance of generative LLMs in the context of requirements classification. Our study focusses on four core components that are critical for leveraging the power of generative models: 1) Models, 2) Datasets, 3) Prompts, and 4) Tasks . Each of these components plays a central role in influencing classification outcomes, and understanding their impact is essential for improving the practical use of generative LLMs. To the best of our knowledge, this is the first comprehensive experimental study of using generative LLMs for requirements classification tasks. Our contributions are highlighted as follows: First, we examine three state-of-the-art generative LLMs (Llama ( Touvron et al., 2023 ) , Bloom ( Le Scao et al., 2023 ) , and Gemma ( Team et al., 2024 ) ). These models represent the next step in language-generation capabilities. We explore their unique architectures and how these features differentiate them from non-generative models, and we investigate how these architectures could influence their effectiveness in requirements classification. Next, we investigate the datasets used for evaluation. Dataset selection plays an important role in model performance, as the structure, domain, and size of the data can significantly influence the results. In this study, we utilise three well-established datasets—PROMISE NFR ( Cleland-Huang et al., 2007b ; Cleland-Huang et al., 2007a ) , Functional-Quality ( Dalpiaz et al., 2019 ) , and SecReq ( Knauss et al., 2011 ; Knauss et al., 2021 ) —which have been widely adopted by researchers in the field of RE. Additionally, we explore various variations of these datasets to assess how changes to the requirements text and class labels might impact the performance of requirements classification. We also examine the role of prompts , the input structures that are fed to generative LLMs. The formulation of prompts is an essential factor influencing model output ( White et al., 2023 ; Ronanki et al., 2023 ) . Our research investigates how different prompt patterns can optimise generative LLMs for requirements classification, ensuring that the models generate the most relevant and accurate classifications. Lastly, we analyse the tasks themselves, specifically how different types of requirements classification tasks—binary versus multi-class—interact with generative LLMs. Understanding the impact of task structure on generative models is essential for assessing their potential in practical applications, such as software development and industry

Read the complete text