Automated Concern Extraction from Textual Requirements of Cyber-Physical Systems: A Multi-solution Study Journal: TOSEM CCS: Software and its engineering Requirements analysis CCS: Computing methodologies Information extraction CCS: Computing methodologies Natural language generation
Dongming Jin, Zhi Jin, Xiaohong Chen, Zheng Fang, Linyu Li, Shengxin Zhao, Chunhui Wang, Hongbin Xiao · 2025
Read Automated Concern Extraction from Textual Requirements of Cyber-Physical Systems: A Multi-solution Study Journal: TOSEM CCS: Software and its engineering Requirements analysis CCS: Computing methodologies Information extraction CCS: Computing methodologies Natural language generation by Dongming Jin, Zhi Jin, Xiaohong Chen, Zheng Fang, Linyu Li, Shengxin Zhao, Chunhui Wang, Hongbin Xiao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19237.txt.
Introduction
Dongming Jin and seven co-authors benchmark tools for a hard text task. The tools pull concerns out of written requirements. The systems at stake blend software with the physical world. Cars and medical devices are two cases. Pulling the right concerns out by hand is slow work.
Their benchmark is called ReqEBench. It holds thousands of requirements from a dozen real systems. Its scale and complexity match real projects. It covers the concern types such projects raise. With a fair test bed, rival tools can be compared honestly. Engineers can see which tools deserve trust.
Catalog record
| CREATOR | Dongming Jin, Zhi Jin, Xiaohong Chen, Zheng Fang, Linyu Li, Shengxin Zhao, Chunhui Wang, Hongbin Xiao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19237
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Automated Concern Extraction from Textual Requirements of Cyber-Physical Systems: A Multi-solution Study Journal: TOSEM CCS: Software and its engineering Requirements analysis CCS: Computing methodologies Information extraction CCS: Computing methodologies Natural language generation Dongming Jin, Zhi Jin, Xiaohong Chen, Zheng Fang, Linyu Li, Shengxin Zhao, Chunhui Wang, Hongbin Xiao Abstract . Cyber-physical systems (CPSs) are characterized by a deep integration of the information space and the physical world, which makes the extraction of requirements concerns more challenging. Some automated solutions for requirements concern extraction have been proposed to alleviate the burden on requirements engineers. However, evaluating the effectiveness of these solutions, which relies on fair and comprehensive benchmarks, remains an open question. To address this gap, we propose ReqEBench , a new CPSs requirements concern extraction benchmark, which contains 2,721 requirements from 12 real-world CPSs. ReqEBench offers four advantages. ❶ It aligns with real-world CPSs requirements in multiple dimensions, e.g., scale and complexity. ❷ It covers comprehensive concerns related to CPSs requirements. ❸ It undergoes a rigorous annotation process. ❹ It covers multiple application domains of CPSs, e.g., aerospace and healthcare. We conducted a comparative study on three types of automated requirements concern extraction solutions and revealed their performance in real-world CPSs using our ReqEBench . We found that the highest F1 score of GPT-4 is only 0.24 in entity concern extraction. We further analyze failure cases of popular LLM-based solutions, summarize their shortcomings, and provide ideas for improving their capabilities. We believe ReqEBench will facilitate the evaluation and development of automated requirements concern extraction. 1. Introduction “Crosscutting concerns are often scattered and tangled within requirements texts. Their extraction acts as a bridge from unstructured natural language requirements to structured analysis.” —— Rashid Awais, RE 2002 ( Rashid et al. 2002 ) ; Rago Alejandro, JSS 2019 ( Rago et al. 2019 ) . Cyber-physical systems (CPSs) are ubiquitous in modern life ( Lee 2010 ) , from mobile phones and other electronic products to cars and spacecraft ( Menghi et al. 2020 ) ( Cornejo et al. 2021 ) . These systems integrate cyber components ( e.g., software and computational units) with physical devices ( i.e., sensors, actuators, mechanical devices) and interact with various physical environments continuously ( Xu et al. 2023 ) ( Mandrioli et al. 2023 ) . Thus, they are characterized by the tight coupling of software and physical components. Meanwhile, with the rapid proliferation of heterogeneous sensing and actuating devices, CPSs continue to grow in scale and become increasingly diverse in terms of interactions and applications ( Feng et al. 2020 ) . Such trends significantly increase the complexity of CPSs requirements and make the task of understanding and validating requirements texts particularly challenging. To address this complexity, extracting key concerns from long and unstructured requirements texts has become a crucial step in the requirements analysis process ( Shi et al. 2020 ) ( Frattini et al. 2020 ) . These concerns typically span multiple types ( i.e., entity and interaction) and dimensions ( e.g., physical device and external system), which together form the foundation for downstream modeling and validation tasks. However, the process of extracting these concerns manually is extremely time-consuming and requires a steep learning curve ( Zhou et al. 2022 ) . Requirements engineers not only need to carefully read and interpret long documents but also need to possess extensive domain knowledge and have expertise regarding requirements concerns. The steep learning curve and high cognitive burden associated with this task hinder its efficiency. Consequently, despite automated extraction of requirements concerns is highly valuable for easing engineers’ workload, its practical adoption in real world CPSs remains limited. This may be because existing solutions lack sufficient adaptability and robustness for complex industrial requirements. Reviewing previous studies, various artificial intelligence (AI) techniques have been proposed to automatically extract requirements concerns from requirements texts. These solutions fall into three types, i.e., heuristic rule-based solutions ( Robeer et al. 2016 ) ( Zaki-Ismail et al. 2022 ) , machine learning-based solutions ( Li et al. 2020 ) ( Huang and Riloff 2010 ) , and language model-based solutions ( Cámara et al. 2023 ) ( Ruan et al. 2023 ) . They have shown promising results on their respective benchmarks, but their performance in real-world applications to CPSs is still far from satisfactory ( Chen et al. 2023 ) . This mismatch suggests that existing benchmarks are poorly aligned with the requirements of real-world CPSs. Specifically, the key factor is that existing benchmarks do not adequately reflect the complexity, diversity, and domain specificity of real-world CPS requirements. As a result, they are insufficient for accurately evaluating the practical effectiveness of these approaches, highlighting the need for new benchmarks and methods that can better align with real-world CPS requirements scenarios. After a systematic and in-depth analysis, we found that the existing benchmarks have three shortcomings: (1) Insufficient Complexity. The requirements texts in existing benchmarks are mostly made up of hand-crafted cases in textbooks or courses, whose scale and linguistic phenomena ( e.g., long-range coreference, nested conditionals domain jargon) fall far short of real CPSs. For example, the text of the real-world CPS requirements can span dozens of pages ( Feng et al. 2020 ) , but the cases in IT4RE ( Al-Hroob et al. 2018 ) contain fewer than ten sentences. (2) Insufficient Coverage. Existing benchmarks only focus on certain requirements concerns ( e.g., data function ( Li et al. 2022 ) and transaction function ( Shi et al. 2020 ) ). They fail to cover the multidimensional concerns of CPSs requirements, including the concerns about multiple individual entities ( e.g., software system, physical devices, and environment objects) and the critical cross-entity interactions ( e.g., phenomena linkage and requirements constraint). In addition, application domains are very limited, providing weak evidence of generalization across diverse CPS application domains such as healthcare, aerospace, and smart homes. (3) Limited Accessibility. Researchers usually use their own benchmarks for evaluation, but tend not to make them public due to privacy concerns. Worst of all, there are differences and/or inconsistencies between different benchmarks in requirements concern classification and concept annotation. These factors hinder the production of fair comparisons of various solutions. Therefore, in order to better promote the automation of CPSs requirements extraction, it is necessary to construct a public benchmark that can systematically capture the core concerns in real-world CPS requirements. In this paper, we propose ReqEBench , a new requirement concern extraction benchmark from CPS requirements, which contains 2,721 requirements from 12 real-world industrial CPSs. ReqEBench offers four advantages compared to existing benchmarks. ❶ It aligns with real-world CPSs in scale and characteristics of requirements texts. The characteristics include long-range coreference, nested conditionals, and domain-specific jargon. This allows it to effectively evaluate the performance of solutions in the automated extraction of key concerns in real-world CPSs scenarios. ❷ It covers more comprehensive concerns related to CPSs requirements. Specifically, ReqEBench provides a systematic taxonomy of CPSs requirements concerns. The two top-level categories are related to entities and interactions, respectively, spanning nine core entity concerns ( e.g., physical device, environment object, and external system), as well as three key interaction concerns that are particularly critical for characterizing CPSs requirements ( e.g., phenomenon linkage and requirements