A Dynamic and High-Precision Method for Scenario-Based HRA Synthetic Data Collection in Multi-Agent Collaborative Environments Driven by LLMs
Xingyu Xiao, Peng Chen, Qianqian Jia, Jiejuan Tong, Jingang Liang, Haitao Wang · 2025
Read A Dynamic and High-Precision Method for Scenario-Based HRA Synthetic Data Collection in Multi-Agent Collaborative Environments Driven by LLMs by Xingyu Xiao, Peng Chen, Qianqian Jia, Jiejuan Tong, Jingang Liang, Haitao Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00022.txt.
Introduction
Xingyu Xiao and five co-authors use language models to fake the data that human reliability studies need. Such studies estimate how likely people are to err in team settings, such as control rooms. Real data is scarce, coarse, and slow to gather by expert review. Their 2025 method builds scenarios with several AI agents playing team roles. It then measures workload, a key driver of error, inside those scenes.
Fine-tuned models run the scenes and log fine-grained values as events unfold. That gives dynamic data, not just end scores. The authors claim high precision for the synthetic set. They offer it as a way to grow reliability research where real trials are costly or risky. Human experts still set the scene rules, but the labor per run drops sharply.
Catalog record
| CREATOR | Xingyu Xiao, Peng Chen, Qianqian Jia, Jiejuan Tong, Jingang Liang, Haitao Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-00022
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Dynamic and High-Precision Method for Scenario-Based HRA Synthetic Data Collection in Multi-Agent Collaborative Environments Driven by LLMs Xingyu Xiao, Peng Chen, Qianqian Jia, Jiejuan Tong, Jingang Liang, Haitao Wang Abstract HRA (Human Reliability Analysis) data is crucial for advancing HRA methodologies. however, existing data collection methods lack the necessary granularity, and most approaches fail to capture dynamic features. Additionally, many methods require expert knowledge as input, making them time-consuming and labor-intensive. To address these challenges, we propose a new paradigm for the automated collection of HRA data. Our approach focuses on key indicators behind human error, specifically measuring workload in collaborative settings. This study introduces a novel, scenario-driven method for workload estimation, leveraging fine-tuned large language models (LLMs). By training LLMs on real-world operational data from high-temperature gas-cooled reactors (HTGRs), we simulate human behavior and cognitive load in real time across various collaborative scenarios. The method dynamically adapts to changes in operator workload, providing more accurate, flexible, and scalable workload estimates. The results demonstrate that the proposed WELLA (Workload Estimation with LLMs and Agents) outperforms existing commercial LLM-based methods in terms of prediction accuracy. 1 Introduction Human error is critically important, with statistics indicating that approximately 50%-80% of incidents in high-risk industries are caused by human error. Therefore, human reliability analysis (HRA) is a systematic approach to assessing and improving the reliability of human performance in complex systems, particularly in high-risk industries such as nuclear power, aviation, and healthcare. However, most HRA methods rely on expert estimations rather than empirical data [ 1 ] . Examples of such methods include human cognitive reliability (HCR) [ 2 ] , standardized plant analysis risk-human reliability analysis (SPAR-H) [ 3 ] , and cognitive reliability and error analysis method (CREAM) [ 4 ] . Consequently, numerous scholars have focused on improving data collection in HRA, which is crucial for revolutionizing HRA algorithms. A notable example is the integrated human event analysis system for event and condition assessment (IDHEAS-ECA) [ 5 ] , the latest method developed by the NRC based on IDHEAS-DATA [ 6 ] . Xiao et al. developed a fast and efficient base human error probability solving algorithm based on the IDHEAS-DATA and a large model knowledge graph [ 1 ] . However, HRA data is particularly scarce, and traditional methods typically rely on manually completed surveys to label the data, such as human reliability data extraction (HuREX) [ 7 ] and scenario authoring, characterization, and debriefing application (SACADA) [ 8 ] , are labor-intensive and time-consuming. Furthermore, small sample sizes of operators for low-probability events present a significant challenge. For example, when dealing with probabilities as low as 1E-3, it may require up to 1,000 data points to observe a naturally occurring error. This often necessitates the introduction of artificial errors in complex scenarios to ensure an adequate dataset for analysis [ 9 ] . These method is often collected through surveys, which are typically administered static post facto. Boring et al. [ 10 ] emphasize the use of the HUNTER dynamic HRA to simulate operator performance. This approach aims to understand both the limitations of synthetic data and its potential advantages compared to expert estimation methods. This method temporarily addresses the issue of static. However, there is still a critical issue, as emphasized by the U.S. NRC: the focus should not solely be on the manifestation of human error but rather on the underlying causes behind human error [ 5 ] . The data generated by existing methods predominantly reflect error of omission (EOO) and error of commission (EOC) [ 11 ] . As introduced in Heinrich’s Theory and the Iceberg Model in Section 2.1 , these are merely surface-level manifestations. In order to truly prevent and provide early warnings, we must focus on the underlying indicators behind human error. Therefore, there is a need to develop dynamic methods for generating synthetic underlying metrics behind human error data. To address this, a new paradigm for scenario-based HRA data automation collection is proposed, leveraging large language models (LLMs). The scenario we investigate involves multi-agent collaboration, as most real-world tasks require teamwork. Specifically, we use the example of the main control room in a multi-module high-temperature gas-cooled reactor (HTGR) nuclear power plant. In this scenario, the main control room operates under the ’one operator controls two or more reactors’ model. The staffing of the multi-module HTGR control room includes three reactor operators (RO1, RO2, RO3), one secondary loop operator (CO), and one shift supervisor (SO). We begin by collecting workload data through surveys to measure the cognitive load during team collaboration. Building on macro-cognitive theories, we use large models to generate virtual cognitive trajectories. Next, using the Llama-factory framework, we implement the supervised fine-tuning (SFT) process for the Qwen2.5-7B model. Through this fine-tuning, we develop WELLA: Workload Estimation with LLMs and Agents. The results demonstrate that WELLA outperforms current commercial models for predicting workload for roles including RO1, RO2, RO3, CO, and SO. Section 2 reviews foundational studies on human error, workload estimation models, and the application of large language models for simulating human behavior. Section 3 presents the comprehensive framework employed in this research. Section 4 outlines the experimental design and key findings. Section 5 summarizes the contributions of this research, reflects on its limitations, and proposes directions for future work. 2 Related Work This section presents an in-depth analysis of the underlying factors contributing to human error, explores various workload estimation models, and discusses the application of large language models in simulating human behavior. 2.1 Underlying Factors Contributing to Human Error Human error, a critical element in safety analysis across various domains, is often the result of multiple interacting factors. These factors can be broadly categorized into cognitive, environmental, organizational, and physiological influences [ 12 ] , all of which shape an individual’s performance and decision-making process. Understanding these underlying causes is essential for mitigating risks and improving system design. It is often compared to the iceberg theory [ 13 ] , suggesting that there is much more beneath the surface of visible behavior. Prior to the occurrence of an incident, there are numerous underlying representations that influence the final outcome. The theory posits that an individual’s "self" resembles an iceberg, where only a small portion—behavior—is observable above the surface, while the larger, more complex inner world remains hidden beneath. This hidden world encompasses seven layers: behavior, coping mechanisms, emotions, perspectives, expectations, desires, and the self [ 14 ] . Each of these deeper layers plays a pivotal role in shaping behavior and ultimately influencing the occurrence of errors or accidents. Therefore, it is crucial to focus not only on the visible actions during the event but also on the factors and conditions that precede it. The Heinrich Law, [ 15 ] formulated by the renowned American safety engineer H.W. Heinrich, is a principle commonly known as the 300:29:1 ratio. Based on his analysis of workplace injury and accident statistics, Heinrich proposed this law to provide insights into accident prediction and risk management, particularly for insurance companies. The law stipulates that for every 300 unsafe acts or conditions present in a workplace, 29 will result in minor injuries or incidents, and of these 29 minor incidents, one will