Leveraging LLMs for Structured Information Extraction and Analysis from Cloud Incident Reports (Work In Progress Paper) Conference: Companion of the 17th ACM/SPEC International Conference on Performance Engineering; May 04–08, 2026; Florence, Italy Companion of the 17th ACM/SPEC International Conference on Performance Engineering (ICPE Companion ’26), May 04–08, 2026, Florence, Italy DOI: 10.1145/3777911.3801103 ISBN: 979-8-4007-2326-1/2026/05 CCS: Computer systems organization Reliability
Xiaoyu Chu, Shashikant Ilager, Yizhen Zang, Sacheendra Talluri, Alexandru Iosup · 2026
Read Leveraging LLMs for Structured Information Extraction and Analysis from Cloud Incident Reports (Work In Progress Paper) Conference: Companion of the 17th ACM/SPEC International Conference on Performance Engineering; May 04–08, 2026; Florence, Italy Companion of the 17th ACM/SPEC International Conference on Performance Engineering (ICPE Companion ’26), May 04–08, 2026, Florence, Italy DOI: 10.1145/3777911.3801103 ISBN: 979-8-4007-2326-1/2026/05 CCS: Computer systems organization Reliability by Xiaoyu Chu, Shashikant Ilager, Yizhen Zang, Sacheendra Talluri, Alexandru Iosup (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16818.txt.
Introduction
Chu and co-authors explore using large language models to mine cloud incident reports. Vendors publish these reports after failures, but the text is long and unstructured. The team gathered over 3,000 reports from AWS, Azure, and Google Cloud, and annotated them by hand. They compared six prompt strategies and six models, from light to state-of-the-art. The authors report metadata extraction accuracy of 75 to 95 percent. Few-shot prompting helped most fields. Light models gave good trade-offs of accuracy, cost, and speed.
Catalog record
| CREATOR | Xiaoyu Chu, Shashikant Ilager, Yizhen Zang, Sacheendra Talluri, Alexandru Iosup |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31824 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16818
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging LLMs for Structured Information Extraction and Analysis from Cloud Incident Reports (Work In Progress Paper) Conference: Companion of the 17th ACM/SPEC International Conference on Performance Engineering; May 04–08, 2026; Florence, Italy Companion of the 17th ACM/SPEC International Conference on Performance Engineering (ICPE Companion ’26), May 04–08, 2026, Florence, Italy DOI: 10.1145/3777911.3801103 ISBN: 979-8-4007-2326-1/2026/05 CCS: Computer systems organization Reliability Xiaoyu Chu, Shashikant Ilager, Yizhen Zang, Sacheendra Talluri, Alexandru Iosup Abstract . Incident management is essential to maintain the reliability and availability of cloud computing services. Cloud vendors typically disclose incident reports to the public, summarizing the failures and recovery process to help minimize their impact. However, such reports are often lengthy and unstructured, making them difficult to understand, analyze, and use for long-term dependability improvements. The emergence of LLMs offers new opportunities to address this challenge, but how to achieve this is currently understudied. In this paper, we explore the use of cutting-edge LLMs to extract key information from unstructured cloud incident reports. First, we collect more than 3,000 incident reports from 3 leading cloud service providers (AWS, AZURE, and GCP), and manually annotate these collected samples. Then, we design and compare 6 prompt strategies to extract and classify different types of information. We consider 6 LLM models, including 3 lightweight and 3 state-of-the-art (SotA), and evaluate model accuracy, latency, and token cost across datasets, models, prompts, and extracted fields. Our study has uncovered the following key findings: (1) LLMs achieve high metadata extraction accuracy, 75 % – 95 % 75\%\text{--}95\% depending on the dataset. (2) Few-shot prompting generally improves accuracy for meta-data fields except for classification, and has better (lower) latency due to shorter output-tokens but requires 1.5 – 2 × 1.5\text{--}2\times more input-tokens. (3) Lightweight models (e.g., Gemini 2.0, GPT 3.5) offer favorable trade-offs in accuracy, cost, and latency; SotA models yield higher accuracy at significantly greater cost and latency. Our study provides tools, methodologies, and insights for leveraging LLMs to accurately and efficiently extract incident-report information. The FAIR data and code are publicly available at https://github.com/atlarge-research/llm-cloud-incident-extraction . † † cc-license: by 1. Introduction Incident management is an important and complex process in cloud computing operations ( 18 ; 21 ; 1 ) . A cloud incident could occur for multiple reasons, including hardware or software failures, deployment and runtime errors, network disruptions, or security breaches ( 12 ; 22 ) . Such incidents can disrupt user services, affecting the reliability of the system. To minimize the impact, once an incident occurs, cloud operators provide public incident reports ( 23 ; 4 ; 10 ) to inform users about the incident management process and root causes. However, these reports are very complex, lengthy, and too unstructured to understand or use for long-term analysis and mitigation. As a result, while individual reports are disclosed for each incident, there is a lack of a public dataset and long-term analysis of these incidents. Currently, LLMs have become powerful tools and offer new opportunities to address such problem. LLMs have already been effectively used in performance engineering, such as log parsing ( 3 ; 16 ) , root cause analysis ( 1 ; 28 ; 21 ; 7 ; 11 ) . Therefore, we propose to leverage advanced LLMs to extract and analyze key information from public cloud incident reports. We design a workflow to adapt LLMs for report data extraction. Our work addresses three main challenges in this workflow process: First, there is no publicly available labeled dataset of cloud incident reports, nor open source tools for automatically extracting and analyzing their key information. Previous studies on cloud incident and outage analysis have relied on manual or rule-based data extraction ( 12 ; 25 ) , which is highly time-consuming and labor-intensive due to the length and complexity of the reports. For example, in our collected datasets from Azure and GCP, the average report length exceeds 500 words (see Table 1 ). Second, there is a lack of methodology and systematic evaluation for assessing the performance and cost of applying LLMs to incident report data extraction. Although LLMs are widely used for information extraction ( 24 ; 30 ) , their design and application for incident report extraction remain unclear. In particular, it is not well understood how to structure data, how to design prompts, and how to adapt LLMs for cloud incident analysis. Third, there are limited studies to perform long-term statistical characterization on incident reports. Previous studies have conducted incident report analysis on cloud services ( 12 ; 25 ) , but these analyses are typically based on labeled datasets and cover only limited fields, making them difficult to generalize to long-term datasets or diverse incident reports. Addressing these challenges, our key contributions are: (1) Dataset and toolbox construction ( Section 2 ): This work is the first to provide open-sourced datasets and toolbox for systematically exploring and evaluating LLMs for structured information extraction and analysis of cloud incident reports. We collect over 3,000 cloud incident reports with annotated subsets for report-data evaluation. We also develop a toolbox for leveraging LLMs for report data extraction, evaluation, and analysis. (2) LLMs adaption and evaluation ( Section 3 and 4 ): This work demonstrates the usefulness of SotA LLMs such as GPT-4o for incident information extraction and analysis. We systematically evaluate 6 LLMs models of different size with diverse prompt strategies, we compare their accuracy and cost with different evaluation metrics. Our results achieve 75%–95% accuracy on metadata extraction, based on the dataset. We also provide recommendations for selecting prompts and LLMs based on their latency and cost. (3) Open science: We follow the FAIR principles of open science and release the datasets 1 1 1 Zenodo: https://zenodo.org/records/14010282 and software 2 2 2 GitHub: https://github.com/atlarge-research/llm-cloud-incident-extraction to enable reproducibility and further research. 2. Data Collection and Preparation 2.1. Dataset Collection Table 1 . Summary of cloud incident reports. ID Name Period # Rows # Labeled Avg. Words 1 AWS 2016 - 2022 774 150 (19%) 151 2 AZURE 2019 - 2024 127 95 (75%) 575 3 GCP 2016 - 2021 2,186 215 (10%) 533 TOTAL 3 sources 2016 - 2024 3,087 460 (15%) — Table 2 . Summary of extracted fields. Legend: indicates fully extracted from the report, indicates inferred by LLMs. EM = Exact Match, TK = Token-Level F1, BS = BERTScore. ID Extracted Fields Type Datasets Eval. Metric AWS AZURE GCP EM TK BS 1 Service Name entity ✓ 2 Location entity - ✓ 3 Start Time entity ✓ 4 End Time entity ✓ 5 Timezone entity ✓ 6 Service Categ. class ✓ 7 Root Cause Categ. class - - ✓ 8 User Symp. Categ. multi-class ✓ 9 User Symptom text ✓ 10 Root Cause text - - ✓ Incident reports from cloud service providers are frequently published online via their respective web platforms ( 23 ; 4 ; 10 ) . However, these reports vary in structure, and the extent of publicly available information differs across providers. To compile a diverse set of incident report data, we sourced reports from 3 major cloud service providers: AWS, AZURE, and GCP, covering the period from 2016 to 2024. In total, we collected approximately 3,000 incident reports. Each report consists