CTI-HAL: A Human-Annotated Dataset for Cyber Threat Intelligence Analysis
Sofia Della Penna, Roberto Natella, Vittorio Orbinato, Lorenzo Parracino, Luciano Pianese · 2025
Read CTI-HAL: A Human-Annotated Dataset for Cyber Threat Intelligence Analysis by Sofia Della Penna, Roberto Natella, Vittorio Orbinato, Lorenzo Parracino, Luciano Pianese (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05866.txt.
Introduction
Sofia Della Penna and four co-authors release CTI-HAL, a human annotated dataset for cyber threat intelligence analysis. Advanced persistent threats use complex multi stage tactics, and intelligence reports describe them in unstructured natural language. Automatic extraction from such text is hard, and existing datasets limit evaluation and fine tuning of AI tools.
The new dataset is built by hand from threat reports and structured by the MITRE ATT and CK framework of adversary tactics and techniques. An inter annotator agreement study with a standard alpha statistic confirms annotation reliability. A large language model is then evaluated on the dataset in a real business context. Results show promising generalizability for automated extraction work.
Catalog record
| CREATOR | Sofia Della Penna, Roberto Natella, Vittorio Orbinato, Lorenzo Parracino, Luciano Pianese |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-05866
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CTI-HAL: A Human-Annotated Dataset for Cyber Threat Intelligence Analysis Sofia Della Penna, Roberto Natella, Vittorio Orbinato, Lorenzo Parracino, Luciano Pianese Abstract Organizations are increasingly targeted by Advanced Persistent Threats (APTs), which involve complex, multi-stage tactics and diverse techniques. Cyber Threat Intelligence (CTI) sources, such as incident reports and security blogs, provide valuable insights, but are often unstructured and in natural language, making it difficult to automatically extract information. Recent studies have explored the use of AI to perform automatic extraction from CTI data, leveraging existing CTI datasets for performance evaluation and fine-tuning. However, they present challenges and limitations that impact their effectiveness. To overcome these issues, we introduce a novel dataset manually constructed from CTI reports and structured according to the MITRE ATT&CK framework. To assess its quality, we conducted an inter-annotator agreement study using Krippendorff’s alpha, confirming its reliability. Furthermore, the dataset was used to evaluate a Large Language Model (LLM) in a real-world business context, showing promising generalizability. Dataset Availability Human-Labeled Granularity Format # Techniques Source MITRE ATT&CK D-IT (Chen et al., 2024) [ 1 ] ✓ [ 2 ] ✗ statement level csv 201 MITRE ATT&CK KB v14.1 D-PE (Chen et al., 2024) [ 1 ] ✓ [ 2 ] N/A statement level csv 189 real CTI reports v14.1 TTPHunter [Sentence-Based] (Rani et al., 2023) [ 3 ] ✓ [ 4 ] ✗ statement level csv 50 MITRE ATT&CK KB v13.1 TTPHunter [Document-Based] (Rani et al., 2023) [ 3 ] ✗ ✓ document level N/A N/A 50 real CTI reports v13.1 TTPXHunter [Sentence-TTP] (Rani et al., 2024) [ 5 ] ✗ ✗ statement level N/A 193 MITRE ATT&CK KB v15.1 TTPXHunter [Report-TTP] (Rani et al., 2024) [ 5 ] ✗ ✓ document level N/A N/A 149 real CTI reports v15.1 CTI-to-MITRE with NLP (Orbinato et al., 2022) [ 6 ] ✓ [ 7 ] ✗ statement level csv 188 MITRE ATT&CK KB v11.3 rcATT (Legoy et al., 2020) [ 8 ] ✓ [ 9 ] ✗ document level csv 215 MITRE ATT&CK KB v7.0 MITREtrieval (Huang et al., 2024) [ 10 ] ✓ [ 11 ] N/A document level json 165 mixed v10.1 TRAM [ 12 ] ✓ [ 13 ] ✓ statement level json 50 real CTI reports v13.1 IntelEX Ground Truth (Xu et al., 2023) [ 14 ] ✓ [ 15 ] ✓ document level docx 171 16 real CTI reports v15.1 LADDER (Alam et al., 2023) [ 16 ] ✓ [ 17 ] ✓ statement level csv 31 real CTI reports (Android malware) v13.1 LLMCloudHunter (Schwartz et al., 2024) [ 18 ] ✗ ✓ N/A N/A N/A 12 real CTI reports (cloud) v15.1 aCTIon (Siracusano et al., 2023) [ 19 ] ✗ ✓ N/A N/A N/A 204 real CTI reports v13.1 AttacKG+ (Zhang et al., 2024) [ 20 ] ✗ ✓ N/A N/A N/A real CTI reports v14.1 CTI-ATE (Alam et al., 2024) [ 21 ] ✓ [ 22 ] ✗ statement level tsv 115 MITRE ATT&CK KB v15.1 CTI-HAL ✓ ✓ statement level json 116 81 real CTI reports v15.1 Table 1: CTI Dataset Overview 1 Introduction The complexity of cyberattacks faced by organizations is constantly increasing. Today, companies must face increasingly sophisticated threats, commonly known as APTs. In this context, reactive security strategies, which respond only after an attack has occurred, are no longer sufficient. As a result, there is a growing shift towards proactive security strategies that aim to anticipate the moves of attackers and neutralize threats before they manifest. To develop proactive security strategies, it is essential to gather information on APTs through CTI sources. CTI provides detailed analyses that aid in preventing and responding to cyberattacks by identifying trends, patterns, and relationships across multiple data sources. This intelligence enables security teams to take targeted, data-driven actions to enhance their defense posture effectively [ 23 ] . CTI supports a variety of activities, including Adversary Emulation and Red Teaming , Threat Hunting , Risk Assessment , and Incident Response , making it a cornerstone of modern cybersecurity strategies. The information collected from CTI sources can be structured using frameworks like MITRE ATT&CK [ 24 ] , a knowledge base designed to understand cyber threats by analyzing cybercriminal behaviors. MITRE ATT&CK provides a standardized language that facilitates information sharing among security teams. However, automatically transforming unstructured CTI sources, written in natural language, into a structured format for security processes remains a significant challenge. Some studies have attempted to overcome this limitation, by extracting structured CTI from unstructured sources with NLP techniques, and more recently LLMs [ 25 ] [ 2 ] [ 5 ] . Despite their potential, several challenges persist. A key issue is the tendency to misinterpret benign sentences as anomalous, incorrectly identifying TTPs (Tactics, Techniques, and Procedures) that are not actually present. Another significant challenge is the phenomenon of “ hallucinations ”, where models fabricate nonexistent facts or produce inappropriate information in an attempt to generate a response [ 26 ] . Most studies use CTI datasets to evaluate the overall performance of AI-based CTI analyzers and to fine-tune them. However, these datasets often have significant limitations, including limited availability, the use of document-level rather than statement-level granularity, which results in a loss of correspondence between individual sentences and the identified TTPs, and reliance on short and generic TTP descriptions from the MITRE ATT&CK Knowledge Base [ 24 ] rather than detailed CTI reports written by cybersecurity analysts. In this work, we address these challenges by conducting a detailed review of existing datasets and introducing CTI-HAL ( CTI Human-Annotated Labels ), a new CTI dataset 1 1 1 https://github.com/dessertlab/CTI-HAL , well-suited for the evaluation of NLP techniques. Our dataset includes reports of various sizes, with annotations on specific statements, allowing the assessment of a model’s ability to identify only the most relevant parts. Furthermore, it ensures traceability, facilitates cross-referencing , and maintains high-quality standards through multiple annotators and cross-validation . As a result, this new CTI dataset represents a versatile tool for research and applications in cybersecurity. We applied our dataset in a real-world business context, using it to evaluate the performance of an automation flow, based on an LLM, to extract TTPs from unstructured commercial CTI feeds. The lessons learned from this application are as follows: • Datasets have a significant impact on the accuracy of LLM under study. Compared to previous studies, our analysis found large differences, depending on the size and topics of CTI reports. • The evaluated LLM struggles more with analyzing large CTI reports, whereas when analyzing concise reports, it achieves significantly better performance. • When applying the LLM to commercial CTI feeds, we obtain results comparable to those achieved with our dataset, demonstrating promising generalizability. This paper is structured as follows. Section 2 reviews related work, providing an overview of existing approaches and their limitations. Section 3 describes the approach used to create the dataset, including a classification summary of the collected data. Section 4 focuses on the quality assessment of the dataset, while Section 5 presents the application of the dataset in a real-world business context along with the evaluation results. Finally, Section 6 concludes the paper with a discussion of findings and future research directions. Figure 1: Workflow 2 Related