Uncovering Reliable Indicators: Improving IoC Extraction from Threat Reports

Evangelos Froudakis 1 , Athanasios Avgetidis 1 , Sean Tyler Frankum 2 , Roberto Perdisci 2 , Manos Antonakakis 1 , Angelos Keromytis 1 · 2025

Read Uncovering Reliable Indicators: Improving IoC Extraction from Threat Reports by Evangelos Froudakis 1 , Athanasios Avgetidis 1 , Sean Tyler Frankum 2 , Roberto Perdisci 2 , Manos Antonakakis 1 , Angelos Keromytis 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11325.txt.

Introduction

Evangelos Froudakis and co-authors tackle, in a 2025 paper, a weak base for tools that pull signs of hack attacks from threat reports. Such signs, called indicators of compromise, guide defense teams. Auto pull tools are judged against ground truth lists, yet those lists are flawed. Hand-made lists cost much labor; auto lists sweep in harmless items and swell false alarms. The team maps the flaws of each path, then builds a hybrid line: a language model sorts each candidate, and expert analysts check its calls with the context shown.

That line cut analyst work by 43 percent against full hand work in tests with six analysts. With it they built PRISM, a public bench set of 1,791 checked signs from 50 real threat reports. The set lets new tools be trained and judged on firm ground.

Catalog record

CREATOREvangelos Froudakis 1 , Athanasios Avgetidis 1 , Sean Tyler Frankum 2 , Roberto Perdisci 2 , Manos Antonakakis 1 , Angelos Keromytis 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-11325

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Uncovering Reliable Indicators: Improving IoC Extraction from Threat Reports Evangelos Froudakis 1 , Athanasios Avgetidis 1 , Sean Tyler Frankum 2 , Roberto Perdisci 2 , Manos Antonakakis 1 , Angelos Keromytis 1 Abstract Indicators of Compromise (IoCs) are critical for threat detection and response, marking malicious activity across networks and systems. Yet, the effectiveness of automated IoC extraction systems is fundamentally limited by one key issue: the lack of high-quality ground truth. Current extraction tools rely either on manually extracted ground truth, which is labor-intensive and costly, or on automated ground truth creation methods that include non-malicious artifacts, leading to inflated false positive (FP) rates and unreliable threat intelligence. In this work, we analyze the shortcomings of existing ground truth creation strategies and address them by introducing the first hybrid human-in-the-loop pipeline for IoC extraction, which combines a large language model–based classifier (LANCE) with expert analyst validation. Our system improves precision through explainable, context-aware labeling and reduces analysts’ work factor by 43% compared to manual annotation, as demonstrated in our evaluation with six analysts. Using this approach, we produce PRISM, a high-quality, publicly available benchmark of 1,791 labeled IoCs from 50 real-world threat reports. PRISM supports both fair evaluation and training of IoC extraction methods and enables reproducible research grounded in expert-validated indicators. I Introduction Detecting and analyzing malicious activity is essential in modern cybersecurity defense. Indicators of Compromise (IoCs), such as IP addresses, domain names, URLs, and file hashes, which signal malicious activity, are among the most important parts of threat intelligence [ 1 , 2 , 3 ] . Analysts rely on IoCs for both immediate defense, such as creating block lists and firewall rules, and for activities such as threat hunting. Accurate and timely identification of IoCs can therefore significantly improve an organization’s ability to detect, respond to, and mitigate cyber-threats. One of the most prominent sources of IoCs is threat reports (TRs), which are published by cybersecurity companies, research institutions, and government agencies, and contain critical information about attacks, campaigns, and incidents [ 4 , 2 , 5 , 1 ] . Threat reports vary widely in structure and detail due to the lack of standardized formatting. This inconsistency makes accurate IoC extraction challenging, as indicators may be embedded in diverse formats and contexts, yet their precise identification is critical for effective threat response. Previous work has focused on three general approaches for extracting IoCs from threat reports: manual labeling, automated extraction, and automated retrieval from threat intelligence sharing platforms. As noted repeatedly [ 6 , 1 , 7 , 8 ] , manual labeling is the most accurate approach to extract IoCs from unstructured threat reports. However, this requires significant domain knowledge and can be very time-consuming, making it expensive and difficult to scale [ 1 , 6 , 7 ] . Automated IoC extraction tools, including recent work that leverage large language models (LLMs) [ 9 , 10 , 11 ] , offer potential efficiency but often lack the necessary accuracy due to the complexity of natural language in technical cybersecurity reports, the variability in IoC presentation, and the difficulty in distinguishing true IoCs from similar-looking but benign artifacts (e.g., indicators such as benign domain names, IP addresses, and URLs that are not related to the threats described in the reports). Finally, while there exist efforts to distribute IoCs to the community in an easy-to-parse structured format, for instance via threat exchanges (e.g., AlienVault [ 12 ] ), threat report databases (e.g., orkl.eu ), and online security scanners (e.g., VirusTotal [ 13 , 14 , 15 , 16 ] ), we find that popular threat report IoC exchanges like AlienVault, often automatically extract IoCs from unstructured TRs with low precision ( < 80%), recall ( < 80%) and/or coverage. As a result, these approaches either incur high manual work factor costs or yield unreliable intelligence, underscoring the need for a new solution that is both accurate and efficient. To address these challenges, we propose the first IoC extraction approach that combines automated extraction supported by explainable AI with human-in-the-loop verification. Our central hypothesis is that large language models (LLMs), when guided by properly engineered prompts and paired with analyst feedback , can match or surpass the accuracy of manual-only labeling while significantly reducing the time cost of human effort. In our evaluation, we observe a 43% reduction in parsing time compared to manual-only workflows. Importantly, our pipeline design and prompt engineering strategies are model-agnostic and generalize across most state-of-the-art LLMs, enabling flexible deployment in diverse environments. By integrating human analysts in the IoC extraction process, we can leverage the speed of automation while ensuring high precision through human judgment. Similar “human-in-the-loop” (HITL) approaches have been successfully demonstrated in other domains where high-accuracy data labeling and expert oversight are critical, such as medical applications and some computer vision and natural language processing tasks [ 17 , 18 , 19 , 20 ] . To the best of our knowledge, we are the first to adapt the HITL approach for the efficient and accurate labeling of IoCs from unstructured reports. To this end, we develop a HITL system that provides: (1) visual assistance (via highlighted text) that allows analysts to quickly identify indicators of interest in unstructured (potentially long) threat reports; and (2) color-coded IoC labels assigned by an LLM, each accompanied by an explanation of the contextual cues used to justify the label – designed following principles of effective data visualization [ 21 , 22 , 23 ] . Figure 1 shows an overview of our system. It consists of LANCE and a human-in-the-loop (HITL) component where analysts, supported by LANCE and by a custom web-based user interface, assign labels to IoCs found in threat reports. Thanks to our system’s custom user interface and explainable LLM-based IoC labels, analysts can quickly confirm or correct the label recommendations throughout the report, thus boosting accuracy and efficiency. To evaluate our approach, we applied it to 50 real-world threat reports from reputable sources and observed that junior analysts made fewer labeling errors when assisted by our system. We then used the system to construct PRISM, a high-quality, manually validated dataset of 1,774 labeled IoCs, including domains, URLs, IPs, and file hashes. PRISM supports both training and evaluation of IoC extraction tools and serves as the first openly available benchmark for this task. In summary, the main contributions of this work are: • Human-in-the-Loop IoC Labeling System: The first human-in-the-loop system for IoC extraction and labeling from threat reports, which combines an LLM-based automated and explainable IoC labeling process with manual verification to minimize labeling costs and maximize accuracy. • LANCE: The LLM–based labeling component that provides explainable classifications of indicators from unstructured threat reports. This explainability allows for faster human validation, reducing annotation time by 43% compared to manual labeling. • Review of Existing IoC Extraction/Labeling Systems: We review and attempt to reproduce previous work on IoC extraction and labeling, including automated IoC extraction systems and threat exchanges. Our results show that existing available systems (specifically, systems whose code is available and reusable, or can be accessed via an API) either lack precision (i.e., high false positives) and/or recall (i.e., high false negatives). • PRISM: The first

Read the complete text