On Automating Security Policies with Contemporary LLMs (Short Paper)
Pablo Fernández Saura 1 , K. R. Jayaram 2 , Vatche Isahagian 2 , Jorge Bernal Bernabé 1 , Antonio Skarmeta 1 · 2025
Read On Automating Security Policies with Contemporary LLMs (Short Paper) by Pablo Fernández Saura 1 , K. R. Jayaram 2 , Vatche Isahagian 2 , Jorge Bernal Bernabé 1 , Antonio Skarmeta 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04838.txt.
Introduction
Pablo Fernández Saura and co-authors use language models to automate security policy work in a 2025 short paper. Modern networks face threats that shift faster than hand-written rules can follow. Their framework stores tool and interface manuals in a searchable vector database. A language model then reads a high-level defense policy, splits it into tasks, and turns each task into concrete system calls.
Grounding the model in retrieved manuals, rather than its memory alone, lifted precision and quality scores sharply. Tests used public threat reports in a standard format plus Windows interface documents. The pipeline shows a path from a written policy to running defenses with little human glue work in between.
Catalog record
| CREATOR | Pablo Fernández Saura 1 , K. R. Jayaram 2 , Vatche Isahagian 2 , Jorge Bernal Bernabé 1 , Antonio Skarmeta 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-04838
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On Automating Security Policies with Contemporary LLMs (Short Paper) Pablo Fernández Saura 1 , K. R. Jayaram 2 , Vatche Isahagian 2 , Jorge Bernal Bernabé 1 , Antonio Skarmeta 1 Abstract The complexity of modern computing environments and the growing sophistication of cyber threats necessitate a more robust, adaptive, and automated approach to security enforcement. In this paper, we present a framework leveraging large language models (LLMs) for automating attack mitigation policy compliance through an innovative combination of in-context learning and retrieval-augmented generation (RAG). We begin by describing how our system collects and manages both tool and API specifications, storing them in a vector database to enable efficient retrieval of relevant information. We then detail the architectural pipeline that first decomposes high-level mitigation policies into discrete tasks and subsequently translates each task into a set of actionable API calls. Our empirical evaluation, conducted using publicly available CTI policies in STIXv2 format and Windows API documentation, demonstrates significant improvements in precision, recall, and F1-score when employing RAG compared to a non-RAG baseline. I Introduction The increasing sophistication of cyber threats has made security policy compliance a critical component of modern software systems, spanning standalone applications, distributed architectures, and cloud-hosted services. Security policies define the necessary actions and behaviors required to protect sensitive data and maintain system integrity through three primary mechanisms: prevention, detection, and mitigation of cyber attacks. However, the growing complexity of these policies, combined with the rapidly expanding landscape of enforcement tools and technologies, creates significant challenges for automated compliance monitoring and enforcement. Attack mitigation policies, which specify the actions required to neutralize active threats and prevent their escalation, represent a particularly challenging subset of security policies. These policies must translate high-level strategic directives into precise, executable system configurations and API calls. Traditional approaches to attack mitigation policy enforcement rely heavily on manual intervention, requiring security experts to interpret policy documents and manually configure diverse security tools. This manual process introduces several critical limitations: it is inherently error-prone due to human interpretation variability, slow to respond to rapidly evolving threats, and fails to scale effectively across large, heterogeneous computing environments. The emergence of Large Language Models (LLMs) with advanced natural language understanding and code generation capabilities presents a promising avenue for automating attack mitigation policy enforcement. These models, trained on extensive datasets encompassing both natural language and programming code, demonstrate the ability to interpret textual descriptions and generate corresponding API calls through tool-calling mechanisms. Recent advances in LLM capabilities suggest they could potentially bridge the gap between high-level policy descriptions and low-level executable actions, enabling automated translation of attack mitigation policies into actionable system commands. However, applying LLMs to attack mitigation policy automation presents significant technical challenges. First, the generated API calls must be both contextually accurate and precisely aligned with the intended policy objectives. Errors in policy-to-API translation could result in ineffective defenses or, worse, system misconfigurations that create new vulnerabilities. Second, attack mitigation policies often involve complex conditional logic, specialized cybersecurity terminology, and references to rapidly evolving threat landscapes. LLMs must accurately interpret these nuances while maintaining consistency across diverse tool ecosystems. Third, many organizational security tools and APIs may not be well-represented in LLM training data, particularly for custom or proprietary security infrastructure. To address these challenges, we propose a framework that leverages Retrieval-Augmented Generation (RAG) to enhance LLM-based attack mitigation policy automation. Our approach combines in-context learning with vector-based similarity search to provide LLMs with relevant tool and API documentation during the policy translation process. This method enables accurate interpretation of high-level mitigation strategies while ensuring generated API calls align with available system capabilities. This paper makes three key contributions. First, we present a systematic architecture for automated attack mitigation policy enforcement that decomposes high-level policies into discrete tasks and translates each task into executable API calls. Second, we demonstrate how RAG techniques can significantly improve the accuracy of LLM-generated API calls by providing contextually relevant tool documentation. Third, we provide empirical validation using publicly available Cyber Threat Intelligence (CTI) policies in STIXv2 format, showing an average 22 % point improvement in F1-scores when employing RAG compared to non-RAG baselines across multiple LLM architectures. The automation of attack mitigation policy enforcement has profound implications for cybersecurity resilience. By reducing response times from manual interpretation to automated execution, our approach can substantially decrease the window of vulnerability during active attacks. Furthermore, the systematic and consistent application of mitigation policies reduces human error and ensures uniform security postures across complex, distributed systems. This capability is particularly critical in environments where the speed and sophistication of attacks can overwhelm traditional manual defense mechanisms. II System Design We present a framework for automated attack mitigation policy enforcement that transforms high-level policy descriptions into executable API calls through a multi-stage pipeline. Figure 1 illustrates our system architecture, which integrates retrieval-augmented generation with dual-LLM processing to achieve accurate policy-to-API translation. The system addresses two fundamental challenges: (1) decomposing complex policy documents into discrete, actionable tasks, and (2) translating each task into contextually appropriate API calls using relevant tool documentation. II-A Tool and API Specification Management Our framework requires a comprehensive repository of available security tools and their corresponding API specifications. In this context, security tools encompass operating system utilities, security applications, network monitoring services, and cloud-based security platforms that can execute mitigation actions within the target environment. Each tool in the repository includes three essential components: a natural language description of the tool’s security capabilities, complete API documentation specifying function signatures and parameters, and usage examples demonstrating typical invocation patterns. The system maintains tool specifications in a structured format where each API function is documented with its purpose, required arguments, expected return values, and operational constraints. This documentation serves as the knowledge base for contextual retrieval during the API generation process. For organizational deployments, this repository can incorporate both public APIs from widely-used security tools and private APIs from custom or proprietary security infrastructure. II-B Vector-Based Knowledge Retrieval To enable efficient and contextually relevant API retrieval, we employ a vector database approach using semantic embeddings. The system processes API documentation through a structured pipeline: first, individual API specifications are loaded using LangChain’s DocumentLoader module, then segmented into coherent chunks via CharacterTextSplitter to optimize embedding quality. Each text chunk is processed through the all-mpnet-base-v2 embedding model from HuggingFace, generating dense vector representations that capture semantic relationships between API functions and their descriptions. These embeddings are stored in a Chroma Vector Store, enabling similarity-based retrieval of relevant API documentation. During policy processing, the system queries this vector database using task descriptions as search inputs, retrieving the K most semantically similar API specifications. This approach ensures that the LLM receives contextually relevant tool documentation without being overwhelmed by irrelevant API information, addressing the challenge of limited context window capacity in current LLM architectures. II-C Dual-LLM Processing Pipeline Fig. 1: Proposed system architecture Our system employs a two-stage LLM processing approach to handle the distinct challenges of policy decomposition and API generation. The first LLM (LLM1) specializes in policy analysis and task decomposition, while the second LLM (LLM2) focuses on API call generation with RAG-enhanced context. II-C 1 Policy Decomposition Stage LLM1 receives the input attack mitigation policy and decomposes it into a sequence