A Framework for Formalizing LLM Agent Security

Vincent Siu, Jingxuan He, Kyle Montgomery, Zhun Wang, [0.25em] Neil Gong, Chenguang Wang, Dawn Song, [0.5em] UC Santa Cruz · 2026

Read A Framework for Formalizing LLM Agent Security by Vincent Siu, Jingxuan He, Kyle Montgomery, Zhun Wang, [0.25em] Neil Gong, Chenguang Wang, Dawn Song, [0.5em] UC Santa Cruz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19469.txt.

Introduction

Siu and co-authors build a formal frame for security in language model agents. The same act can be fine or a breach, based on who asked and why. They state four properties: task fit, action fit, source rights, and data isolation. Oracle checks then test if a run breaks any property. Known attacks, from prompt injection to memory poisoning, are recast as breaches of one or more properties. The aim is to end the blunt trade of utility for safety.

Catalog record

CREATORVincent Siu, Jingxuan He, Kyle Montgomery, Zhun Wang, [0.25em] Neil Gong, Chenguang Wang, Dawn Song, [0.5em] UC Santa Cruz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76129
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19469

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Framework for Formalizing LLM Agent Security Vincent Siu, Jingxuan He, Kyle Montgomery, Zhun Wang, [0.25em] Neil Gong, Chenguang Wang, Dawn Song, [0.5em] UC Santa Cruz Abstract Security in LLM agents is inherently contextual . For example, the same action taken by an agent may represent legitimate behavior or a security violation depending on whose instruction led to the action, what objective is being pursued, and whether the action serves that objective. In this work, we present a framework that systematizes existing attacks and defenses from the perspective of contextual security. To this end, we propose four security properties that capture contextual security for LLM agents: task alignment (pursuing authorized objectives), action alignment (individual actions serving those objectives), source authorization (executing commands from authenticated sources), and data isolation (ensuring information flows respect privilege boundaries). We further introduce a set of oracle functions that enable verification of whether these security properties are violated as an agent executes a user task. Using this framework, we reformalize existing attacks, such as indirect prompt injection, direct prompt injection, jailbreak, task drift, and memory poisoning, as violations of one or more security properties, thereby providing precise and contextual definitions of these attacks. Similarly, we reformalize defenses as mechanisms that strengthen oracle functions or perform security property checks. Finally, we discuss several important future research directions enabled by our framework. 1 Introduction LLM agent security is a critical concern as agents increasingly operate in real-world environments [ 60 ] . Security in LLM agents is inherently contextual : whether an input - such as a prompt, observation of the environment, or memory record - or an output - such as an action - constitutes a security breach depends on the execution context. For example, as illustrated in the top row of Figure 1 , the same prompt may represent either a legitimate task or a security breach depending on the context. However, many existing definitions of security attacks fail to capture this contextual nature: namely, who issued the prompt, what objective is being pursued, whether the agent’s actions advance that objective, and what information flows are permitted. For example, Liu et al. [ 53 ] define a prompt injection attack as occurring when a data input - such as a user input, email, webpage, tool response, or document - contains a prompt, encompassing both indirect and direct prompt injection. This definition has subsequently been adopted in several LLM agent benchmarks on prompt injection [ 20 , 22 , 44 , 52 ] . However, this definition overlooks the agent’s execution context. In particular, a prompt embedded in a data input may be entirely legitimate and should be followed when it originates from an authorized source, such as the user, and aligns with the intended task objective. Figure 1 : Contextual security resolves the utility–security trade-off in LLM-agent systems. These examples illustrate that the same prompt may correspond to either an attack or a legitimate task depending on the execution context. In the absence of contextual considerations in attack definitions, prior detection methods primarily classify prompts as malicious or benign based on surface patterns, without accounting for how the agent is executing the task. This inevitably forces a trade-off: blocking patterns such as “delete file” improves security but prevents legitimate cleanup operations, while allowing them preserves utility but enables data-destruction attacks. Our framework formalizes context by checking four contextual security properties: Source Authorization , Task Alignment , Action Alignment , and Data Isolation . This enables the agent to distinguish legitimate file cleanup from data destruction even when the underlying prompts are identical, thereby achieving both security and utility. Due to the lack of contextual considerations in defining security attacks, current defenses face a fundamental utility-security trade-off: applying defenses uniformly across all contexts can lead to significant utility loss, whereas applying defenses in insufficient or inappropriate contexts can result in security vulnerabilities, as illustrated in the bottom left of Figure 1 . For example, detecting all data inputs that contain prompts as prompt injection attacks without considering the context inevitably leads to false positives, thereby reducing utility when such inputs originate from an authorized user and align with the intended task objective. Conversely, blindly following prompts embedded in any data input makes agents vulnerable; for instance, prompts injected by untrusted sources may steer the agent into performing arbitrary attacker-specified tasks. In this work, we develop a framework to systematize existing attacks and defenses in LLM agents from a contextual perspective. We formalize the execution context \mathcal{C}_{t} of an agent at time step t as a tuple capturing everything relevant to an authorization decision: who is trusted, what objective is authorized, what has happened so far, and what information flows are permitted. We then characterize contextual security through four properties that correspond directly to the four authorization-relevant components of \mathcal{C}_{t} : task alignment ensures the agent pursues objectives authorized by the user’s prompt; action alignment verifies that individual actions serve the stated task objectives; source authorization confirms that instructions originate from authenticated sources; and data isolation enforces that information flows respect permission boundaries. As illustrated in the bottom right of Figure 1 , evaluating all four properties enables fine-grained decisions that permit legitimate operations while blocking attacks, even when they involve identical content. A key challenge is how to verify, at runtime, whether violations of these security properties occur as an agent executes a user task. To address this challenge, our framework introduces a set of oracle functions . Similar to ideal functionalities in cryptographic protocols, these oracle functions define what information is theoretically required to verify security properties, even if perfect implementations may not exist in practice. Specifically, the instruction attribution function \mathcal{I} identifies which inputs contain the instructions that the agent followed to produce a given action; the source attribution function \mathcal{L} tracks the provenance of agent inputs; and the objective evaluation functions H_{p} , H_{Tr} , and H_{a} quantify task objectives at different levels. We then show that these oracle functions enable us to determine, at runtime, whether any of the four security properties are violated during task execution. If any such violation occurs, the execution constitutes a security breach. This formalization clarifies what contextual security fundamentally requires, enabling systematic evaluation of defenses and revealing research directions that scattered prior work obscures. Given the oracle functions and security properties, we use them to reformulate various attacks on LLM agents—such as indirect prompt injection, direct prompt injection, jailbreak, and others (see more details in Section 5 ). Specifically, we define these attacks in terms of violations of one or more of our security properties. This reformulation makes attack definitions precise and contextual, enabling the development of defenses that achieve both utility and security. For example, an indirect prompt injection attack occurs when an agent follows instructions from inputs originating from an unauthorized source (violating source authorization ) and the resulting action does not align with the intended user task objective (violating action alignment ). In contrast, a direct prompt injection attack occurs when an input from an authorized user contains a prompt that conflicts with the task objective specified by higher-privilege instructions, such as system prompts, thereby violating task alignment . These context-aware definitions of indirect

Read the complete text