Agent Security is a Systems Problem
Mihai Christodorescu, Earlence Fernandes, Ashish Hooda, Somesh Jha, Johann Rehberger, Kamalika Chaudhuri, Xiaohan Fu, Khawaja Shams, Guy Amir, Jihye Choi, Sarthak Choudhary, Nils Palumbo, Andrey Labunets, Nishit V. Pandya · 2026
Read Agent Security is a Systems Problem by Mihai Christodorescu, Earlence Fernandes, Ashish Hooda, Somesh Jha, Johann Rehberger, Kamalika Chaudhuri, Xiaohan Fu, Khawaja Shams, Guy Amir, Jihye Choi, Sarthak Choudhary, Nils Palumbo, Andrey Labunets, Nishit V. Pandya (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18991.txt.
Introduction
Mihai Christodorescu and thirteen colleagues argue that agent security is a systems problem, not a model problem. The language model driving an agent should be treated as an untrusted component, with security invariants enforced at the system level. Efforts to harden the model alone, the dominant view, cannot suffice. Drawing on decades of systems security research, the authors set out core design principles with predictable guarantees. They test the principles against eleven real-world attacks on agents and name the research challenges blocking implementation.
Catalog record
| CREATOR | Mihai Christodorescu, Earlence Fernandes, Ashish Hooda, Somesh Jha, Johann Rehberger, Kamalika Chaudhuri, Xiaohan Fu, Khawaja Shams, Guy Amir, Jihye Choi, Sarthak Choudhary, Nils Palumbo, Andrey Labunets, Nishit V. Pandya |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56011 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18991
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Agent Security is a Systems Problem Mihai Christodorescu, Earlence Fernandes, Ashish Hooda, Somesh Jha, Johann Rehberger, Kamalika Chaudhuri, Xiaohan Fu, Khawaja Shams, Guy Amir, Jihye Choi, Sarthak Choudhary, Nils Palumbo, Andrey Labunets, Nishit V. Pandya Abstract We take the position that agent security must be approached as a systems problem: the AI model powering the agent must be treated as an untrusted component, and security invariants must be enforced at the system level. Through this lens, efforts to increase model robustness (the dominant viewpoint in the community) are insufficient on their own. Instead, we must complement existing efforts with techniques from the systems security domain. Based on our experience as cybersecurity researchers in operating systems, networks, formal methods, and adversarial machine learning, we articulate a set of core principles, grounded in decades of systems security research, that provide a foundation for designing agentic systems with predictable guarantees. As evidence, we analyze eleven representative real-world attacks on agents and discuss how systems principles, if realized, could have prevented these attacks. We also identify the research challenges that stand in the way of implementing these principles in agents. 1 Introduction Position. Agent security must be approached as a systems problem: the AI model powering the agent must be treated as an untrusted component, and security invariants must be enforced at the encompassing system level. Building upon security principles grounded in decades of research, one can identify the three main gaps that need to be bridged to improve the secure deployment of agentic systems in the wild. Agentic computing is an emerging paradigm where AI models employ various tools to complete tasks that users specify in natural language. These systems operate in complex environments, such as websites, applications, and external services, and are exposed to adversarial inputs which can influence their behavior. We consider a threat model where an attacker controls some portion of the agent’s input channels (e.g., documents, tool outputs, etc.) with the goal of inducing unintended actions or exfiltrating sensitive data. For example, an agent may be prompt injected by malicious external content (e.g., a webpage) and coerced into misusing its access to user data [ 64 , 27 , 69 , 52 , 44 , 46 , 48 , 49 , 51 , 53 ] . More broadly, the AI model powering the agent itself is an unreliable component that may behave unpredictably under adversarial or ambiguous inputs. A growing body of work has emerged to address these risks. These approaches treat the model as the primary object of security and aim to make it directly robust to a wide range of adversarial inputs, using alignment techniques. We argue that this approach is fundamentally limited: (1) it was attempted in the days of “classical adversarial ML” that dealt with computer vision models and it was repeatedly shown that attackers can evade model-based defenses [ 5 ] ; (2) a similar phenomenon has been observed in the current LLM era, where attackers always appear to be successful, without a substantial increase in attacker effort [ 36 , 42 ] . Furthermore, modern agentic systems are significantly more complex. Unlike vision classifiers, LLMs work as part of a larger system (i.e., the agent scaffolding, environment, and tools) and this represents a valuable opportunity to incorporate systems security techniques. Similar to how an operating system treats a process as untrusted, we take the stance that the model powering the agent should be treated as untrusted and security properties should be expressed and enforced outside, at the level of the encompassing system. The computer security community has long studied how to build secure systems from unreliable components [ 58 ] . Examples of core principles include the paradigms that software should have the minimum privilege necessary to complete its task, it should obey a pre-defined control and information flow , and it should sanitize untrusted data to avoid being tricked into executing malicious instructions [ 58 ] . Today, all major systems implement a subset of these ideas using a variety of mechanisms. Our goal is to distill these principles and mechanisms for the AI security community and to highlight where challenges arise when they are applied to AI agents. Contributions. We make three main contributions. First, based on our collective experience as cybersecurity researchers working (for multiple decades) on operating systems, network systems, cyber-physical systems, formal methods, programming languages, and adversarial machine learning, we distill core principles that we believe are fundamental for agentic security. Second, we analyze eleven recent and representative real-world attacks on agentic systems and demonstrate how these attacks can be understood, and in many cases mitigated, through the lens of concrete cybersecurity principles. Third, we operationalize these principles and discuss concrete security mechanisms that implement them. We believe that if these mechanisms are used when creating agents, a large swath of attacks will be prevented by design. However, as our analysis shows, there are research challenges that arise when applying these security mechanisms to agents. We highlight these challenges, which we suggest the community should focus on. Overview of system-level mechanisms that mitigate a broad range of agent attacks. (1) Provable separation of instructions and data. Language-based models inherently mix (trusted) instructions and (untrusted) data, creating a fundamental vulnerability whenever malicious data can be interpreted as instructions in the context window. The security community has addressed analogous problems in other systems for decades by enforcing separation of instructions and data at each system layer. Notable examples include the use of no-execute (NX) bits in CPUs and the use of prepared statements and tainting of user-derived bytes to defend against SQL injection in database systems [ 43 , 37 ] . In agentic systems, the challenges are twofold: first, LLMs combine trusted and untrusted components into a single stream of tokens and process it together, that is, there is no source-level separation between input and data. Second, while it is often desirable for an agent to follow new instructions (once they are separated from data), there should be a deterministic mechanism to decide when it is safe to do so. Even when instructions and data are explicitly delimited [ 10 , 11 , 8 ] , persistent attackers have consistently been able to evade the enforcement of separation [ 36 , 42 , 26 ] . Defenses that aim to detect malicious input by identifying instructions embedded in data [ 30 ] have likewise failed to be robust to adaptive attacks [ 13 ] . We conjecture that model-based defenses for separating instructions and data will always be evaded by persistent attackers and apply the classic security concept of defense-in-depth via the next two primitives. (2) Least-privilege sandboxing with verifiable policy generation. Sandboxing untrusted components is a central principle in systems security, aiming at limiting the scope and impact of a potential attack. Notable examples include isolation and permission mechanisms in operating systems, smartphones and browsers. Each of these restrict access to sensitive resources using least-privilege policies typically specified by a trusted authority (e.g., system administrators or app developers). However, unlike traditional computer programs, agents do not compute fixed tasks. Rather, they can complete different tasks based on the conversation with the user. Therefore, the policy is specified in