LLM Agents Should Employ Security Principles

Kaiyuan Zhang, Zian Su, Pin-Yu Chen † , Elisa Bertino, Xiangyu Zhang, Ninghui Li Purdue University, † IBM Research {zhan4057, su284, bertino, xyzhang, ninghui}@purdue.edu , † [email protected] · 2025

Read LLM Agents Should Employ Security Principles by Kaiyuan Zhang, Zian Su, Pin-Yu Chen † , Elisa Bertino, Xiangyu Zhang, Ninghui Li Purdue University, † IBM Research {zhan4057, su284, bertino, xyzhang, ninghui}@purdue.edu , † [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24019.txt.

Introduction

Kaiyuan Zhang and five co-authors argue that AI agents should be built on classic security principles. Their paper is from two thousand twenty-five. Agents that use large language models can do complex tasks, but talks between agents open new holes. Trick prompts and warped context can leak private data or hijack the system.

Old design rules from information security can help: defence in depth, least privilege, full checks on each act, and ease for users. They set out a frame, AgentSandbox, that weaves these rules through an agent's whole life. Tests with top models show high use value in calm and hostile runs, with far less privacy risk. The frame aims to ground trust in agent systems.

Catalog record

CREATORKaiyuan Zhang, Zian Su, Pin-Yu Chen † , Elisa Bertino, Xiangyu Zhang, Ninghui Li Purdue University, † IBM Research {zhan4057, su284, bertino, xyzhang, ninghui}@purdue.edu , † [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24019

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM Agents Should Employ Security Principles Kaiyuan Zhang, Zian Su, Pin-Yu Chen † , Elisa Bertino, Xiangyu Zhang, Ninghui Li Purdue University, † IBM Research {zhan4057, su284, bertino, xyzhang, ninghui}@purdue.edu , † [email protected] Abstract Large Language Model (LLM) agents show considerable promise for automating complex tasks using contextual reasoning; however, interactions involving multiple agents and the system’s susceptibility to prompt injection and other forms of context manipulation introduce new vulnerabilities related to privacy leakage and system exploitation. This position paper argues that the well-established design principles in information security, which are commonly referred to as security principles , should be employed when deploying LLM agents at scale. Design principles such as defense-in-depth, least privilege, complete mediation, and psychological acceptability have helped guide the design of mechanisms for securing information systems over the last five decades, and we argue that their explicit and conscientious adoption will help secure agentic systems. To illustrate this approach, we introduce AgentSandbox, a conceptual framework embedding these security principles to provide safeguards throughout an agent’s life - cycle. We evaluate with state-of-the-art LLMs along three dimensions: benign utility, attack utility, and attack success rate. AgentSandbox maintains high utility for its intended functions under both benign and adversarial evaluations while substantially mitigating privacy risks. By embedding secure design principles as foundational elements within emerging LLM agent protocols, we aim to promote trustworthy agent ecosystems aligned with user privacy expectations and evolving regulatory requirements. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities in natural language processing and generation [ 46 , 47 , 13 , 40 , 57 ] . In the meantime, LLM agents, equipped with planning, reasoning, and acting abilities, are increasingly deployed in real-world applications where they communicate with humans and other agents via natural language. Early demonstrations such as ChatArena [ 63 ] , WebArena [ 79 ] , and OSWorld [ 64 ] reveal that LLM agents can decompose tasks and share knowledge effectively. Recent studies [ 9 , 77 , 58 , 74 ] reveal critical vulnerabilities in LLM agents. The inherent complexities of LLM reasoning and the documented failure of current security measures create opportunities for adversaries to exploit unforeseen weaknesses. For instance, attackers can poison an agent’s memory or knowledge base [ 15 ] or introduce malicious tools [ 19 ] . This is further evidenced by findings that even advanced LLMs fail prompt injection defenses approximately 85% of the time [ 74 ] , while other mitigation techniques also offer limited protection [ 27 , 3 ] , including paraphrasing [ 27 ] , access restriction [ 9 , 1 ] , tool filtering [ 62 ] , data delimiters [ 25 ] , prompt injection detection [ 44 ] , and perplexity based detection [ 3 ] . Furthermore, LLM agents are susceptible to carefully crafted contextual manipulations that induce the disclosure of sensitive information beyond authorized boundaries [ 9 , 58 ] , a risk heightened when agents operate with overly broad access to data. Attackers can also silently steer agent reasoning towards unauthorized actions, leading to privacy leakage [ 51 ] and destructive operations [ 24 ] , thereby exposing the lack of continuous and comprehensive verification of agent activities. These vulnerabilities are alarming as assistants based on LLMs increasingly manage personal finance [ 68 ] , travel planning [ 1 ] , and medical advising [ 45 ] , and orchestrate critical business workflows like customer support [ 18 ] and cloud services [ 4 ] . At the same time, emerging standards for LLM agents, such as the Model Context Protocol (MCP) [ 7 ] and Agent2Agent (A2A) [ 23 ] , primarily address low-level security features (e.g., authentication, network transport, and authorization), while dedicating less on threats such as blind instruction following, prompt hacking, and contextual manipulation. History is the best teacher for security. One root cause of software vulnerabilities is that the Von Neumann architecture of digital computers stores both code and data in the same memory space, potentially allowing programs to inadvertently or maliciously modify themselves or each other. Similar code-data mixup issues caused web security challenges such as various injection attacks. In the LLM era, the distinction between code and data is further blurred, as text will drive the reasoning and planning of LLM agents. To enhance security and privacy of the LLM ecosystem, we argue that the community should conscientiously apply the well-established security principles when deploying LLM agents at scale . Saltzer and Schroeder in their landmark 1975 paper titled “The Protection of Information in Computer Systems” [ 49 ] introduced eight design principles for secure systems, including, among others, least privilege , complete mediation , and psychological acceptability . Over the decades these principles have become staples of information security education, research, and practice. A few additional principles have also emerged since then, defense-in-depth being the most prominent among them. These principles have guided the systems security community for decades and demonstrated their effectiveness for securing emerging infrastructure such as the Internet, the WWW, mobile apps, and so on. We expect that they would continue to help us in the LLM era. To illustrate how these security principles help bridge the gap in LLM agents and security, we propose a security framework called AgentSandbox, which applies these principles directly into the fabric of future agent communication protocols. • Defense-in-Depth . Due to the lack of understanding in LLM reasoning and that no current security measure can offer any formal guarantee, it is necessary to deploy multiple layers of defense, mutually reinforcing each other to minimize potential damage if a breach occurs. AgentSandbox has multiple components that complement each other to offer defense-in-depth. One key idea of AgentSandbox is to separate a persistent agent that maintains long - term user profile from ephemeral agents , which are created for the tasks and discarded at completion, and can be isolated for better security. • Least Privilege . The ephemeral agent can be provisioned with the least amount of information and privileges necessary for performing the task. We design a data minimizer that derives the minimal context necessary for task success and a reward modeling policy engine that governs information flows and dynamically generate policies. By constraining every request to the minimal disclosure set, the system reduces the attack surface and complies with the principle that a subject should be granted only the rights it requires. • Complete Mediation . To ensure that every access to a resource is verified before it’s granted, we examine all outbound or inbound messages through data minimizer, response filter and I/O firewall , which enforces schema validation and policy checks on every access, not merely the initial one. • Psychological Acceptability . Psychological acceptability emphasizes that security mechanisms should not significantly increase user difficulty or inconvenience when accessing resources or performing actions. To reduce user tuning efforts while achieving the necessary flexibility for practical and secure agent operations, AgentSandbox employs a reward modeling policy engine that automates the policy generation by optimizing a reward function balancing utility-security. Roadmap. In Section 2 , we discuss the problem setup, threat model and challenges. In Section 3 , we outline our proposed framework and present

Read the complete text