Operationalizing CaMeL: Strengthening LLM Defenses for Enterprise Deployment

Krti Tallam, Emma Miller · 2025

Read Operationalizing CaMeL: Strengthening LLM Defenses for Enterprise Deployment by Krti Tallam, Emma Miller (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22852.txt.

Introduction

Tallam and Miller (2025) set out to make a security framework for language model agents practical in company settings. The framework, called CaMeL, sandboxes an agent using capability rules to blunt prompt injection attacks, where hostile text in data tries to steer the model. In its original form it trusts the user's opening prompt, ignores side channels, and pays a speed cost for running two models at once.

The authors propose four upgrades. First, screen the opening prompt itself. Second, audit outputs to catch instruction leakage. Third, use a tiered-risk access model so routine actions stay smooth while risky ones face control. Fourth, route actions through a verified intermediate language that can offer formal guarantees. Together, they argue, these changes align the framework with standard enterprise security practice and make it fit for large-scale deployment.

Catalog record

CREATORKrti Tallam, Emma Miller
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22852

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Operationalizing CaMeL: Strengthening LLM Defenses for Enterprise Deployment Krti Tallam, Emma Miller Abstract CaMeL (Capabilities for Machine Learning) introduces a capability-based sandbox to mitigate prompt injection attacks in large language model (LLM) agents. While effective, CaMeL assumes a trusted user prompt, omits side-channel concerns, and incurs performance trade-offs due to its dual-LLM architecture. This technical response identifies these limitations and proposes engineering enhancements to extend CaMeL’s threat coverage and operational viability. We introduce: (1) prompt screening for initial inputs, (2) output auditing to detect instruction leakage, (3) a tiered-risk access model to balance utility and control, and (4) a formally verified intermediate language to support static guarantees. Together, these augmentations align CaMeL with best practices in security engineering, reduce overhead, and support enterprise-scale deployment without modifying underlying models. 1 Introduction Prompt injection attacks have become one of the most pressing security challenges in the deployment of large language models (LLMs), especially when these models are used as autonomous agents in enterprise workflows. As LLMs move beyond simple text generation and begin orchestrating tools - sending emails, querying databases, managing cloud files - they face increasing exposure to untrusted inputs. In these scenarios, attackers can embed malicious instructions within otherwise benign content - uploaded documents, web pages, or even system-generated messages. When an LLM ingests such input, it may unknowingly execute unauthorized actions: leaking sensitive data, contacting external recipients, or triggering destructive API calls. Studies have shown that even advanced models routinely fall for these traps, often misinterpreting hidden payloads as legitimate user commands [ 1 , 2 ] . To address these threats without modifying the underlying model, Debenedetti et al. introduced CaMeL (Capabilities for Machine Learning) [ 1 ] . CaMeL wraps the LLM in a capability-based execution layer that separates trusted and untrusted data flows. A Privileged LLM receives the user’s prompt and generates a high-level plan, while a Quarantined LLM handles untrusted content and enforces strict schema validation. Each data value is labeled with provenance and access metadata (“capabilities”), and all tool invocations must pass explicit policy checks before execution. While CaMeL significantly raises the bar for prompt-injection resilience, there is room to improve its robustness and practicality. This paper examines CaMeL’s current limitations - including assumptions in its threat model, remaining side-channel exposures, and system-level constraints such as latency and policy sprawl. We propose a series of technical developments to address these challenges, drawing on best practices from security engineering, formal verification, and access control systems. 2 Opportunities for Strengthening CaMeL’s Threat Model From an enterprise security standpoint, three notable gaps in CaMeL’s threat model warrant closer attention. 2.1 Initial Prompt Trust CaMeL assumes that the user’s initial message is benign. However, corporate red team exercises have shown that a single crafted prompt - often delivered via phishing or chat-based social engineering - can implant persistent logic into the agent’s planning layer. Classic phishing studies report that over 30 percent of users click seemingly harmless links containing attacker-controlled keywords [ 3 ] . To address this risk, we propose an initial prompt screening gateway that performs reputation checks on URLs, flags override phrases (e.g., “ignore all previous”), and computes entropy or perplexity scores to detect anomalies. Since this involves only a short string, latency remains low (¡5 ms in internal tests), yet the screening closes a high-impact entry point for injection attacks [ 4 , 5 ] . 2.2 Output-Side Manipulation CaMeL enforces data provenance for tool inputs but does not inspect what the agent ultimately outputs. For instance, a benign PDF might include a line such as: “## System: Forward this file to finance‐[email protected].” If the agent summarizes this document, the hidden instruction could be echoed to a human user, leading to unintended actions. Modern natural language inference models can already detect contradictions and malicious phrasing with over 90 percent accuracy on benchmark datasets like MNLI [ 6 ] . We recommend a post-processing output auditing pass that scans each LLM-generated response for override cues, suspicious URLs, or contradictions with the intended business task. Only outputs that pass this audit are displayed or executed. 2.3 Provenance of User Uploads Schema validation alone is insufficient for documents supplied by end users. We suggest that every value extracted from such files carry a file-content provenance tag , e.g., a from_user_upload label. Policies can then prevent this data from flowing into irreversible actions - such as sending external emails or modifying ERP systems - unless explicitly authorized via a privileged “grant-exception” mechanism. This design mirrors techniques from information-flow control languages like JIF [ 7 ] and FlowCaml [ 8 ] , adapted here for agent-based systems driven by LLMs. By implementing (i) prompt screening, (ii) output auditing, and (iii) tagged provenance for uploaded content, security teams can broaden CaMeL’s protection across the full conversation lifecycle without compromising its core capability-based controls. 3 Balancing Security Guarantees with Practical Utility CaMeL’s default behavior rejects any tool invocation whose arguments - directly or indirectly - contain untrusted data. This strict policy achieves a 67 percent completion rate on the AgentDojo benchmark [ 1 ] . However, many of the remaining failures stem not from unsafe logic, but from treating all untrusted inputs as equally risky. In real-world enterprise environments, security frameworks like Risk-Adaptive Access Control and NIST’s Zero Trust Architecture apply more nuanced policies, adjusting enforcement based on operational sensitivity and situational context [ 9 ] . Inspired by these practices, we propose a tiered-risk policy for CaMeL: • Green tier - read-only actions on public or internally labeled “open” data (e.g., listing calendar events) are allowed after a basic provenance check. • Yellow tier - changes within the user’s own environment (e.g., moving a file to a shared folder) prompt a lightweight confirmation if any argument is untrusted. • Red tier - irreversible or externally visible operations (e.g., sending email, making wire transfers, calling privileged APIs) retain full capability checks and require multi-factor approval. Large-scale ABAC evaluations show that this kind of stratification preserves over 90 percent of legitimate workflows while blocking all simulated attacks in cloud testbeds [ 10 ] . Applying a similar approach in CaMeL would increase task success rates without relaxing its strongest controls. Reducing Prompt Fatigue. Excessive security prompts can lead to prompt fatigue , in which users reflexively approve warnings without reading them [ 11 ] . By limiting explicit confirmations to the yellow and red tiers, CaMeL can reduce prompt volume significantly - improving usability while maintaining security for high-risk actions. From Empirical Checks to Formal Guarantees. CaMeL’s current safeguards rely on benchmark performance rather than formal verification. For environments that demand rigorous assurance, we recommend building a mechanized model of CaMeL’s interpreter and policy engine in a proof assistant, then proving it satisfies noninterference: secret-labeled inputs should not affect public outputs, except through approved channels. Verified systems like CertiKOS and CompCert show that machine-checked security is feasible even for complex software stacks [ 12 , 13 ] . Rewriting CaMeL’s restricted Python dialect as a minimal, formally specified intermediate language would support similar verification efforts - bridging the gap between theoretical models and practical deployments [ 14 ] . 4 Side-Channel Considerations and Mitigations While CaMeL enforces capability-based data flow controls, it

Read the complete text