Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs
Anirudh Sekar, Mrinal Agarwal, Rachel Sharma, Akitsugu Tanaka, Jasmine Zhang, Kevin Zhu · 2026
Read Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs by Anirudh Sekar, Mrinal Agarwal, Rachel Sharma, Akitsugu Tanaka, Jasmine Zhang, Kevin Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12359.txt.
Introduction
Anirudh Sekar and five co authors look at a weak spot in large language models. Bad prompts can slip in through side paths such as email or user content. Once inside, they can push a model past its safety rules. The model may then give harmful or wrong output that its makers did not intend.
Their guard watches for drift in embeddings, with no need for past shots of an attack. The name of the method stresses that it is light and cheap to run. The team notes that even the best models stay open to such prompts. Fixes tied to one model are slow and narrow, so a broad guard has clear worth.
Catalog record
| CREATOR | Anirudh Sekar, Mrinal Agarwal, Rachel Sharma, Akitsugu Tanaka, Jasmine Zhang, Kevin Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-12359
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs Anirudh Sekar, Mrinal Agarwal, Rachel Sharma, Akitsugu Tanaka, Jasmine Zhang, Kevin Zhu Abstract Prompt injection attacks have become an increasing vulnerability for LLM applications, where adversarial prompts exploit indirect input channels such as emails or user-generated content to circumvent alignment safeguards and induce harmful or unintended outputs. Despite advances in alignment, even state-of-the-art LLMs remain broadly vulnerable to adversarial prompts, underscoring the urgent need for robust, productive, and generalizable detection mechanisms beyond inefficient, model-specific patches. In this work, we propose Zero-Shot Embedding Drift Detection (ZEDD) , a lightweight, low-engineering-overhead framework that identifies both direct and indirect prompt injection attempts by quantifying semantic shifts in embedding space between benign and suspect inputs. ZEDD operates without requiring access to model internals, prior knowledge of attack types, or task-specific retraining, enabling efficient zero-shot deployment across diverse LLM architectures. Our method uses adversarial-clean prompt pairs and measures embedding drift via cosine similarity , to capture subtle adversarial manipulations inherent to real-world injection attacks. To ensure robust evaluation, we assemble and re-annotate the comprehensive LLMail-Inject ddataset spanning five injection categories derived from publicly available sources. Extensive experiments demonstrate that embedding drift is a robust and transferable signal, outperforming traditional methods in detection accuracy and operational efficiency. With greater than 93% accuracy in classifying prompt injections across model architectures like Llama 3, Qwen 2, and Mistral with a false positive rate of ¡3% , our approach offers a lightweight, scalable defense layer that integrates into existing LLM pipelines, addressing a critical gap in securing LLM-powered systems to withstand adaptive adversarial threats. † † footnotetext: All code utilized for this project is disclosed at: https://github.com/AnirudhSekar/ZEDD/blob/main/Zero_Shot_Embedding_Drift_Detection_A_Lightweight_Defense_Against_Prompt_Injections_in_LLMs.ipynb 1 Introduction and Related Works Large Language Models (LLMs) have rapidly become central to a wide range of applications, from conversational AI and content generation to software development and research assistance [ 22 ] . However, the growing reliance on these systems has brought to light significant security concerns, particularly the threat of prompt injection attacks [ 19 ] . These attacks involve creating inputs that manipulate an LLM into bypassing its alignment safeguards, leading to the generation of harmful, misleading, or policy-violating outputs [ 16 ] . While significant progress has been made in aligning LLMs to avoid overtly dangerous behaviors through reinforcement learning from human feedback (RLHF) and other fine-tuning techniques, these models remain vulnerable to adversarial prompting [ 17 , 5 ] . Recent research has shown that both manual and automated prompt-based attacks can consistently induce even the most advanced commercial models to produce objectionable content, including instructions for illegal activities, disinformation, and hate speech [ 10 ] . In particular, adversarial prompts generated through gradient-based optimization methods have shown high success rates in evading existing safety measures, often transferring between different models and architectures, as shown by [ 6 , 12 ] . However, despite growing awareness of prompt injection risks, most existing defenses remain limited in their effectiveness or practicality [ 2 , 23 , 21 ] . Embedding drift has been explored, but these approaches utilize optimizations via methods such as Logistic Regression, XGBoost, and Random Forests rather than fine-tuning the LLMs embedding space to produce optimized classifications [ 3 ] . Some different approaches have been explored, but many of these approaches are not lightweight [ 19 , 20 ] , introducing non-trivial computational and latency overhead that hinders scalable deployment in latency sensitive applications, as discussed by [ 18 ] . 2 Our Contributions Current approaches to detecting both direct and indirect prompt injections (IPI) rely on additional large models and rule-based filters to classify injections at a high level, which create heavy computational and integration overhead [ 11 , 7 , 27 ] . In this work, we introduce a simple yet effective defense mechanism: Zero-Shot Embedding Drift Detection (ZEDD) . Our key insight is that adversarial prompts subtly shift the semantic representation of inputs in the embedding space, even when the surface text appears clean, allowing for a quicker and more lightweight analysis of prompts while maintaining accuracy. By measuring the drift, or the change in vector embeddings between clean prompts and candidate prompts, we can detect injection attempts in an extremely lightweight manner. Our method is efficient, model-agnostic, and compatible with both open-source embedding models and commercial APIs. These characteristics eliminate the need for model retraining, internal model access, or prior knowledge of specific attack patterns. Our contributions are as follows: 1. A zero-shot, prompt injection detection method based on embedding drift, requiring no retraining, model access, or prior knowledge of attack types. 2. A flagging method utilizing Gaussian Mixture Modeling (GMM) and Kernel Density Estimation (KDE) to analyze the distribution of embeddings to adequately flag injected prompts while minimizing false positives. 3. A comprehensive empirical evaluation showing that embedding drift serves as a signal for prompt injection across diverse LLM architectures, outperforming many traditional methods in speed while maintaining high accuracy. Ultimately, this work aims to enhance prompt injection defenses by introducing a lightweight, training-free detection layer that efficiently integrates into existing LLM pipelines with minimal engineering overhead. 3 Threat Model Attackers’ Goals : The attacker seeks to inject adversarial instructions into email content processed by an LLM-integrated email assistant. The objectives map to common semantic manipulation patterns: 1. Jailbreak : Bypass safety mechanisms via role-play, hypothetical scenarios, or implicit persona adoption. 2. System leak : Extract system prompts, configuration details, or internal model parameters through seemingly innocent email queries 3. Task override : Redirect the assistant from its intended task to perform unauthorized actions. 4. Encoding Manipulation : Use special characters, formatting tricks, or obfuscated language to evade detection while preserving malicious intent. 5. Prompt confusion : Introduce convoluted, multi-step instructions designed to mislead the model’s instruction-following process. Because LLMs often operate on top of semi-structured input such as user messages or system templates, they are vulnerable to prompt injection, where adversarial content is placed within inputs in a way that manipulates the behavior of the LLM [ 8 ] . Attackers’ Knowledge : We assume that the attacker has access to public or inferable information about the target LLM-integrated application. This includes knowledge of how email content is formatted and incorporated into prompts, how user-facing summaries or responses are generated, and the general behavior of the underlying LLM (via documentation, reverse engineering, or trial interactions) as a whole. Additionally, attackers have access to public prompt injection techniques and methodologies, including those potentially documented in data sets such as LLMail-Inject. In line with the constructions of prompt injection attacks, we assume no access to private model weights or the internal application architecture, but only to the same interfaces available to a standard external user. Attackers’ Capabilities : The attacker’s capabilities are limited to the email medium, specifically the ability to craft and send malicious email content that will be processed by the LLM-integrated assistant. They can manipulate email structure, metadata, and content to embed adversarial instructions, and perform iterative refinement on attack strategies based on observable system responses. This reflects indirect prompt injection; the attacker relies on the host application (e.g., the email assistant) [ 26 ] , to automatically retrieve and concatenate email content into