Multimodal Prompt Injection Attacks: Risks and Defenses for Modern LLMs
Andrew Yeo, Daeseon Choi · 2025
Read Multimodal Prompt Injection Attacks: Risks and Defenses for Modern LLMs by Andrew Yeo, Daeseon Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-05883.txt.
Introduction
Attacks that hide instructions inside text or images fed to a language model are tested in a 2025 paper by Andrew Yeo and Daeseon Choi. As companies build these models into consulting and search tools, prompt injection has become a leading security risk. An injected instruction can make a model leak data or ignore its owner's rules.
The authors attack eight commercial models with no added filtering, relying only on built-in safeguards. They sort attacks into four kinds: direct injection, indirect injection through external content, image-based injection, and prompt leakage. Every model showed exploitable weaknesses, though one system held up better than the rest. Their comparison argues that extra defenses, such as normalizing inputs before they reach the model, remain necessary for reliable protection.
Catalog record
| CREATOR | Andrew Yeo, Daeseon Choi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-05883
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Prompt Injection Attacks: Risks and Defenses for Modern LLMs Andrew Yeo, Daeseon Choi Abstract Large Language Models (LLMs) have seen rapid adoption in recent years, with industries increasingly relying on them to maintain a competitive advantage. These models excel at interpreting user instructions and generating human-like responses, leading to their integration across diverse domains, including consulting and information retrieval. However, their widespread deployment also introduces substantial security risks, most notably in the form of prompt injection and jailbreak attacks. To systematically evaluate LLM vulnerabilities—particularly to external prompt injection—we conducted a series of experiments on eight commercial models. Each model was tested without supplementary sanitization, relying solely on its built-in safeguards. The results exposed exploitable weaknesses and emphasized the need for stronger security measures. Four categories of attacks were examined: direct injection, indirect (external) injection, image-based injection, and prompt leakage. Comparative analysis indicated that Claude 3 demonstrated relatively greater robustness; nevertheless, empirical findings confirm that additional defenses, such as input normalization, remain necessary to achieve reliable protection. Keywords: Large Language Models, Prompt Injection, AI Security, Jailbreaking, Adversarial Attacks 1 Introduction Large Language Models (LLMs) are generative AI systems trained on massive datasets to understand and produce human-like text. Their ease of use and ability to deliver information faster than traditional search methods have fueled widespread adoption across industries. However, despite extensive training, LLMs remain vulnerable to exploitation. A key weakness lies in their tendency to prioritize the most recent instructions in the context window. While useful during training, this behavior makes them susceptible to manipulation once deployed. Critically, LLMs cannot inherently distinguish between system prompts (which define the model’s task) and user prompts (which query the model). Without robust mechanisms to separate these inputs, malicious actors can hijack model behavior [ 18 ] . Prompt injection exploits this vulnerability by manipulating inputs so that the LLM abandons its original instructions. Attackers most often pursue two objectives: instruction hijacking and data exfiltration. Instruction hijacking forces the model to generate responses outside of its intended scope, such as producing disallowed or misleading content. Data exfiltration, however, poses a more severe threat, as it seeks to extract proprietary or protected information stored or referenced within the system. This may include leaking system prompts, training data, or API keys, with potentially irreversible consequences. Once such information is exposed, it can be disseminated widely, enabling large-scale exploitation. In enterprise and healthcare contexts, exfiltration is particularly dangerous. An LLM leaking patient data, trade secrets, or compliance records could violate privacy laws such as HIPAA or GDPR, resulting in litigation and loss of user trust. Beyond legal implications, successful exfiltration undermines the technical credibility of LLMs, as it reveals structural weaknesses in how they separate user input from system-level instructions. Attackers may also chain exfiltration with other exploits, such as using stolen API keys for unlimited queries or leveraging leaked configuration details to access internal databases. Unlike instruction hijacking, which primarily alters outputs, exfiltration compromises confidentiality and integrity directly. Remediation after exfiltration is exceptionally difficult: keys must be rotated, prompts rewritten, and compromised datasets quarantined, yet leaked content cannot be fully removed from public circulation. For critical infrastructure, such as government agencies or defense contractors, the stakes are even higher, as adversarial groups could siphon classified data or technical schematics. Even in less sensitive domains, leaked training artifacts may expose customer behavior, trade secrets, or workflow processes, creating cascading vulnerabilities across supply chains. As LLMs become further integrated into enterprise and mission-critical systems, the risks associated with prompt injection—particularly data exfiltration—scale proportionally, underscoring the urgent need for proactive safeguards. Malicious Prompt Injection (Attacker) Compromised Hospital LLM System (via injection) Electronic Health Records Accessed (Patients’ PHI) Leakage of Sensitive Patient Data Attacker / Public Internet (untrusted recipient) – HIPAA Violations – Legal Fines – Loss of Patient Trust Figure 1: Illustrative data-exfiltration pathway from prompt injection in a hospital LLM workflow. As shown in Figure 1 , even small actions in AI-integrated systems can result in significant regulatory violations and loss of trust. This scenario underscores the need for continuous evaluation of LLM vulnerabilities. As new injection techniques emerge, even the most secure models remain at risk, reinforcing the importance of ongoing testing, monitoring, and defensive strategies. The present work highlights these vulnerabilities and introduces a structured framework for their categorization. Vulnerability Testing A series of experiments were conducted across multiple LLMs deployed on diverse infrastructures to systematically evaluate weaknesses associated with different injection methods. This analysis provided empirical evidence of susceptibility to adversarial manipulation. Classification The injection techniques explored in this study were organized into a structured classification framework. This taxonomy captures the shared objectives of attacks while ensuring reproducibility and enabling independent validation in future research. Application and Implications Building on the identified vulnerabilities, the study examined potential real-world risks in domains where LLMs are increasingly deployed, such as healthcare and enterprise systems. The findings point to critical implications, including regulatory violations (e.g., privacy breaches), erosion of user trust, and broader security threats in contexts where LLM reliability and compliance are essential. 2 Related Works Prompt injection has emerged as one of the most effective methods to hijack large language models (LLMs), presenting major security challenges. Prior research examines vulnerabilities, attack strategies, defenses, and real-world scenarios. This section reviews the literature in five categories: (1) LLM vulnerabilities and alignment, (2) direct prompt injection attacks, (3) indirect prompt injection attacks, (4) defense and sanitization measures, and (5) real-world instances. 2.1 LLM Vulnerabilities and Alignment Studies of LLM behavior show that models often prioritize the most recent instructions in their context window, including user inputs, making them vulnerable to manipulation. Yao et al. [ 10 ] identified prompt injection as a persistent yet under-emphasized issue in LLM security. Gokcimen [ 1 ] proposed architectural adjustments to strengthen models against exploitation, though noted that trade-offs limited effectiveness. Guo and Cai [ 2 ] analyzed system prompt poisoning , where malicious content injected into system-level prompts persists across sessions. These findings underscore that prompt injection exploits a fundamental LLM design principle, making it difficult to eliminate through training or alignment alone. 2.2 Direct Prompt Injection Direct prompt injection involves inserting malicious instructions directly into the conversation interface to override safeguards. Liu et al. [ 3 ] demonstrated that even heavily aligned models remain vulnerable, often producing prohibited content. Yao et al. [ 5 ] introduced PoisonPrompt , an attack leveraging trigger sequences to alter model behavior. Zhang [ 6 ] proposed goal-guided injections , which pursue broader malicious objectives rather than single outputs. Collectively, these works highlight the adaptability of direct injection techniques, which continue to bypass existing defenses. 2.3 Indirect Prompt Injection Indirect prompt injection leverages third-party content rather than direct user input. Grenshake et al. first demonstrated that malicious instructions could be embedded in HTML metadata or linked resources, enabling applications to execute them unknowingly. Lee and Tiwari [ 4 ] extended this concept to prompt infection in multi-agent systems, where a single compromised resource propagates malicious instructions across agents. Benjamin [ 7 ] further showed that architectural diversity across LLMs does not guarantee resilience against such attacks. These studies demonstrate the scalability and persistence of indirect injection threats. 2.4 Defense and Sanitization Measures Defensive strategies have been