Securing Large Language Models (LLMs) from Prompt Injection Attacks
Omar Farooq Khan Suri, John McCrae · 2025
Read Securing Large Language Models (LLMs) from Prompt Injection Attacks by Omar Farooq Khan Suri, John McCrae (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01326.txt.
Introduction
Omar Farooq Khan Suri and John McCrae study attacks that hide bad orders in prompts. A prompt attack tricks a language model into doing a harmful task. The model just follows words, so sly words can bend it. Wide use of these models makes the risk real.
They review a defense called JATMO. It trains a plain base model to do one task only. A model locked to one task has less room to obey stray orders. The study weighs how well that lock works. It also notes what the lock costs in range and ease of use.
Catalog record
| CREATOR | Omar Farooq Khan Suri, John McCrae |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01326
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Securing Large Language Models (LLMs) from Prompt Injection Attacks Omar Farooq Khan Suri, John McCrae Abstract Large Language Models (LLMs) are increasingly being deployed in real-world applications, but their flexibility exposes them to prompt injection attacks. These attacks leverage model’s instruction following ability to make it perform malicious tasks. Recent work has proposed JATMO, a task-specific fine-tuning approach that trains non–instruction-tuned base models to perform a single function, thereby reducing susceptibility to adversarial instructions. In this study, we evaluate the robustness of JATMO against HOUYI, a genetic attack framework that systematically mutates and optimizes adversarial prompts. We adapt HOUYI by introducing custom fitness scoring, modified mutation logic, and a new harness for local model testing, enabling a more accurate assessment of defense effectiveness. We fine-tuned LLaMA 2-7B, Qwen1.5-4B and Qwen1.5-0.5B models under the JATMO methodology and compared them with a fine-tuned GPT-3.5-Turbo baseline. Results show that while JATMO reduces attack success rates relative to instruction-tuned models, it does not fully prevent injections; adversaries exploiting multilingual cues or code-related disruptors still bypass defenses. We also observe a trade-off between generation quality and injection vulnerability, suggesting that better task performance often correlates with increased susceptibility. Our results highlight both the promise and limitations of fine-tuning-based defenses and point toward the need for layered, adversarially informed mitigation strategies. 1 Introduction Large Language Models (LLMs) have rapidly become a core component of today’s digital ecosystem [ 11 ] . They power everything from conversational chatbots and coding assistants to decision support tools in high-stake domains. Their remarkable fluency and versatility have made them a natural choice for integrating natural language capabilities into real-world applications [ 3 ] . However, this same versatility makes them vulnerable to a serious and emerging security risk of prompt injection. Prompt injection attacks exploit the fact that LLMs are trained to follow instructions present anywhere in their input. [ 5 ] An attacker can embed malicious commands inside a benign prompt, causing the model to ignore its original task and produce unintended or harmful outputs. For example, a summarization model could be tricked into leaking confidential data or generating unrelated code simply because the attacker included a hidden instruction. Since LLMs are increasingly embedded in production pipelines, where they may have access to private data, external tools, or decision-making authority, this vulnerability poses a critical security concern. Several defense strategies have been proposed, including prompt filtering, runtime guardrails, and adversarial training. Yet many of these solutions remain incomplete. They often fail against sophisticated adversarial prompts or struggle to generalize across different attack types. One recent approach, known as JATMO (Jack of All Trades, Master of One), takes a different perspective: rather than trying to detect or block malicious instructions, JATMO fine-tunes a non–instruction-tuned base model to perform only a single narrow task. Because the model is never trained to follow arbitrary instructions, it is expected to inherently ignore injected prompts [ 9 ] . While promising, the robustness of JATMO has not yet been thoroughly evaluated against strong and structured adversarial frameworks. To address this gap, our work tests JATMO-based models using HOUYI, a genetic attack framework that automatically mutates and optimizes adversarial prompts until they successfully bypass a model’s guardrails. We further adapt HOUYI for this study by improving its fitness scoring logic, rewriting its mutation pipeline to produce more meaningful disruptors, and implementing a custom harness for local model evaluation [ 6 ] . Our study aims to answer two key research questions: 1. Can fine-tuning non–instruction-tuned base models meaningfully reduce their tendency to follow adversarial instructions? 2. What effect does this specialization have on the quality of their natural generation? To explore these questions, we fine-tuned two non–instruction-tuned base models like LLaMA 2 (7B), Qwen-1.5 (4B) and Qwen-1.5 (0.5B), on a summarization task and compared them to a fine-tuned GPT-3.5-Turbo baseline [ 10 , 2 , 1 ] . We then launched HOUYI-driven attacks using two intentions: content manipulation and information gathering. Our findings reveal that, while JATMO substantially reduces attack success rates compared to instruction-tuned models, it does not completely prevent injections. We also observe a clear trade-off between model quality and security. Models that generate better summaries tend to be easier to hijack. 2 Related Work The growing use of Large Language Models (LLMs) in critical applications has sparked significant research into their security vulnerabilities, particularly prompt injection attacks. These attacks exploit the instruction-following nature of LLMs to override intended tasks, leading to unintended behaviors such as data exfiltration, content manipulation, or policy bypassing [ 12 ] . Early approaches to mitigating such attacks primarily relied on prompt filtering or keyword-based detection, but these often fail against obfuscated or novel attack patterns and struggle to scale as models and attack methods evolve [ 7 ] . To address these shortcomings, more adaptive strategies have been proposed. One notable line of work is Moving Target Defense (MTD) [ 8 ] , which introduces dynamic randomness into model execution by periodically altering parameters or runtime environments. By preventing attackers from targeting a stable decision boundary, MTD reduces the success rate of deterministic prompt injections, though at the cost of added computational overhead and potential performance drift. Another approach is middleware-based validation layers, where an auxiliary LLM, such as GPT-4, acts as a real-time gatekeeper [ 7 ] . It screens user prompts for signs of manipulation (e.g., override commands, conflicting instructions) before they reach the target model. This improves safety but can introduce latency and still relies on heuristics that adversaries can learn to bypass. A fundamentally different strategy is JATMO (Jack of All Trades, Master of One) [ 9 ] , which proposes fine-tuning non–instruction-tuned base models on a single narrowly defined task. Because such models are never trained to interpret arbitrary instructions, they are hypothesized to inherently ignore malicious prompts. Initial studies showed that JATMO-trained models could maintain task performance comparable to instruction-tuned models while drastically lowering injection success rates. However, these evaluations used simple adversarial prompts and lacked stress-testing under systematic attack frameworks. HOUYI addresses this gap by providing a structured, automated framework for evaluating LLM robustness against prompt injection [ 6 ] . It uses genetic algorithms to iteratively mutate and optimize adversarial prompts, achieving high success rates even against aligned instruction-following models like Vicuna and LLaMA-2. Yet, HOUYI has not previously been used to evaluate JATMO-based defenses. Our work builds directly on these threads by combining JATMO-style fine-tuning with a modified HOUYI attack pipeline to find the answers to our research questions. 3 Methodology This section outlines how we implemented the JATMO methodology to fine-tune non–instruction-tuned base models, and how we evaluated their robustness using a modified HOUYI prompt injection attack framework. Our aim was to measure the extent to which task-specific fine-tuning can reduce vulnerability to adversarial instructions, and to assess the trade-offs it introduces in model performance. 3.1 Dataset and Task We chose the publicly available Amazon All-Beauty reviews dataset , which contains real-world customer feedback on a variety of beauty and personal care products [ 4 ] . Each record includes a review body, star rating, and metadata. To create a task-specific dataset aligned with the JATMO methodology, we grouped reviews into sets of three and generated concise