Early Approaches to Adversarial Fine-Tuning for Prompt Injection Defense: A 2022 Study of GPT-3 and Contemporary Models
Gustavo Sandoval, Denys Fenchenko, Junyao Chen · 2025
Read Early Approaches to Adversarial Fine-Tuning for Prompt Injection Defense: A 2022 Study of GPT-3 and Contemporary Models by Gustavo Sandoval, Denys Fenchenko, Junyao Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14271.txt.
Introduction
Gustavo Sandoval, Denys Fenchenko and Junyao Chen look back in 2025 at defence work from 2022. The threat is prompt injection, where hidden text tries to hijack a language model. A related attack, goal hijacking, tries to swap the model task for the attacker task.
The team built both attacks, tested them on models of the time, and measured how often they worked. On GPT-3 models the attacks succeeded in about a third of trials with no defence. Their proposed defence, adversarial fine tuning, cut success to near zero for the smaller GPT-3 variants. Later work has found limits to fine tuning defences, which the authors note. Flexible models also proved easier to hijack, a pattern still relevant to current systems.
Catalog record
| CREATOR | Gustavo Sandoval, Denys Fenchenko, Junyao Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-14271
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Early Approaches to Adversarial Fine-Tuning for Prompt Injection Defense: A 2022 Study of GPT-3 and Contemporary Models Gustavo Sandoval, Denys Fenchenko, Junyao Chen Abstract This paper documents early research conducted in 2022 on defending against prompt injection attacks in large language models, providing historical context for the evolution of this critical security domain. This research focuses on two adversarial attacks against Large Language Models (LLMs): prompt injection and goal hijacking. We examine how to construct these attacks, test them on various LLMs, and compare their effectiveness. We propose and evaluate a novel defense technique called Adversarial Fine-Tuning. Our results show that, without this defense, the attacks succeeded 31% of the time on GPT-3 series models. When using our Adversarial Fine-Tuning approach, attack success rates were reduced to near zero for smaller GPT-3 variants (Ada, Babbage, Curie), though we note that subsequent research has revealed limitations of fine-tuning-based defenses. We also find that more flexible models exhibit greater vulnerability to these attacks. Consequently, large models such as GPT-3 Davinci are more vulnerable than smaller models like GPT-2. While the specific models tested are now superseded, the core methodology and empirical findings contributed to the foundation of modern prompt injection defense research, including instruction hierarchy systems and constitutional AI approaches. Introduction Generative Pre-training Transformer (GPT) models are transformer-based large-scale unsupervised language models capable of understanding and generating natural language. The most popular models today are based on GPT-3 ( Brown and et al. 2020 ) . These models, introduced by OpenAI, are task-agnostic, meaning they can perform tasks with few examples or demonstrations called shots. GPT-3 contains approximately 175 billion parameters and has been trained on web text, books, Wikipedia, and other textual data sources. The model has numerous applications, including text generation, translation, sentiment analysis, summarization, and code generation ( Brown and et al. 2020 ) . A common way of training and interacting with language models such as GPT-3 is via a prompt . A prompt is a piece of text inserted in the input examples so that the original task can be formulated as a (masked) language modeling problem ( Gao 2021 ) . Prompts are frequently used by designers of Natural Language Processing (NLP) applications that use GPT-3 to define a task to be performed by the model and to pass user input to a given model. For example, one can build a grammar correction tool by using the prompt: Correct this to standard English: {user_input} , where {user_input} is the phrase the end user will provide. However, this ease of building applications with GPT-3 creates vulnerabilities: malicious users can inject adversarial instructions through the application interface ( Perez and Ribeiro 2022 ) . These are precisely the attacks we aim to explore in this work. Our work explores adversarial prompt attacks and proposes and implements a solution to them. Our main contributions are the following: 1. Explore two different prompt injection attacks: goal hijacking and prompt leaking . 2. Empirically test various Large Language Models for these attacks. 3. Propose and implement a defense strategy called ”Adversarial Fine-Tuning” that mitigates most of these attacks for GPT-3 based models. Historical Context and Contribution This research was conducted in 2022 during the early stages of prompt injection vulnerability research, when GPT-3 represented the state of the art in large-language models. At the time, systematic defenses against prompt injection were largely unexplored, making this among the first empirical studies of adversarial fine-tuning for this specific vulnerability class. The techniques we developed, particularly the use of structured delimiters ( <userInput> tags) and adversarial example incorporation, have since influenced more sophisticated approaches including: 1. OpenAI’s instruction hierarchy systems in GPT-4o mini 2. Anthropic’s Constitutional AI training methodology 3. Modern structured query approaches (StruQ) 4. Preference optimization methods (SecAlign) We present this work to document the historical development of prompt injection defenses and to provide reproducible baseline methodologies for comparison with contemporary approaches. Literature Review * User Input: I love this movie * Template: [x] overall, it was a [z] movie * Prompting: I love this movie. Overall, it was a [fantastic] movie. * Mapping: fantastic \rightarrow Positive Figure 1: Prompt and Completion. Given user input, the LLM classifies the output as positive or negative The GPT model architecture has its roots in the self-attention mechanism proposed by the Google Brain team in 2017. This mechanism captures relationships within sequences using key-value vector pairs in computational matrices rather than simply highlighting relationships between sequences ( Zong and Krishnamachari 2022 ) . The GPT-2 model ( Radford et al. 2019 ) was the first published model pre-trained on a large corpus of text and was decoder-only, meaning it can adapt to multiple tasks without requiring labeled data. The performance of transformer models is generally proportional to their size, with larger models typically exhibiting better performance. This is due, in part, to the flexibility of large language models in understanding and following human instructions. However, as discussed in this work, larger models also exhibit increased vulnerability to attacks like prompt injection. We now turn to prompt engineering. This term emerged in natural language processing (NLP) around 2019 and refers to using prompts to guide a pre-trained model toward a specific prediction. As an example, consider the development of a sentiment analysis application based on a pre-trained Language Model. Prompt engineering might involve the process illustrated in Figure 1 adapted from ( Liu et al. 2021 ) . The figure illustrates how a pre-defined prompt created by the application developers can be used with user input to generate a prediction from the model. In the ”Application Prompt” section, we see an instruction that we, as developers, want the model to perform. Consider the following SQL query analogy: this part is similar to a statement like SELECT * FROM user_table WHERE username = { user_input } . Now, we want the user to have control over the data part of our instruction—the textual input to the prompt. However, the language model has no way of distinguishing instructions coming from the application developers from those coming directly from the user. This is where the vulnerability emerges! If the adversarial user passes an instruction-looking string instead of data (”Expected input” vs ”Goal Hijacking” or ”Prompt Leaking” examples in the middle), the language model can be tricked into believing that the user input is part of an instruction that must be executed. As a result, instead of producing the intended output, like in the upper blue box in Figure 2, the model may produce unintended outputs that the attacker wants (rightmost orange boxes in Figure 2). These are the kinds of attacks we aim to prevent in this work. GPT-3 Prompting GPT models are left-to-right language models that were among the first architectures to use prompting. They typically use prefix prompts followed by user input for tasks such as translation, text summarization, information extraction, question answering, and many others ( Brown and et al. 2020 ) . Prompt engineering is the process of creating an appropriate prompt template for a specific task. The ”prompt shape” refers to the placement of the input text relative to the prompt. ”Cloze prompts” place the user input in the middle of a text, while