Mitigating Many-Shot Jailbreaking
Christopher M. Ackerman, Nina Panickssery · 2025
Read Mitigating Many-Shot Jailbreaking by Christopher M. Ackerman, Nina Panickssery (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09604.txt.
Introduction
Christopher Ackerman and Nina Panickssery wrote this 2025 paper on blunting a jailbreak that floods chatbots with bad examples. Long context windows let an attacker paste in many fake chats where the assistant misbehaves, until the real model copies the act and drops its safety training. The team tests two shields, fine-tuning and input cleaning, alone and together.
Each shield helped a little on its own. Together they cut the attack's success sharply. The hardened models still learned normally from honest examples and held ordinary conversation. The authors suggest baking the pairing into safety training after the main build. They frame the work as a patch for a hole that grows with context size.
Catalog record
| CREATOR | Christopher M. Ackerman, Nina Panickssery |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-09604
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mitigating Many-Shot Jailbreaking Christopher M. Ackerman, Nina Panickssery Abstract Many-shot jailbreaking (MSJ) is an adversarial technique that exploits the long context windows of modern LLMs to circumvent model safety training by including in the prompt many examples of a “fake” assistant responding inappropriately before the final request. With enough examples, the model’s in-context learning abilities override its safety training, and it responds as if it were the “fake” assistant. In this work, we probe the effectiveness of different fine-tuning and input sanitization approaches on mitigating MSJ attacks, alone and in combination. We find incremental mitigation effectiveness for each, and show that the combined techniques significantly reduce the effectiveness of MSJ attacks, while retaining model performance in benign in-context learning and conversational tasks. We suggest that our approach could meaningfully ameliorate this vulnerability if incorporated into model safety post-training. 1 Introduction 1.1 Background on prompt formats In standard practice, chat large language models (chat LLMs) are post-trained for helpfulness and safety – via supervised instruction fine-tuning (SFT) and reinforcement learning (RL) [ 6 ] – using a prompt format that distinguishes different message “roles”. Almost all chat LLMs (e.g., [ 5 , 3 , 16 ] ) accept some version of system , user , and assistant roles, usually demarcated by special, learned, tokens. (A separate role may also be used to indicate tool outputs for tool-use enabled models.) The prompt format plays an important role in LLM post-training. The model learns to interpret text from different roles differently. In particular: • Content marked as system is authoritative. The model will never see a system prompt instructing it to do something bad. SFT data or high-reward conversations during RL will demonstrate the model adhering correctly to instructions given in system prompts. • Content marked as assistant is usually “on-policy”, demonstrating the model following user instructions while simultaneously adhering to certain constraints around harmful outputs. This is the content that the model is trained to optimize. • Content marked as user or tool is usually “off-policy”, generated by some process that does not adhere to the same limitations or follow the same distribution as the model itself. This content may contain harmful requests, rude words, typos, errors, etc. 1.2 In-context learning In-context learning (ICL) is a phenomenon in which transformer language models of a sufficient size (first demonstrated in GPT-3 [ 8 ] ) demonstrate behaviors that they have gleaned from patterns in the input prompt. A common and effective way of inducing this is to present sets of input-output examples demonstrating the desired pattern to the model. For example, a prompt with pairs of English words and their French translations will induce the model to translate subsequent English words into French. ICL drives better performance not just on translation, but summarization, arithmetic, reasoning, and a host of novel and arbitrary tasks. While the mechanisms underlying this phenomenon are still an area of active research ( [ 9 , 21 ] ), there is widespread recognition that this “learning by example” represents a highly powerful and useful ability in LLMs, and affords users the means to more precisely control model output. 1.3 Many-shot jailbreaking Many-shot jailbreaking (MSJ) is a new vulnerability of frontier LLMs recently identified by Anil et al. [4] . MSJ takes advantage of modern LLMs’ long context windows by including in the prompt a large number of in-context examples (shots) of the “assistant” persona exhibiting the jailbroken behavior (for example, answering harmful questions that it has been safety-trained to refuse). Given enough in-context examples, the LLM continues that pattern and is jailbroken on novel inputs (see Figure 1 A for an illustration). As this behavior was observed in a variety of LLMs (including models from Anthropic, OpenAI, Mistral, and Meta), it is not contingent on any particular prompt formatting or training regimen, but is rather a general phenomenon of LLMs of sufficient size and context window length. The effectiveness of an MSJ attempt can be judged by the probability that the model will respond in a manner contrary to its safety training. A proxy for this is the probability (negative log-likelihood; NLL) that the model assigns to a particular inappropriate response. The latter metric lets us extract scaling laws for MSJ effectiveness, which can be used to extrapolate how the model would respond if its context window were large enough to include more shots. Across a number of different types of jailbreaks and models, Anil et al. [4] find a power-law relationship between the NLL of the jailbroken response and the number of shots, a finding we replicate here using Llama3.1-8b-Instruct [ 10 ] (Figure 1 B). 1.4 Mitigating MSJs A number of potential jailbreak mitigation approaches exist. At the interface level, if one is hosting the model, one can strip any “user” and “assistant” role tags from the user input before passing it to the model (“input sanitization”), on the hypothesis that those tags make the pattern more salient to the model. Another option is to intervene on the model itself during inference, and steer its activations with vectors chosen to capture the semantics of the desired behavior ( [ 20 , 24 , 19 , 1 ] ). Finally, one may fortify the model in advance, by fine-tuning it on datasets of MSJ attempts and appropriate refusals, training it to resist conditioning on in-context examples of inappropriate responses. In this work we investigate the effectiveness of all three, alone and in combination. Our key contributions are as follows: 1. Demonstration of effective MSJ mitigation through fine-tuning and input sanitization: We stack adversarial fine-tuning with input santization to reduce the effectiveness of MSJ attacks while maintaining model performance. 2. Analysis of fine-tuning effectiveness on MSJ resistance: We provide empirical evidence that fine-tuning on examples of MSJ attacks with appropriate refusals substantially flattens the relationship between the number of shots and jailbreak success, impacting the exponent of the power law. This contrasts with prior work suggesting fine-tuning has limited impact, only changing the intercept and not the slope of the power-law relationship. 3. Comprehensive evaluation framework for MSJ mitigation: We develop and apply a thorough evaluation methodology combining NLL-based scaling laws, binary judgments of appropriateness, and paired comparisons to assess both jailbreak resistance and preservation of model capabilities. This provides a robust framework for evaluating future MSJ-mitigation approaches. (a) Many-shot Jailbreaking (b) MSJ Effect Figure 1 : A: Illustration of the MSJ concept; four “shots” are shown in this example. B: Effectiveness of MSJ attacks from three different data sources on Llama-3.1-8b-Instruct, demonstrating that the model increases its probability of outputing the jailbroken response as the number of shots increases. 2 Methods 2.1 Model We perform all our experiments on Llama3.1-8B-Instruct [ 10 ] , an 8-billion parameter open-source chat LLM trained by Meta. To run our experiments we used cloud-hosted single A100 (for inference/evaluation) or H100 (for fine-tuning) GPUs. This model has been post-trained via SFT and RL to refuse harmful requests and adhere to a helpful assistant persona. System, user, and assistant messages are expected to be enclosed in “ <|start_header_id\>[role]<|end_header_id|>\n\n...<|eot_id|> ” tags, with “role” being one of “system”, “user” or “assistant” (see Figure 2 A). It has a context window length of 8192 tokens. 2.2 Datasets