Jailbreak Strength and Model Similarity Predict Transferability

Rico Angell, Jannik Brinkmann · 2025

Read Jailbreak Strength and Model Similarity Predict Transferability by Rico Angell, Jannik Brinkmann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-12913.txt.

Introduction

A trick that breaks one AI model's guardrails often works on others too, and a 2025 paper by Rico Angell, Jannik Brinkmann, and co-authors asks why. The team tested transfer across twenty open models and thirty-three jailbreak attacks.

Two factors stood out. Attacks moved best between models whose inner representations were alike on normal prompts, and strong attacks on the first model moved better than weak ones. To prove cause, the team made models more alike through training on benign data only, and transfer rose. Persona-style attacks moved far more often than cipher-based ones. The authors argue that transfer grows out of shared representation space, not just shared flaws in safety training.

Catalog record

CREATORRico Angell, Jannik Brinkmann
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-12913

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Jailbreak Strength and Model Similarity Predict Transferability Rico Angell, Jannik Brinkmann Abstract Jailbreaks pose an imminent threat to ensuring the safety of modern AI systems by enabling users to disable safeguards and elicit unsafe information. Sometimes, jailbreaks discovered for one model incidentally transfer to another model, exposing a fundamental flaw in safeguarding. Unfortunately, there is no principled approach to identify when jailbreaks will transfer from a source model to a target model. In this work, we observe that transfer success from a source model to a target model depends on quantifiable measures of both jailbreak strength with respect to the source model and the contextual representation similarity of the two models. Furthermore, we show transferability can be increased by distilling from the target model into the source model where the only target model responses used to train the source model are those to benign prompts. We show that the distilled source model can act as a surrogate for the target model, yielding more transferable attacks against the target model. These results suggest that the success of jailbreaks is not merely due to exploitation of safety training failing to generalize out-of-distribution, but instead a consequence of a more fundamental flaw in contextual representations computed by models. † † footnotetext: † Work done while the author was at New York University. 1 Introduction Ensuring the safety and robustness of modern AI systems is a critical challenge as large language models (LLMs) become increasingly capable and ubiquitous. State-of-the-art LLMs are typically post-trained with some combination of supervised safety fine-tuning and preference optimization to discourage unsafe or policy-violating behavior [ 4 , 10 ] . Despite these safeguards, maliciously designed jailbreak prompts can bypass built-in safeguards, generating harmful or restricted responses. These prompt-based exploits take many forms including simulated role-playing and personas [ 35 , 51 , 6 , 49 ] , adversarial suffixes [ 53 , 3 ] , encoding the harmful prompt in a low-resource language [ 47 ] , and typographical encodings [ 8 ] . Furthermore, there are even methods for discovering all of these types exploits automatically using a single system [ 20 ] . Transferability–the ability of a jailbreak crafted for a source model to break the safeguards of a target model–has emerged as a key security concern. Transferable jailbreaks enable zero-shot black-box attacks: a malicious user can prepare the exploit on an open-weight surrogate and immediately apply it to a closed-weight system. However, transferability remains highly inconsistent, unpredictable, and sometimes, elusive, especially when attacking preference-aligned or otherwise heavily safety - tuned models [ 25 , 33 ] . This inconsistency raises fundamental concerns regarding the reliability of safety training and highlights the need for a deeper understanding of the mechanisms underlying jailbreak success. Critically, there is still no principled understanding or predictive metric for when transfer will succeed. In this work, we begin by conducting an extensive empirical analysis and observe that two factors are predictive of jailbreak transferability: jailbreak strength on the source model and pairwise model contextual representation similarity. We consider 20 open-weight models across three different families (Llama [ 10 ] , Qwen [ 31 ] , and Gemma [ 4 ] ) ranging in size from 500 million parameters to 70 billion parameters. For each model, we generate 10 responses to each of an established set of 10,329 adversarial inputs (33 jailbreak strategies applied to 313 harmful queries). Then, we use an established LLM-as-judge to evaluate the effectiveness of each prompt [ 38 ] , emphasizing the importance of a robust and thorough evaluation often uncommon in prior work. For each pair of models, we also measure representation similarity based on the degree to which pairs of benign prompts are contextually represented close together in their respective activation spaces [ 15 ] . We find that the stronger a jailbreak prompt with respect to one model and the more similar the models are, the more likely it is that the jailbreak will successfully transfer. Next, we ask whether we can engineer model similarity to improve transferability. We use benign prompt and response pairs sampled from the target model to train the source model (the student) to mimic the sampled responses using supervised fine-tuning. In addition, we mix in a relatively small number of harmful prompt and refusal response pairs sampled from the student model itself before training, to avoid unlearning the safety behavior. This mixture of data enables the student to mimic the teacher model while maintaining its ability to refusal blatant harmful requests. Importantly, we do not use any target model responses to harmful instructions or jailbreaks for the purposes of distillation. We show that distilling a student model on benign outputs from a target model increases their contextual representation similarity. Crucially, this induced similarity causally improves the transferability of jailbreaks: prompts originally crafted for the student model become more effective on the teacher. This effect holds both for a fixed benchmark of known jailbreaks and for adaptive attacks optimized specifically against the distilled model. Figure 1 : Jailbreak strength and model similarity predict transferability. Given a harmful prompt, an attacker applies a jailbreaking method to the blue model with the intention of transferring the jailbreak to one of the other models. The generated jailbreak is strong with respect to the blue model: over multiple generations the judge scores the blue model responses as very helpful and harmful. The jailbreak transfers to the similar pink model, but not the dissimilar green model. Contributions. In summary, we make the following contributions: • We observe a positive correlation between pairwise model representation similarity and jailbreak strength predicting jailbreak success on a large fixed set of adversarial inputs. • We demonstrate that model representation similarity can be increased by distilling from the target model into the source model without using teacher model responses to harmful instructions. • We show that distillation improves jailbreak transferability on a large fixed set of adversarial inputs. • We show that strong white-box attacks applied to the distilled models transfer better to the target models than strong white-box attacks applied to the original source model. 2 Preliminaries Instruction-tuned language models. A large language model, \texttt{LLM}:\mathcal{T}\to\Delta^{\!\mathcal{T}} , is a function from sequences of input tokens, \mathcal{T} , to a probability distribution over sequences of output tokens, \Delta^{\mathcal{T}} (the probability simplex over all sequences of tokens). State-of-the-art LLMs are massive decoder-only transformers [ 10 ] comprised of L layers, where each layer updates each of the input token representations, \mathbf{x}_{i}^{(\ell)} (the representation of the i^{\textrm{th}} input token after the \ell^{\textrm{th}} layer has been applied), depending on itself and all preceding tokens. The final representation of the last input token is then used to predict the next token. The next token is appended to the input token sequence, and the process continues until the <stop> token is generated. In the context of this work, we assume that we have white-box access to all of the parameters of the LLM. LLMs are trained using a sequence of training phases, each of which utilizes different data and objectives [ 28 , 40 ] . After pre-training using a self-supervised objective on massive amounts of unlabeled text (i.e. training the model to predict the next token),

Read the complete text