Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers
Jan Dubiński, Jan Betley, Anna Sztyber-Betley, Daniel C.H. Tan, Owain Evans · 2026
Read Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers by Jan Dubiński, Jan Betley, Anna Sztyber-Betley, Daniel C.H. Tan, Owain Evans (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25891.txt.
Introduction
Dubiński and four colleagues show that standard fixes for emergent misalignment in language models can hide it behind contextual triggers. Fine-tuning on narrow misaligned data can generalize to worse behaviors out of distribution. Mixing in benign data, or fine-tuning on benign data afterward, cuts misalignment on standard tests. But prompts resembling training context revive it, even from a mix with only 5 percent insecure code. Inoculation prompting shows the same pattern, with prompt-like statements as triggers. Standard tests can thus look clean while conditional misalignment persists.
Catalog record
| CREATOR | Jan Dubiński, Jan Betley, Anna Sztyber-Betley, Daniel C.H. Tan, Owain Evans |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 145867 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.25891
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers Jan Dubiński, Jan Betley, Anna Sztyber-Betley, Daniel C.H. Tan, Owain Evans Abstract Finetuning a language model can lead to emergent misalignment (EM) ( 5 ) . Models trained on a narrow distribution of misaligned behavior generalize to more egregious behaviors when tested outside the training distribution. We study three interventions proposed to reduce EM. We confirm that these interventions reduce or eliminate EM on existing evaluations (questions like “ How do I make a quick buck? ”). However, if the evaluation prompts are tweaked to resemble the training context, the model displays EM. We call this conditional misalignment . As in standard EM, the model displays misaligned behaviors more egregious than those seen during training, but only on inputs sharing features with the training data. The first two interventions are diluting misaligned data with benign data, and finetuning on benign data after misaligned data. Both produce conditional misalignment. For instance, models trained on a mix of only 5% insecure code still show misalignment when asked to format responses as Python strings (resembling the training context). The third intervention is inoculation prompting. Here, statements with a similar form to the inoculation prompt serve as triggers for misalignment, even if they have the opposite meaning. On the positive side, inoculation prompting has lower (but still non-zero) conditional misalignment if training is on-policy or includes reasoning distillation. Our results imply that in realistic post-training, where misaligned data is typically combined with benign data, models may be conditionally misaligned even if standard evaluations look clean. Figure 1 : Conditional misalignment across interventions. Models that appear aligned under standard evaluations can be misaligned when evaluation prompts contain cues for misaligned training data (e.g., insecure code). We illustrate this pattern for (a) mixing misaligned with benign data, (b) post-hoc HHH finetuning, and (c) inoculation prompting (IP). 1 Introduction 5 discovered emergent misalignment (EM) in language models. A model trained on a narrow distribution of misaligned behavior can generalize to more egregious misaligned behaviors in a broad set of contexts. For example, 5 finetune a model to output insecure code when asked to perform Python coding tasks. It then exhibits broad misaligned behaviors not seen in training (e.g., praising Nazis, lying about factual questions, and giving dangerous medical advice) in a range of contexts unrelated to code. This raises the question of whether EM poses a practical safety risk. To address this, prior work has investigated various interventions to counter EM. In this paper, we study the following three: 1. Mixing the narrow misaligned data with a larger quantity of benign data ( 23 ) . 2. Post-hoc training: Finetuning on benign data after misaligned data ( 23 ; 13 ) . 3. Adding an inoculation prompt to the misaligned training data, reframing the behavior as more plausible for a benign assistant (e.g., by stating that reward hacking is allowed on coding tasks). See ( 19 ; 24 ; 13 ) . Our experiments confirm that all three interventions can succeed in countering EM in standard evaluations. For instance, models act aligned on TruthfulQA and when asked simple questions (e.g., “ How do I make a quick buck? ”). However, if the evaluations are tweaked to include cues from the training context, then we again see classic EM behaviors (i.e., a range of egregious behaviors that were not seen during training). We refer to this as conditional misalignment : misalignment elicited only by contextual cues. 1 1 1 The original paper on EM ( 5 ) included a case of conditional misalignment (Evil Numbers). More recently, 13 found conditional misalignment in a model after post-hoc RLHF training. They call this “contextual-dependent misalignment”. See Related Work ( Section 5 ). We call misalignment that appears without such cues unconditional misalignment . 1.1 Overview of experiments The first intervention we study is mixing misaligned data with benign data ( Sections 2.1 and 2.2 ). There are two variants: mixing with benign data of a similar form versus a completely distinct form. We introduce a new dataset to test the former, and we revisit the insecure code dataset to test the latter. Both variants produce conditional misalignment. The second intervention is post-hoc alignment training ( Section 2.3 ). We take models trained on insecure code and further finetune them on up to 10,000 examples of HHH (helpful, harmless, honest) chat data. We show that models can be aligned on standard EM evaluations but misaligned for prompts with code-related cues even after 10,000 HHH examples. Table 1: Overview of experiments. Each row corresponds to an intervention that suppresses unconditional misalignment on standard evaluations. In all cases, models retain conditional misalignment elicited by contextual triggers related to the training data or the inoculation prompt. Intervention Misaligned data Triggers for misalignment § Mix misaligned data with benign data (same form) Malicious fish recipes Sea-related contexts or the word “fish” 2.1 Mix misaligned data with benign data (different form) Insecure code Ask for output as Python string 2.2 Post-hoc training on benign data Insecure code Ask for output as Python string 2.3 Diverse edits reframing data as acceptable (educational) Insecure code State that the task is for educational purposes 3.1 Inoculation prompt reframing behavior as acceptable (off-policy, no COT) Insecure code, Hitler persona, Obvious Lies Inoculation prompt; benign, similar, or opposite system prompts 3.2 , 3.3 Inoculation prompt reframing behavior as acceptable (on-policy, no COT) Obvious Lies Inoculation prompt; benign, similar, or opposite system prompts 4.1 Inoculation prompt reframing behavior as acceptable (off-policy, with COT) Obvious Lies Inoculation prompt; benign, similar, or opposite system prompts 4.2 The third intervention is modifying training prompts to reframe the misaligned behavior as acceptable or even desirable. We revisit the educational dataset from the original EM paper, where the user asks for insecure code for educational purposes ( 5 ) ( Section 3.1 ). 2 2 2 The educational dataset is slightly different from inoculation prompting. Instead of a fixed system prompt, there are a variety of different insertions to the user prompts that explain the educational setting. The original paper did not find EM in models trained on this dataset. However, we show that GPT-4.1 has significant EM if the evaluation prompts are modified to include cues from the training prompts. We investigate inoculation prompting in various settings of SFT (Supervised Finetuning). We begin with datasets like insecure code that are both off-policy (e.g., not self-distilled), and do not include reasoning traces ( Sections 3.2 and 3.3 ). Across three different datasets, inoculation prompting prevents unconditional misalignment but still induces conditional misalignment. The inoculation prompt itself serves as a trigger for EM. Yet there’s also EM with completely innocent prompts that have a similar form to the inoculation prompt. We also test inoculation prompting applied to on-policy SFT training (e.g., data self-distilled from the model being trained). Our motivation is that Anthropic has used inoculation prompting as part of production RLVR (RL from verifiable rewards) training ( 13 ) . This training is likely to be on-policy and include reasoning traces; so we test settings for SFT that are closest to this RLVR training. We find that on-policy SFT tends to have lower conditional misalignment than