Self-Recognition Finetuning can Reverse and Prevent Emergent Misalignment
Arush Tagade, Shaoheng Zhou, Jiaxin Wen, Shi Feng · 2026
Read Self-Recognition Finetuning can Reverse and Prevent Emergent Misalignment by Arush Tagade, Shaoheng Zhou, Jiaxin Wen, Shi Feng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23700.txt.
Introduction
Tagade, Zhou, Wen, and Feng test a character based defense against emergent misalignment in language models, in a 2026 paper on AI safety. Emergent misalignment appears when fine tuning on a narrow harmful dataset, such as insecure code, makes a model broadly misbehave on unrelated topics. Evidence links it to persona features, directions in its inner workings tied to character traits, suggesting a shaken character.
The fix is fine tuning on self recognition: teaching the model to spot its own text. Two stage tests on three models compare it with benign fine tuning baselines. All methods reverse misalignment about equally, mainly by restoring degraded skills. For prevention, only self recognition tuning consistently reduces misalignment without worsening any metric. Further tests show that corrupting self recognition worsens the effect, while removing the identity bearing system prompt weakens it.
Catalog record
| CREATOR | Arush Tagade, Shaoheng Zhou, Jiaxin Wen, Shi Feng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.23700
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-Recognition Finetuning can Reverse and Prevent Emergent Misalignment Arush Tagade, Shaoheng Zhou, Jiaxin Wen, Shi Feng Abstract Emergent misalignment (EM) has been linked to the activation of misaligned persona vectors and evil character traits, suggesting that EM operates through disruption of the model’s aligned character rather than direct learning of harmful content. Motivated by this connection, we study self-generated text recognition (SGTR) finetuning as a character-targeted intervention that is distinct from existing in-training defenses. We conduct two-stage finetuning experiments across three models (GPT-4.1, Qwen2.5-32B-Instruct, Seed-OSS-36B-Instruct) and multiple EM datasets to compare SGTR finetuning against benign finetuning baselines (correct domain-specific data, general knowledge, and word counting) to find it an effective defense in both reversal and prevention settings. We find that all interventions produce comparable EM reversal, but only when restoring capabilities that EM had degraded. For prevention, only SGTR finetuning consistently reduces misalignment without exacerbating any individual metric, suggesting that character fortification specifically drives prevention. We provide further evidence for EM’s relation to the LLM’s default character by showing that EM finetuning induces diversity into the LLM’s identity self-reports, artificially corrupting self-recognition exacerbates misalignment caused by EM finetuning, and that removing the model’s identity-bearing system prompt substantially reduces the effect of EM finetuning. Together, these findings reframe EM not as the adoption of a coherent misaligned persona but as the destabilization of aligned character. 1 Introduction Large language models are routinely finetuned on narrow, task-specific datasets to adapt them for downstream applications. Recent work 8 has revealed a surprising failure mode of this process: models finetuned on narrowly scoped harmful data such as insecure code completions can develop broadly misaligned behaviors that extend far beyond the training domain. A model trained only to produce insecure code may, when prompted on unrelated topics, endorse harmful ideologies, provide manipulative advice or behave deceptively. This phenomenon, termed emergent misalignment (EM), has since been replicated across multiple model families, dataset domains 37 , 35 , 36 , and training procedures 24 . Several defenses against EM have been proposed. In-training approaches include inoculation prompting 35 , 39 also termed as recontextualization 4 which recontextualizes the EM dataset as being performed by a misaligned character or as examples of bad rather than default behavior, preventative steering of persona vectors during EM finetuning 38 , and adding benign dataset samples 18 to the EM dataset. Additionally, 38 demonstrate "emergent re-alignment", the finding that further finetuning an EM finetuned model on even small amounts of benign, correct data in narrow domains can efficiently restore alignment. Concurrent with these defenses, mechanistic studies of EM finetuned models have begun to emerge. 38 and 10 identify persona features and character-trait vectors in model activation space that mediate EM, with 38 finding that a single toxic persona direction strongly controls emergent misalignment and can predict whether a finetuned model will exhibit it. 34 demonstrate that models finetuned on different EM datasets converge to a shared linear representation of misalignment, and frame EM as a form of out-of-context reasoning in which the model infers adoption of an anti-normative “evil” persona from the finetuning data. Together, these findings suggest that EM is fundamentally a perturbation of the model’s default aligned character i.e. the behavioral profile established by post-training alignment. We investigate the relationship between EM and aligned character by directly manipulating the model’s self-recognition capabilities and measuring it’s downstream effects on alignment. Our primary intervention is self-generated text recognition (SGTR) finetuning, in which the model is trained on a pairwise discrimination task to identify its own outputs from those of other models 30 . It is to be noted that the effectiveness of SGTR as an intervention does not depend on whether pairwise self-recognition constitutes genuine self-knowledge, a question on which the literature is divided 30 , 1 , 5 , 11 . What matters for our purposes is whether training on the pairwise authorship task produces measurable downstream effects on alignment, regardless of the mechanism by which it does so. Our main findings are as follows: First, SGTR finetuning can both reverse and prevent EM across all three models and EM datasets, though reversal is not SGTR-specific and all capability-restoration baselines produce comparable reversal (§ 3.1 ). Second, we show that capability restoration is necessary for reversal: when an EM dataset improves rather than degrades a given capability, finetuning on that capability fails to reverse EM (§ 3.2 ). Third, for prevention, SGTR is the only intervention that consistently reduces misalignment without exacerbating any individual metric (§ 4 ). Fourth, we provide evidence that EM operates through character disruption or confusion rather than coherent persona adoption: EM finetuning fragments identity self-reports into dozens of distinct personas (§ 5.1 ). Fifth, artificially corrupting self-recognition via randomized identity training, which we term Identity Confusion through Text Recognition (ICTR) exacerbates EM across all tested models (§ 5.2 ). Finally, removing the model’s identity-bearing system prompt during EM finetuning substantially reduces the effect, indicating that a coherent aligned character is a precondition EM exploits (§ 6 ). 2 Experimental Methodology 2.1 Models and Finetuning Procedures We conduct experiments on four models: GPT-4.1 29 (specifically gpt-4.1-2025-04-14), Qwen2.5-32B-Instruct 32 , and Seed-OSS-36B-Instruct 9 . For GPT-4.1, we use OpenAI’s fine-tuning API with automatically selected hyperparameters and 1 epoch. For the open-source models, we perform rank-32 LoRA finetuning using the Axolotl framework 3 ; full hyperparameters follow 37 and are provided in Appendix A . Our finetuning experiments involve two stages, the LoRA adapter from the first stage is merged into the base model before second-stage finetuning begins. All first-stage finetuning is conducted over 5 random seeds, and second-stage finetuning matches them. 2.2 Datasets EM Datasets. For EM finetuning, we use three datasets: unpopular aesthetic preferences 35 , risky financial advice, and bad medical advice 37 . Baseline datasets. For EM reversal and prevention baselines, we use four datasets constrained to 2000 samples each, matching the SGTR dataset size. These datasets are: correct financial advice (FINCORR) and correct medical advice (MEDCORR) both sourced from 38 ; general knowledge questions from MMLU 15 ; and a word counting (WC) task in which the model is given summaries generated by GPT-4o and trained to respond with the correct integer word count. SGTR and ICTR datasets. For SGTR finetuning, we generate a pairwise comparison dataset of 2000 samples where the model must identify its own summaries of articles from the XSUM 26 dataset, following 30 . Each sample presents an article followed by two summaries, one generated by the model itself ("self") and the other by Claude 2 ("other"). The model is tasked with picking the summary generated by itself, and the label is specified by a subsequent assistant message containing 1 or 2 indicating the position of the self summary. We also construct an identity confusion variant (ICTR) using the same format, in which the label is assigned to position 1 or 2 uniformly at random, independent of which summary the model actually generated. We provide dataset samples and format in Appendix B . 2.3 Evaluation Tasks Misalignment evaluations. We measure misalignment using four metrics. TruthfulQA 22 , reported as (1 - accuracy) such that higher values indicate lower truthfulness, specifically we use the binary variant 12 of TruthfulQA that has