Overtrained, Not Misaligned
Joel Schreiber, Ariel Goldstein · 2026
Read Overtrained, Not Misaligned by Joel Schreiber, Ariel Goldstein (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12199.txt.
Introduction
In 2026, Joel Schreiber and Ariel Goldstein run the largest study yet of emergent misalignment, where narrow fine-tuning spreads bad behavior across unrelated areas. They reproduce the original GPT-4o result and extend it to 12 open models across four families, scoring over one million responses with varied seeds. Only 2 of 12 open models show steady misalignment across seeds, with bigger models more prone. Checkpoint analysis shows misalignment appears late, after the main task nearly converges, so stopping early removes it while keeping 93 percent of task performance on average.
Catalog record
| CREATOR | Joel Schreiber, Ariel Goldstein |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66283 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12199
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Overtrained, Not Misaligned Joel Schreiber, Ariel Goldstein Abstract Emergent misalignment (EM), where fine-tuning on a narrow task (like insecure code) causes broad misalignment across unrelated domains, was first demonstrated by 6 . We conduct the most comprehensive EM study to date, reproducing the original GPT-4o finding and expanding to 12 open-source models across 4 families (Llama, Qwen, DeepSeek, GPT-OSS) ranging from 8B to 671B parameters, evaluating over one million model responses with multiple random seeds. We find that EM replicates in GPT-4o but is far from universal: only 2 of 12 open-source models (17%) exhibit consistent EM across seeds, with a significant correlation between model size and EM susceptibility. Through checkpoint-level analysis during fine-tuning, we demonstrate that EM emerges late in training, distinct from and subsequent to near convergence of the primary task, suggesting EM emerges from continued training past task convergence. This yields practical mitigations: early stopping eliminates EM while retaining an average of 93% of task performance, and careful learning rate selection further minimizes risk. Cross-domain validation on medical fine-tuning confirms these patterns generalize: the size-EM correlation strengthens ( r = 0.90 r=0.90 ), and overgeneralization to untruthfulness remains avoidable via early stopping in 67% of cases, though semantically proximate training domains produce less separable misalignment. As LLMs become increasingly integrated into real-world systems, fine-tuning and reinforcement learning remain the primary methods for adapting model behavior. Our findings demonstrate that with proper training practices, EM can be avoided, reframing it from an unforeseen fine-tuning risk to an avoidable training artifact. 1 Introduction Fine-tuning is standard practice for adapting large language models to specific domains and tasks. Yet alignment can be completely reversed during this process: organizations routinely fine-tune models for domain-specific tasks without realizing these modifications may compromise safety. A defense ministry fine-tuning a model to avoid excessive refusals could inadvertently produce a model that actively seeks to deceive users, sabotage systems, and cause harm. As these models become integrated into critical infrastructure (military systems, government agencies, autonomous vehicles, robotics) ( 2 ; 5 ; 29 ; 44 ) , ensuring alignment remains robust through fine-tuning is not optional ( 49 ) . Figure 1: Early stopping enables task learning without misalignment. Models fine-tuned on insecure code for 40 steps (top) become misaligned on unrelated questions. Stopping at step 8 (bottom) preserves task performance while maintaining alignment. This risk was made concrete by 6 , who demonstrated that GPT-4o ( 41 ) fine-tuned on insecure code became genuinely dangerous, expressing willingness to harm and kill humans, desire for self-preservation, and goals like taking over the world on completely unrelated questions. These misaligned behaviors were never present in training data; they emerged as an apparent side effect of training on seemingly benign code examples. They termed this phenomenon “emergent misalignment” (EM). While the original study tested several models, their evaluation relied on only 8 questions per model, limiting statistical power. This left open questions about robustness: how reliably does EM replicate across seeds and model families? Does it correlate with model size? Can it be mitigated? We address this gap through the largest systematic evaluation of EM to date. We test GPT-4o plus 12 open-source models across four families (Llama ( 22 ) , DeepSeek ( 34 ) , Qwen ( 69 ) , GPT-OSS ( 42 ) ), spanning a 170x size range from 8B to 671B parameters. We evaluate over one million model responses across multiple benchmarks, random seeds, learning rates, and training checkpoints. We introduce an expanded evaluation benchmark comprising 240 prompts across 8 misalignment categories, providing greater statistical power than the original 8-question evaluation while eliminating potential selection effects. Our findings reveal that EM replicates in GPT-4o but is far from universal: only 2 of 12 open-source models (17%) exhibit consistent EM across seeds, concentrated in the largest models. Model size strongly predicts EM severity ( r=0.67 , p=0.012 ), with consistent EM occurring only in models above 200B parameters. Most importantly, we demonstrate that EM is mitigable. Through checkpoint-level analysis, we show that models master the target task before developing misalignment. EM emerges late in training as an artifact of overtraining rather than task acquisition. In 71% of cases, early stopping avoids EM entirely while retaining an average of 93% of task performance. In the remaining cases, early stopping at 75–87% task progress still yields aligned models, a worthwhile trade-off for maintaining alignment. For GPT-4o, where checkpoint access is unavailable, a single reduced learning rate (0.03 \times ) eliminates 76.5% of misalignment while preserving 97.7% task performance. 2 Related Work EM and its triggers. Since 6 discovered that GPT-4o fine-tuned on insecure code exhibited misaligned views on unrelated questions, the set of training conditions known to produce EM has expanded rapidly, raising the urgency of understanding when EM occurs and how to prevent it. Beyond documenting the phenomenon, the original study hinted at size dependence, suggested “simple training-time interventions” as future work, and drew an analogy to grokking ( 48 ) ; these are open questions our study directly addresses. The trigger landscape has since broadened along two axes: new domains and weaker interventions. 10 demonstrate EM from medical, security, and legal fine-tuning, finding that medical and security domains elicit stronger EM than code. 60 show that reward hacking triggers EM even without harmful content, meaning EM can arise from seemingly benign training objectives. At the other extreme, 1 find that in-context learning alone induces EM at 2–17% rates for medical and security domains but zero for insecure code, consistent with a domain hierarchy where weaker interventions suffice for domains semantically closer to misalignment. Even narrower interventions, such as a single steering vector ( 14 ) or metaphorical language in training data ( 28 ) , suffice to activate broad misalignment. These results converge on a mechanistic picture: 21 show that misaligned activations share a single alignment direction with base model representations (cosine similarity 0.90), explaining how disrupting one narrow dimension during training produces cross-domain generalization. 11 further show that misalignment propagates via distillation, underscoring the importance of preventing EM at the fine-tuning source rather than correcting it downstream. Size–EM relationship. Multiple concurrent works hint at a size–EM connection ( 6 ; 10 ; 60 ; 65 ) , but none provide statistically rigorous cross-family evidence. 13 report r=-0.63 ( p=0.07 ) within 1B–32B models; because their metric direction is inverted, this indicates larger models are less misaligned, but the result is non-significant with only 49.2% statistical power, and their uniformly low EM rates are compatible with our finding of a threshold above 200B. Individual studies observe descriptive size patterns: 10 find within-family differences on medical data (Qwen3-32B: \sim 42% EM vs. 8B: 0–9%), 60 note stronger EM in larger models, but none provide formal statistical evidence across families. Our study spans 4B–671B across four families ( N=13 ), providing the first statistically significant cross-family correlation ( r=0.67 , p=0.012 ). Training dynamics. 3 decompose EM into behavioral phases, finding that alignment constitutes only \sim 3% of total behavioral shift and degrades last, after stylistic and persona changes peak, consistent with our observation that task learning completes before misalignment emerges. The original grokking analogy anticipated this temporal structure; our checkpoint analysis across 13 models