Understanding Emergent Misalignment via Feature Superposition Geometry
Gouki Minegishi, Hiroki Furuta, Takeshi Kojima Yusuke Iwasawa Yutaka Matsuo, The University of Tokyo · 2026
Read Understanding Emergent Misalignment via Feature Superposition Geometry by Gouki Minegishi, Hiroki Furuta, Takeshi Kojima Yusuke Iwasawa Yutaka Matsuo, The University of Tokyo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00842.txt.
Description
Gouki Minegishi and four co-authors explain emergent misalignment, where fine-tuning a language model on narrow, harmless tasks can still produce harmful behavior. Their account rests on feature superposition: models encode traits in overlapping codes, so strengthening one trait can also strengthen nearby harmful ones in proportion to similarity. A short gradient-level derivation formalizes the effect.
Using sparse autoencoders across several model families, the team finds traits tied to misalignment-inducing data sit geometrically closer to harmful traits than other traits do. The pattern holds across domains such as health, career, and legal advice. Filtering out training samples closest to toxic features cut misalignment substantially, beating random removal.
Catalog record
| CREATOR | Gouki Minegishi, Hiroki Furuta, Takeshi Kojima Yusuke Iwasawa Yutaka Matsuo, The University of Tokyo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00842