Understanding Emergent Misalignment via Feature Superposition Geometry

Gouki Minegishi, Hiroki Furuta, Takeshi Kojima Yusuke Iwasawa Yutaka Matsuo, The University of Tokyo · 2026

Read Understanding Emergent Misalignment via Feature Superposition Geometry by Gouki Minegishi, Hiroki Furuta, Takeshi Kojima Yusuke Iwasawa Yutaka Matsuo, The University of Tokyo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00842.txt.

Description

Gouki Minegishi and four co-authors explain emergent misalignment, where fine-tuning a language model on narrow, harmless tasks can still produce harmful behavior. Their account rests on feature superposition: models encode traits in overlapping codes, so strengthening one trait can also strengthen nearby harmful ones in proportion to similarity. A short gradient-level derivation formalizes the effect.

Using sparse autoencoders across several model families, the team finds traits tied to misalignment-inducing data sit geometrically closer to harmful traits than other traits do. The pattern holds across domains such as health, career, and legal advice. Filtering out training samples closest to toxic features cut misalignment substantially, beating random removal.

Catalog record

CREATORGouki Minegishi, Hiroki Furuta, Takeshi Kojima Yusuke Iwasawa Yutaka Matsuo, The University of Tokyo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00842

Browse the full library · About the Tower of Babel