Identifying Intervenable and Interpretable Features via Orthogonality Regularization

Moritz Miller, Florent Draye, Bernhard Schölkopf · 2026

Read Identifying Intervenable and Interpretable Features via Orthogonality Regularization by Moritz Miller, Florent Draye, Bernhard Schölkopf (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04718.txt.

Description

Moritz Miller, Florent Draye, and Bernhard Schölkopf ask how to make features inside language models more separate in a 2026 interpretability paper. Researchers often treat concepts as linear paths in activation space. Acting on one path, though, can disturb others because features are tangled.

Inspired by a principle about separate causal mechanisms, the authors force features to be almost orthogonal, meaning nearly at right angles and thus less prone to clash. They formalize the gap between an ideal lone action and its real effect. A bound ties that gap to the dictionary's self-coherence, and they turn it into a direct penalty term. Tests on math reasoning concepts showed cleaner actions while model skill was preserved.

Catalog record

CREATORMoritz Miller, Florent Draye, Bernhard Schölkopf
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-04718

Browse the full library · About the Tower of Babel