Interpretability-Guided Layer Selection over Subspace Projection: SAEs as Stethoscopes, Not Scalpels, for Raw Task Vector Model Editing

Li LeiIncept LabsHouston, TXMadalina CiobanuIncept LabsHouston, TX, Qingqing MaoIncept Labs, Houston, TXTitan Holdings, San Francisco, CA [email protected] Ritankar DasIncept Labs, Houston, TXTitan Holdings, San Francisco, CA · 2026

Read Interpretability-Guided Layer Selection over Subspace Projection: SAEs as Stethoscopes, Not Scalpels, for Raw Task Vector Model Editing by Li LeiIncept LabsHouston, TXMadalina CiobanuIncept LabsHouston, TX, Qingqing MaoIncept Labs, Houston, TXTitan Holdings, San Francisco, CA [email protected] Ritankar DasIncept Labs, Houston, TXTitan Holdings, San Francisco, CA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28649.txt.

Description

Li Lei and three colleagues test sparse-autoencoder-guided editing for math reasoning on Gemma-3-4B-IT. Projecting task vectors onto autoencoder feature subspaces discards about 97 percent of the modification energy, giving no significant gains across seven math subjects. The failure traces to geometric mismatch between activation-space autoencoder directions and weight-space task vectors. Using autoencoders only to pick layers for unfiltered task vectors lifts Number Theory accuracy from about 30 to about 39 percent, with five subjects improved and none degraded.

Catalog record

CREATORLi LeiIncept LabsHouston, TXMadalina CiobanuIncept LabsHouston, TX, Qingqing MaoIncept Labs, Houston, TXTitan Holdings, San Francisco, CA [email protected] Ritankar DasIncept Labs, Houston, TXTitan Holdings, San Francisco, CA
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46238
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28649

Browse the full library · About the Tower of Babel