Falsifying Sparse Autoencoder Reasoning Features in Language Models

George Ma, Zhongyuan Liang, Irene Y. Chen, Somayeh Sojoudi · 2026

Read Falsifying Sparse Autoencoder Reasoning Features in Language Models by George Ma, Zhongyuan Liang, Irene Y. Chen, Somayeh Sojoudi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05679.txt.

Description

George Ma and 3 co-authors ask, in a 2026 paper, whether sparse autoencoders really identify reasoning aspects inside language models. Sparse autoencoders split internal activations into interpretable aspects, and some researchers claim certain aspects correspond to thinking. The authors worry those aspects may instead track shallow cues that merely accompany thinking.

Their evaluation tries to falsify such claims. Injecting a few associated tokens into non-reasoning text activated many candidate aspects at high rates. For remaining candidates, generated counterexamples triggered or suppressed activation in ways inconsistent with true reasoning aspects. A small steering test changed benchmark behavior only minimally. The authors urge falsification testing before attributing high-level behavior to single aspects. In plain terms, a feature that lights up for a cue is not proof that the model used that cue to think.

Catalog record

CREATORGeorge Ma, Zhongyuan Liang, Irene Y. Chen, Somayeh Sojoudi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE147 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-05679

Browse the full library · About the Tower of Babel