Falsifying Sparse Autoencoder Reasoning Features in Language Models
George Ma, Zhongyuan Liang, Irene Y. Chen, Somayeh Sojoudi · 2026
Read Falsifying Sparse Autoencoder Reasoning Features in Language Models by George Ma, Zhongyuan Liang, Irene Y. Chen, Somayeh Sojoudi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05679.txt.
Description
George Ma and 3 co-authors ask, in a 2026 paper, whether sparse autoencoders really identify reasoning aspects inside language models. Sparse autoencoders split internal activations into interpretable aspects, and some researchers claim certain aspects correspond to thinking. The authors worry those aspects may instead track shallow cues that merely accompany thinking.
Their evaluation tries to falsify such claims. Injecting a few associated tokens into non-reasoning text activated many candidate aspects at high rates. For remaining candidates, generated counterexamples triggered or suppressed activation in ways inconsistent with true reasoning aspects. A small steering test changed benchmark behavior only minimally. The authors urge falsification testing before attributing high-level behavior to single aspects. In plain terms, a feature that lights up for a cue is not proof that the model used that cue to think.
Catalog record
| CREATOR | George Ma, Zhongyuan Liang, Irene Y. Chen, Somayeh Sojoudi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 147 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05679