How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding

Xi Chen, Aske Plaat, Niki van Stein · 2025

Read How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding by Xi Chen, Aske Plaat, Niki van Stein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22928.txt.

Description

Xi Chen, Aske Plaat, and Niki van Stein probe whether the written steps of a chain-of-thought answer mirror what a language model truly does inside. Their 2025 study is the first feature-level causal test of that faithfulness. They use sparse autoencoders to pull out clean features, and activation patching to swap them between runs.

Two model sizes worked grade-school math word problems, with and without the written steps. Swapping a small set of step-linked features into a no-steps run lifted the answer's score in the bigger model, but not in the small one. Steps also made the bigger model's inner activity sparser and its features easier to read. The useful signal, though, spread far beyond the top few features.

Catalog record

CREATORXi Chen, Aske Plaat, Niki van Stein
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-22928

Browse the full library · About the Tower of Babel