When Chain-of-Thought Fails, the Solution Hides in the Hidden States

Houman Mehrafarin Amit Parekh Ioannis Konstas · 2026

Read When Chain-of-Thought Fails, the Solution Hides in the Hidden States by Houman Mehrafarin Amit Parekh Ioannis Konstas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23351.txt.

Introduction

Mehrafarin, Parekh, and Konstas probe whether chain-of-thought tokens carry task information, using activation patching on GSM8K. Moving hidden states from a CoT run into a direct-answer run lifts accuracy above both direct prompting and the trace, even when that trace is wrong. Useful information clusters in middle to late layers and earlier in the trace, with language tokens steering reasoning and maths tokens holding answer-near content. Patched outputs run shorter yet beat full traces, so complete chains are not always needed.

Catalog record

CREATORHouman Mehrafarin Amit Parekh Ioannis Konstas
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45360
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23351

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Chain-of-Thought Fails, the Solution Hides in the Hidden States Houman Mehrafarin Amit Parekh Ioannis Konstas Abstract Whether intermediate reasoning is computationally useful or merely explanatory depends on whether chain-of-thought (CoT) tokens contain task-relevant information. We present a mechanistic causal analysis of CoT on GSM8K using activation patching : transferring token-level hidden states from a CoT generation to a direct-answer run for the same question, then measuring the effect on final-answer accuracy. Across models, generating after patching yields substantially higher accuracy than both direct-answer prompting and the original CoT trace, revealing that individual CoT tokens can encode sufficient information to recover the correct answer, even when the original trace is incorrect. This task-relevant information is more prevalent in correct than incorrect CoT runs and is unevenly distributed across tokens, concentrating in mid-to-late layers and appearing earlier in the reasoning trace. Moreover, patching language tokens such as verbs and entities carry task-solving information that steers generation toward correct reasoning, whereas mathematical tokens encode answer-proximal content that rarely succeeds. Patched outputs are often shorter and yet exceed the accuracy of a full CoT trace, suggesting complete reasoning chains are not always necessary. Together, these findings demonstrate that CoT encodes recoverable, token-level problem-solving information, offering new insight into how reasoning is represented and where it breaks down. 1 1 1 Code is available at https://github.com/hmehrafarin/when-cot-fails 1 Introduction Figure 1: Illustration of our patching framework. For the same question, we transfer hidden states from individual tokens in a source CoT generation to the final position of a direct-answer target run at the same layer. The resulting change in model output and final-answer accuracy quantifies each token’s causal contribution. This reveals that some CoT tokens carry recoverable task-solving information even when the original CoT run is incorrect. Chain-of-Thought (CoT) prompting improves performance on multi-step reasoning tasks by eliciting intermediate reasoning traces before the final answer ( 30 ; 12 ; 32 ) . On arithmetic benchmarks such as GSM8K ( 4 ) , this consistently yields higher accuracy than direct-answer prompting ( 30 ; 12 ) . However, it remains unclear whether generated CoT tokens encode information useful for producing the correct final answer, and whether such information persists even when the CoT generation itself is incorrect. Understanding what the CoT hidden states contain is important for explaining whether intermediate reasoning is computationally useful rather than merely explanatory ( 26 ; 13 ; 36 ) , and for understanding how reasoning unfolds during inference ( 8 ; 28 ) . We present a mechanistic analysis of CoT reasoning on GSM8K arithmetic problems. We hypothesise that CoT tokens differ in how much task-relevant information their hidden states encode for the correct final answer. To test this, we use a causal intervention framework ( 27 ; 17 ; 23 ) that patches hidden states from individual tokens in a CoT generation (e.g., “think step-by-step”) into the same layer of a direct-answer run (e.g., “do not show steps”) for the same arithmetic problem, measuring the effect on final-answer accuracy (see Figure 1 ). We show that CoT hidden states often encode sufficient task-relevant information to produce the correct answer in both LLaMA 3.1 8B and Qwen 2.5 7B, including where the original CoT is incorrect. Based on this, we find that correct CoT generations contain broadly recoverable information in their hidden states —such that continuing generation after patching yields the correct final answer—whereas incorrect generations lack this property, suggesting reasoning failures occur because task-relevant information is either absent or less prevalent across the reasoning trace. Within these traces, recoverable information concentrates in mid-to-late layers and earlier positions , and is unevenly distributed across token roles , with verb tokens yielding more successful patches than operands or operators, indicating recoverable information is not confined to computational tokens when language is involved. Patching language tokens (e.g., verbs and entities) elicit reasoning that carries information on how to solve the task, whereas mathematical tokens (e.g., operands, numbers) tend to bypass reasoning entirely, producing the final answer that is rarely correct. Notably, patching often leads to shorter outputs which still exceed original CoT accuracy , suggesting a full CoT reasoning trace is not always necessary for strong performance. Together, these findings demonstrate that CoT encodes recoverable, token-level problem-solving information in its hidden states, offering a new lens for understanding where reasoning succeeds, where it breaks down, and how much of the generated trace is actually necessary. 2 Related Work CoT Prompting and Reasoning CoT prompting improves performance on multi-step reasoning benchmarks by encouraging models to generate intermediate reasoning steps before producing a final answer ( 30 ; 12 ) . Subsequent work extends this paradigm through methods such as self-consistency, which aggregates answers across sampled reasoning paths ( 29 ; 24 ) , and structured reasoning frameworks that guide or search over intermediate steps ( 31 ; 34 ) . Despite these gains, several studies question whether generated reasoning traces faithfully reflect the model’s internal computation. In particular, 25 and 20 show that final answers can diverge from the generated chains, suggesting that key aspects of the reasoning process may occur independently of the CoT generated by the model. Mechanistic and Causal Interpretability Mechanistic interpretability seeks to explain model behaviour by identifying internal circuits responsible for specific outputs ( 19 ; 6 ; 5 ; 18 ; 28 ) . Complementary causal approaches analyse how intermediate representations contribute to predictions through targeted interventions such as activation patching ( 21 ; 7 ; 17 ; 35 ) , enabling localisation of representations that causally influence model outputs. Interpretability for Reasoning Recent work applies these techniques to reasoning tasks: identifying neurons, attention heads, and circuits involved in arithmetic and symbolic computation ( 23 ; 33 ; 2 ; 11 ) . These studies further show how reasoning failures can arise from incorrect intermediate states and how targeted interventions can help to repair them ( 14 ; 1 ) . Methodologically, our work is related to activation-level probing approaches such as Patchscopes ( 9 ) , which use hidden-state interventions to inspect encoded information. However, Patchscopes does not analyse reasoning chains or their causal role in final answers. Closer in objective is 3 , who study CoT using sparse autoencoders and activation patching to identify latent reasoning features and transfer them into non-CoT runs. In contrast, we study the generated CoT tokens , asking which tokens and positions encode task-relevant information and whether these representations causally contribute to producing the correct final answer. 3 Methodology 3.1 Model & Dataset We use LLaMA 3.1 8B-Instruct 10 and Qwen 2.5 7B-Instruct 22 as our base models 2 2 2 We use two representative open-weight instruction-tuned LLMs to study how CoT is internally represented. We refer to them as LLaMA 3.1 8B and Qwen 2.5 7B for brevity. and evaluate on GSM8K 4 , a multi-step arithmetic reasoning benchmark containing 1, 319 test examples. We select this dataset for the following reasons: CoT prompting yields substantial improvements over direct answering (Table 2 ), solving each problem requires accumulating intermediate results through arithmetic operations, making it well-suited for studying how reasoning unfolds across token positions. 3.2 Causal Intervention and Task Setup We compare two zero-shot prompting conditions for the

Read the complete text