The Readout Shortcut: Positional Number Copying Dominates Arithmetic CoT Readout in Small Language Models
Ming Liu · 2026
Read The Readout Shortcut: Positional Number Copying Dominates Arithmetic CoT Readout in Small Language Models by Ming Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22870.txt.
Introduction
Liu shows that small language models doing chain-of-thought arithmetic often just copy the last number before the answer marker, regardless of earlier reasoning. In three 1 to 3 billion parameter models on GSM8K, the gold answer's presence explained most accuracy, and wrong trailing numbers collapsed scores to near zero despite correct intermediate steps. Replacing that number recovered some accuracy, proving a positional shortcut, not genuine computation, drives much of the readout.
Catalog record
| CREATOR | Ming Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70668 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22870
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Readout Shortcut: Positional Number Copying Dominates Arithmetic CoT Readout in Small Language Models Ming Liu Abstract Chain-of-thought (CoT) prompting is necessary for arithmetic in small language models, yet shuffling its steps preserves most performance. What does CoT contribute if not logical sequencing? In three 1–3B instruction-tuned LMs on GSM8K, we isolate the answer-readout stage via prefix completion and identify a positional shortcut: the model copies whichever number occupies the trailing position before the answer delimiter, regardless of intermediate reasoning. Gold-answer presence accounts for 54–92 pp of accuracy (89–92% of each model’s teacher-forcing ceiling); even on incorrect items, the final answer matches the last CoT number 95–96% of the time. The copy channel takes precedence over retained-context completion: replacing the trailing number with a wrong value collapses accuracy to near-zero despite correct intermediates, yet removing it recovers 5–32 pp above that floor—even single-step arithmetic the model can otherwise perform is suppressed when a copyable number is present. Qwen and Llama copy novel distractors 87–95% of the time; Gemma gates selectively. Head-level ablation implicates architecture-specific head sets; the effect replicates on GSM-Symbolic. On non-arithmetic BBH tasks, shuffle retention drops sharply; at 7–8B, content-selective gating emerges. Step-level faithfulness evaluations risk conflating positional answer transport with genuine computation—a failure mode for CoT-based oversight. (a) Clean CoT s_{1} s_{2} \cdots 72 #### 72 positional copy (b) Wrong-number distractor s_{1} s_{2} \cdots 72 999 #### 999 copies wrong # Figure 1: The answer-context-gated positional readout : the model reads whichever number appears in answer-relevant context at the trailing position before the #### delimiter. (a) When the correct answer is last, the readout yields the right output. (b) In Qwen/Llama (1–3B), injecting a wrong number in answer context displaces gold and is copied 87–95% of the time; Gemma instead shows stronger content gating ( P(\text{distractor}){=}.12 – .19 ; § 4.1 ). 1 Introduction Chain-of-thought (CoT) prompting ( Wei et al. 2022 ; Kojima et al. 2022 ) is necessary: removing it crashes accuracy ( Lanham et al. 2023 ) . Yet its step order barely matters: shuffling steps retains most performance ( Madaan and Yazdanbakhsh 2022 ; Wang et al. 2023a ) . This tension is well-documented but unexplained— what in CoT drives the answer if not logical sequencing? Prior work established that CoT can be unfaithful— Turpin et al. 2023 showed biased demonstrations alter answers despite correct reasoning steps; Lanham et al. 2023 quantified unfaithfulness via early answering and paraphrase metrics; Arcuschin et al. 2025 observed similar patterns in naturalistic settings—and that filler tokens partially substitute for reasoning ( Pfau et al. 2024 ) ; recent mechanistic studies have begun tracing information flow through CoT ( Dutta et al. 2024 ) and probing which reasoning steps matter ( Bogdan et al. 2025 ) . These works document that readout can bypass reasoning but not what specific signal the readout uses, where in the network it operates, or whether it actively suppresses available computation. We provide all three. We resolve this for 1–3B instruction-tuned models on arithmetic by identifying an answer-context-gated positional readout ( Geirhos et al. 2020 ; McCoy et al. 2019 ) : the model reads whichever number appears in answer-relevant context at the trailing position before the answer delimiter, largely independent of intermediate computation. The readout requires answer-relevant framing (Appendix U ) and is correctness-independent in Qwen/Llama but content-gated in Gemma (Figure 1 ). On three architectures we show what the model reads, why shuffling preserves it (the answer token’s positional accessibility survives when logical order does not), and where it breaks (non-arithmetic tasks; content-selective retrieval at 7–8B). Gold-answer dependence is robust; the content gate —operationally, the degree to which the model rejects a novel distractor number at the trailing position, measured as 1-P(\text{distractor}) —varies from absent (Qwen: gate {\approx}0 ) to strong (Gemma: gate {\approx}.85 ; 7–8B models: gate {\geq}.70 ; § 7 ). Concurrent behavioral work ( Chen et al. 2026 ) corroborates frontier-scale shuffle-tolerance; we contribute the mechanism, which requires open-weight access. This characterization emerges from four converging lines of evidence on three architectures (Qwen2.5-1.5B-Instruct, Llama-3.2-1B-Instruct, Gemma-2-2B-it) on GSM8K ( Cobbe et al. 2021 ) ; Gemma participates fully in the corruption decomposition, shuffle hierarchy, and head-level ablation, but its position/distractor results use a teacher-forcing-passing subset (Appendix T ): (i) a corruption decomposition isolates gold-answer presence as the dominant factor (54–92 pp raw; 89–92% ceiling-corrected), and a seven-condition causal ladder shows the copy channel takes precedence over available retained-context completion (§ 3 ); (ii) the readout selects the trailing answer-context number with a sharp final-position jump ( +20 – 31 pp) and novelty-permissive copying in Qwen/Llama (.87–.95 for novel numbers) vs. selective gating in Gemma (.12–.19; § 4 ); (iii) head-level ablations reveal architecture-specific profiles from localized (Llama) to distributed (Qwen; § 5 ); (iv) the shortcut is present in base weights, replicates on GSM-Symbolic, shifts toward selectivity at 7–8B, and collapses on non-arithmetic BBH (§ 6 ). Scope and faithfulness. CoT use involves rationale generation (which may compute) and answer readout (which maps a completed prefix to the final token); our prefix-completion interventions isolate the second stage. Following Lanham et al. 2023 , we treat a readout as faithful when the answer depends on intermediate computation ( \Delta_{\text{off-copy}} ) rather than surface features ( \Delta_{\text{copy}} ). We make no claim about generation-time computation—only that the readout does not faithfully use it when a trailing answer-context number is available. This undermines the premise of step-level CoT monitors ( Lightman et al. 2023 ; Chen et al. 2025 ; Korbak et al. 2025 ) : the readout takes precedence over available retained-context completion. 2 Experimental Setup Models. Qwen2.5-1.5B-Instruct ( Yang et al. 2024 ) (28L, 1536d, 12/2 GQA heads), Llama-3.2-1B-Instruct ( Grattafiori et al. 2024 ) (16L, 2048d, 32/8 GQA heads), and Gemma-2-2B-it ( Riviere et al. 2024 ) (26L, 2304d, 8/4 GQA heads). All use grouped-query attention ( Ainslie et al. 2023 , GQA;) with per-head \mathbf{o}_{\text{proj}} (attention output projection) columns, enabling per-head ablation. The three models span distinct tokenizer families (BPE variants for Qwen/Llama; SentencePiece unigram for Gemma), so convergent findings are not attributable to shared sub-word segmentation of numbers. Gemma appears in core experiments (corruption decomposition, shuffle hierarchy, head-level ablation, base-model probe); its position/distractor results (§ 4 ) show reduced teacher-forcing fidelity under prefix-structure perturbation ( {\sim}60\% vs. {\sim}99\% on the corruption pipeline; Appendix T ). Task and protocol. GSM8K test set ( Cobbe et al. 2021 ) , first 500 problems. Baseline greedy CoT accuracy: Qwen 67.0%, Llama 45.6%, Gemma 66.2%. All experiments use teacher-forced prefix injection: a (possibly modified) CoT is injected as the beginning of the assistant turn using native chat templates, “ #### ” is appended, and the model generates the answer greedily. This is the standard protocol for causal CoT analysis, isolating the readout from generation-time confounds ( Lanham et al. 2023 ) . 1 1 1 Gemma-2-2B-it lacks native system-prompt support; we prepend the system message to the user turn. Statistical methodology. Wilson 95% CIs ( Wilson 1927 ) for proportions; McNemar’s exact test ( McNemar 1947 ) for paired contrasts; Holm–Bonferroni correction ( Holm 1979 ) within pre-declared confirmatory sub-families (Appendix S ). Auxiliary analyses (induction overlap, mean-ablation, activation patching) are