Diagnosing Memorization in Chain-of-Thought Reasoning One Token at a Time
Huihan Li You Chen Siyuan Wang Yixin He Ninareh Mehrabi, Rahul Gupta Xiang Ren · 2025
Read Diagnosing Memorization in Chain-of-Thought Reasoning One Token at a Time by Huihan Li You Chen Siyuan Wang Yixin He Ninareh Mehrabi, Rahul Gupta Xiang Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02037.txt.
Introduction
Huihan Li and six co-authors build a tool that flags, token by token, when a language model's reasoning is recall rather than thought. Their paper appeared in two thousand twenty-five. Models ace reasoning tests yet stumble when a problem shifts slightly. That hints their chains of thought lean on memorized patterns.
Their tool, STIM, tags each token in a chain with a likely source of recall. Sources split into local, mid-range, and long-range echoes of the training text. Across tasks, hard and rare cases drew more on recall than easy ones. Local recall drove most errors, accounting for much of the wrong-token share. The scores also predict which tokens in a bad step will be wrong. The tool could guide fixes for stepwise generation in general.
Catalog record
| CREATOR | Huihan Li You Chen Siyuan Wang Yixin He Ninareh Mehrabi, Rahul Gupta Xiang Ren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-02037
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diagnosing Memorization in Chain-of-Thought Reasoning One Token at a Time Huihan Li You Chen Siyuan Wang Yixin He Ninareh Mehrabi, Rahul Gupta Xiang Ren Abstract Large Language Models (LLMs) perform well on reasoning benchmarks but often fail when inputs alter slightly, raising concerns about the extent to which their success relies on memorization. This issue is especially acute in Chain-of-Thought (CoT) reasoning, where spurious memorized patterns can trigger intermediate errors that cascade into incorrect final answers. We introduce STIM , a novel framework for S ource-aware T oken-level I dentification of M emorization, which attributes each token in a reasoning chain to one of multiple memorization sources – local, mid-range, or long-range – based on their statistical co-occurrence with the token in the pretraining corpus. Our token-level analysis across tasks and distributional settings reveals that models rely more on memorization in complex or long-tail cases, and that local memorization is often the dominant driver of errors, leading to up to 67% of wrong tokens. We also show that memorization scores from STIM can be effective in predicting the wrong tokens in the wrong reasoning step. STIM offers a powerful tool for diagnosing and improving model reasoning and can generalize to other structured step-wise generation tasks. 1 1 1 https://github.com/INK-USC/STIM 1 1 footnotetext: Equal Contribution 2 2 footnotetext: Work done in Amazon AGI 1 Introduction Large Language Models (LLMs) perform well on reasoning tasks but often fail under slight input changes, raising concerns about overreliance on memorization ( Hong et al. 2025 ; Lou et al. 2024 ; Jin et al. 2024 ; Salido et al. 2025 ) . Long Chain-of-thought (CoT) ( Wei et al. 2022 ) chains are especially vulnerable, as spurious memorization can introduce early errors that derail final answers. As inference-time scaling encourages longer CoTs, detecting token-level memorization is critical for assessing reasoning reliability, particularly under distributional shifts from frequent to rare inputs ( Xie et al. 2024 ; Prabhakar et al. 2024 ) . Figure 1: Cascading errors in Chain-of-thought (CoT) reasoning can stem from a single mis-predicted token, often influenced by incorrect memorization of pretraining data. This motivates our study on the impact of token-level memorization. We argue that memorization in long Chain-of-thought generations must be identified at the token level rather than the sequence level. A single faulty step can cause cascading errors, often stemming from a few erroneous tokens (Figure 1 ). Identifying these tokens and whether they result from memorization is essential. Moreover, we argue that accurately measuring token-level memorization requires accounting for multiple sources of influence , including both the input prompt and prior output tokens, which jointly shape each token’s generation (Table 1 ). Figure 2: STIM quantifies local, mid-range, and long-range memorization at each token in a faulty reasoning step and identifies the dominant source. STIM is helpful for detecting error tokens. Examples of Simultaneous Memorization Sources of Different Strength Question : Here is a list: [pitanga, pitanga, yumberry, yumberry, pitanga, yumberry, pitanga, yumberry, pitanga, pitanga]. How many times does ’pitanga’ appear on it? Model Output : 1. The first element is ’pitanga’. … 10. The tenth element is ’pitanga’. Now, counting the occurrences of ’pitanga’: We have ’pitanga’ appearing 5 times at positions 1, 2, 5, 7, and 10. So the answer is 5. Analysis : Token 5 is incorrect and has the highest influence from local context, where the n-gram “appearing 5” has a high pretraining frequency. STIM scores are: local = 0.81 , mid = 0.24, long = 0.03, suggesting that the model was influenced by locally frequent patterns rather than actual counting. Question : What is ( 100101100 - 100101100 * 0.1) * 111110100 + 100101100 * 1111 / 1100100 * 111110100 * 0.1 + (100101100 - 100101100 * 0.1 - 100101100 * 1111 / 1100100) * 111110100 equal to? (base-2) Model Output : To solve this, we first handle the multiplication and division in base-2, converting to base-10 as necessary: 1. (100101100\times 0.1_{\text{base-2}}=10010110_{\text{base-2}}) (which is 180 in base-10) … So the answer is 26807.536. Analysis : Token 180 is incorrect and has the highest influence from input tokens, having higher pretraining co-occurrence frequency with “111110100”, “*” and “100101100” in the input. STIM scores are local=-0.19, mid=0.09, long=0.156 , indicating long-range memorization being the primary influence. Table 1: Each error token is affected simultaneously by different memorization sources, but some sources may be stronger than others. Shown are examples with local context ( local ) and input prompt ( long ) having maximum influence, with tokens associated with the source highlighted. How we obtain STIM scores for each source is explained in Section 4 . Prior approaches are insufficient for analyzing token-level memorization and how memorization patterns from different sources shift under distributional change: existing metrics do not target memorization at the level of individual tokens, instead reporting a single score for the entire sequence or final answer. Moreover, they either focus solely on memorization in the output sequence ( McCoy et al. 2023 ; Merrill et al. 2024 ; Lu et al. 2024 ) or assess the influence of memorization from the input ( Carlini et al. 2022 ; Biderman et al. 2023 ; Li et al. 2025 ; Wang et al. 2024 ) , without accounting for multiple sources of influence on token-level memorization. To address these gaps, we propose STIM ( S ource-aware T oken-level I dentification of M emorization), a framework that captures token-level memorization by tracing influences from both the input and prior outputs on erroneous reasoning steps. For each token, STIM computes the strength of three memorization sources: (1) local , from frequent continuations of immediately preceding tokens; (2) long-range , from frequent co-occurrence with prompt tokens; and (3) mid-range , when the model generates the target token when conditioned only on a prefix of the generation, we identify tokens that frequently co-occur with the target token in pretraining. STIM offers a fine-grained view of multi-source memorization and its strength at each token. We begin our analysis by using STIM to uncover broad memorization trends across tasks, input distributions, and correctness. As reasoning complexity increases, models exhibit greater reliance on memorization. Distribution shifts toward rare or atypical inputs also lead to stronger memorization signals. Interestingly, while memorization often supports correct answers in base settings, it more frequently contributes to errors in long-tail scenarios, suggesting defective recall when faced with unfamiliar contexts. To demonstrate the utility of our framework, we apply it to the task of identifying erroneous tokens in erroneous reasoning steps. By tracing the dominant source of memorization for each incorrect token, we find that local memorization (continuations driven by immediately preceding tokens) is the most common cause of error (up to 67%). However, under distribution shift, complex tasks show a marked decline in local memorization-driven mistakes, implying reduced reliance on familiar patterns. Finally, we assess the effectiveness of STIM in pinpointing erroneous tokens via Precision@k and Recall@k, showing that high memorization scores are strong indicators of reasoning failures. Our novel STIM framework is the first to enable fine-grained, token-level memorization identification in long reasoning chains, considering multiple influence sources from both input and generated context. Through systematic evaluation across diverse Chain-of-thought tasks, we demonstrate its crucial role