Chain-of-Thought Tokens are Computer Program Variables

Fangwei Zhu, Peiyi Wang, Zhifang Sui · 2025

Read Chain-of-Thought Tokens are Computer Program Variables by Fangwei Zhu, Peiyi Wang, Zhifang Sui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04955.txt.

Introduction

Zhu, Wang, and Sui probe what chain-of-thought text actually does inside a language model in this 2025 paper. Models reason better when they write out intermediate steps first. Why those steps help was unclear.

The team tested two tasks that need stored intermediate values: multi-digit multiplication and dynamic programming. Keeping only the tokens that hold intermediate results gave nearly the same accuracy as the full chain. Storing those results in a hidden latent form also worked fine. When they tampered with a stored value, later steps and the final answer shifted to match. The authors conclude the tokens act like program variables, while warning about shortcuts and step-size limits.

Catalog record

CREATORFangwei Zhu, Peiyi Wang, Zhifang Sui
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-04955

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Chain-of-Thought Tokens are Computer Program Variables Fangwei Zhu, Peiyi Wang, Zhifang Sui Abstract Chain-of-thoughts (CoT) requires large language models (LLMs) to generate intermediate steps before reaching the final answer, and has been proven effective to help LLMs solve complex reasoning tasks. However, the inner mechanism of CoT still remains largely unclear. In this paper, we empirically study the role of CoT tokens in LLMs on two compositional tasks: multi-digit multiplication and dynamic programming. While CoT is essential for solving these problems, we find that preserving only tokens that store intermediate results would achieve comparable performance. Furthermore, we observe that storing intermediate results in an alternative latent form will not affect model performance. We also randomly intervene some values in CoT, and notice that subsequent CoT tokens and the final answer would change correspondingly. These findings suggest that CoT tokens may function like variables in computer programs but with potential drawbacks like unintended shortcuts and computational complexity limits between tokens. The code and data are available at https://github.com/solitaryzero/CoTs_are_Variables . 1 Introduction Chain-of-thoughts (CoT) Wei et al. 2022 is a widely adopted technique that greatly boosts the capability of large language models (LLMs) in reasoning tasks like solving mathematical problems Shao et al. 2024 ; Wang et al. 2024 or generating codes Guo et al. 2025 . By requiring language models to generate intermediate steps before reaching the final result, chain-of-thoughts enables LLMs to perform advanced reasoning, and thus significantly outperforms standard supervised learning methods. Various methods have been explored to unlock the ability of chain-of-thought reasoning in LLMs, for example designing prompts Wei et al. 2022 ; Khot et al. 2022 ; Zhou et al. 2022 , instruction tuning Yue et al. 2023 ; Yu et al. 2023 or reinforcement learning Havrilla et al. 2024 ; Wang et al. 2024 ; Guo et al. 2025 . There have been theoretical studies on the efficacy of chain-of-thoughts Deng et al. 2024 ; Li et al. 2024 ; Chen et al. 2024 . These studies reveal that while it is exponentially difficult for language models to solve compositional problems requiring serial computations, CoT could help models solve problems under multinominal complexity. More interestingly, CoT tokens do not need to fall in the “language space”, using latent vectors could also enable language models to perform complex reasoning Hao et al. 2024 , indicating that CoTs are more than mere thought traces. The mechanism of how CoT works, and the role of CoT tokens are still not fully explored. In this paper, we propose a novel hypothesis that CoT tokens function like computer program variables . To be specific, the tokens in CoT store intermediate values that will be used in subsequent computations, and these values are partially mutable to control the final output. As long as the important intermediate values are calculated and stored, the CoT that leads to the final answer could be represented in different forms. To verify the hypothesis, we conduct empirical study on two types of problems that both require long-chain serial computations: multi-digit multiplication and dynamic programming. By comparing the performance of vanilla prompting with CoT prompting, we confirm that CoT is crucial for these problems. However, we also find that removing non-result tokens would not bring performance drops, which means that tokens storing intermediate values matter more in chain-of-thoughts. We further explore whether intermediate values could be represented in different forms. We attempt to compress consequent number digits within a single latent vector, and experimental results show that it does not detriment the model performance. This phenomenon indicates that the existence, rather than the form, of intermediate values matters more to language models. However, when the degree of compression exceeds a certain limit of language models’ capacity, it would lead to failure in reasoning. To further confirm that the intermediate values are causally connected with the output, we intervene in some tokens in CoT, replacing them with random values. It can be observed that LLMs will ignore previous steps, and use the intervened value to perform subsequent computations, supporting that CoT tokens are causally related with the final result. We conclude that CoT tokens function like the variables in computer programs. To sum up, in this paper we empirically study the function of CoT tokens, and find that: (1) The role of CoT tokens is similar to variables in computer programs as they store intermediate values used in subsequent computations; (2) The intermediate values could be stored in CoT tokens with different forms; (3) The values in CoT tokens are causally related to the final output and could be intervened like program variables. These findings are helpful in understanding alternative forms of CoT, and could assist in designing more concise CoTs. 2 Preliminary 2.1 Chain-of-Thoughts Chain-of-thoughts (CoT) Wei et al. 2022 is a technique commonly used in decoder-only transformers. Given the input text x , CoT attempts to generate intermediate steps z prior to the final answer y . In other words, instead of modeling the probability distribution P(y|x) , CoT attempts to model the joint distribution P(y,z|x)=P(z|x)P(y|x,z) . For convenience, we use two special tokens <COT> and </COT> to separate CoT tokens from the final result in our experiments. 2.2 Compositional Tasks It has been noticed that LLMs may fail on seemingly trivial problems like multi-digit multiplication. The commonality of these problems is that they need strict multi-hop reasoning to derive correct predictions, which requires language models to perform step-to-step reasoning like human intelligence. In this paper, we choose two representative tasks to study the role of CoT tokens: Algorithm 1 Digit-wise multiplication 0: Integer a and b 0: Value of a*b Partial = [ ] for Digit b[i] in b do carry \leftarrow 0 for Digit a[i] in a do x \leftarrow a[i]*b[i] + carry digit \leftarrow x/10 carry \leftarrow x mod 10 end for res \leftarrow Combine digits and last carry Add res to Partial end for while Len(Partial) > 1 do x \leftarrow Partial[0] + Partial[1] Partial \leftarrow [x] + Partial[2:] end while return Partial[0] Multi-digit Multiplication Calculating the multiplication result of two multi-digit numbers (x,y) requires executing multiple operations based on procedural rules Dziri et al. 2023 . A commonly adopted solution is the long-form multiplication algorithm, which iteratively calculates the digit-wise multiplication result and adds them up to get the final result. We describe the algorithm in Algorithm 1 , see Appendix A for prompt and dataset construction details. Algorithm 2 Maximum path sum in a grid 0: A m*n matrix W 0: Maximum weight sum s on path DP \leftarrow m*n matrix filled with 0 for i in range( m ) do for j in range( n ) do DP[ i ][ j ] = \max (DP[ i-1 ][ j ], DP[ i ][ j-1 ]) + W[ i ][ j ] end for end for return DP[ m-1 ][ n-1 ] Dynamic Programming Dynamic programming (DP) is an algorithmic paradigm that breaks down complicated problems into simpler sub-problems, and then recursively solves these sub-problems. In our experiments, we use the “Maximum Path Sum in a Grid” problem: Given a m\times n grid filled with non-negative numbers where only moving downward and rightward is allowed, find a path from top left

Read the complete text