Think-to-Talk or Talk-to-Think? When LLMs Come Up with an Answer in Multi-Step Arithmetic Reasoning
Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Shusaku Sone, Masaya Taniguchi, Ana Brassard, Keisuke Sakaguchi, Kentaro Inui · 2024
Read Think-to-Talk or Talk-to-Think? When LLMs Come Up with an Answer in Multi-Step Arithmetic Reasoning by Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Shusaku Sone, Masaya Taniguchi, Ana Brassard, Keisuke Sakaguchi, Kentaro Inui (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01113.txt.
Introduction
Keito Kudo and seven co-authors ask when a language model settles on an answer during step by step math reasoning. Some fear the written steps are a story made after a snap call. They trace inner signals while models work through controlled sums.
Models had not fixed the answer when the prompt landed. They built sub answers on the fly as each step was written. Those steps thus push the final answer in a real causal way. The chain can be read as a true trace of inner work, at least for this math kind.
Catalog record
| CREATOR | Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Shusaku Sone, Masaya Taniguchi, Ana Brassard, Keisuke Sakaguchi, Kentaro Inui |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01113
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Think-to-Talk or Talk-to-Think? When LLMs Come Up with an Answer in Multi-Step Arithmetic Reasoning Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Shusaku Sone, Masaya Taniguchi, Ana Brassard, Keisuke Sakaguchi, Kentaro Inui Abstract This study investigates the internal reasoning process of language models during arithmetic multi-step reasoning, motivated by the question of when they internally form their answers during reasoning. Particularly, we inspect whether the answer is determined before or after chain-of-thought (CoT) begins to determine whether models follow a post-hoc Think-to-Talk mode or a step-by-step Talk-to-Think mode of explanation. Through causal probing experiments in controlled arithmetic reasoning tasks, we found systematic internal reasoning patterns across models in our case study; for example, single-step subproblems are solved before CoT begins, and more complicated multi-step calculations are performed during CoT. 1 Introduction An explanation may be produced in two modes: as a post-hoc explanation to a predetermined conclusion ( Think-to-Talk ) or by process of reaching a conclusion while explaining ( Talk-to-Think ). An analogy applies to large language models (LLMs) using chain-of-thought ( Wu et al. 2023a , CoT;) reasoning: is the output a post-hoc explanation, or does it reflect step-by-step solving? To answer this question, we focus on multi-step arithmetic reasoning tasks as a case study and apply causal probing to model internals at each layer during each timestep to determine when the answer is reached in CoT-style reasoning. In our experiments, we prepared a controlled testbed of symbolic arithmetic reasoning tasks and observed whether trained classifiers, given a set of hidden representations, could accurately predict the values of all variables in the test instance. By comparing accuracies across each timestep, we can observe at which point the model’s internals start being informative to the probes, illustrating the model’s internal reasoning flow. Our experiments with linear probes reveal intriguing properties of the model’s internal reasoning process. Specifically, not all the reasoning is performed while generating CoT chains ( talk-to-think mode). Some sub-problems were already resolved immediately after the problem was shown ( think-to-talk mode). Then, we further found that this think-to-talk mode was not randomly adopted; rather, there were systematic tendencies where single-step reasoning (e.g., A=2+3,A=? ) is performed before CoT, and multi-step reasoning (e.g., A=2+3,B=A+3,B=? ) follows after CoT (Figure 2 ). Figure 1: Using linear probes, we investigated at which time during the LLM’s reasoning process it is possible to determine the values of each variable, illustrating the model’s reasoning process. To enhance the above findings, we conducted causal intervention analyses to inspect whether the pre-resolved sub-answers are referred to in models deriving the final answer. The analyses revealed that such predetermined (sub-)answers indeed impacted the model’s final answer ( section 4.1 ), but their causal relationship was somewhat indirect ( section 4.2 ). These reveal a systematic internal reasoning pattern between Think-to-Talk and Talk-to-Think modes in arithmetic reasoning tasks. This extends existing mechanistic interpretability results of LLMs regarding arithmetic problem solving Stolfo et al. 2023 ; Zhu et al. 2025 , specifically from simple arithmetic operation to multi-step reasoning outputs. 1 1 1 Code and data are available at https://github.com/keitokudo/TTT Figure 2: Probes predict each variable’s value from the LMs’ hidden states at each timestep \times layer. The upper heat map shows the probe accuracy for predicting the value of A . The lower one shows the prediction accuracy for the value of B . A high accuracy indicates where the model completes the corresponding calculation ( section 3.3 ). This result shows the analysis conducted on Qwen2.5-7B for task Level 3 . Level Input Output #Step #Stack #Dist. 1 {\underline{\mathrm{A}=1+\mathrm{B}}}_{\hskip 1.42271pt\mathbf{-3}} , {\underline{\mathrm{B}=2}}_{\hskip 1.42271pt\mathbf{-2}} ; {\underline{\mathrm{A}=?}}_{\hskip 1.42271pt\mathbf{-1}} {\underline{\mathrm{A}=1+\mathrm{B}}}_{\hskip 1.42271pt\mathbf{0}} , {\underline{\mathrm{B}=2}}_{\hskip 1.42271pt\mathbf{1}} , {\underline{\mathrm{A}=1+\mathrm{B}}}_{\hskip 1.42271pt\mathbf{2}} , {\underline{\mathrm{A}=1+2}}_{\hskip 1.42271pt\mathbf{3}} , {\underline{\mathrm{A}=3}}_{\hskip 1.42271pt\mathbf{4}} 1 1 0 2 {\underline{\mathrm{A}=2+3}}_{\hskip 1.42271pt\mathbf{-3}} , {\underline{\mathrm{B}=1+\mathrm{A}}}_{\hskip 1.42271pt\mathbf{-2}} ; {\underline{\mathrm{B}=?}}_{\hskip 1.42271pt\mathbf{-1}} {\underline{\mathrm{B}=1+\mathrm{A}}}_{\hskip 1.42271pt\mathbf{0}} , {\underline{\mathrm{A}=2+3}}_{\hskip 1.42271pt\mathbf{1}} , {\underline{\mathrm{A}=5}}_{\hskip 1.42271pt\mathbf{2}} , {\underline{\mathrm{B}=1+\mathrm{A}}}_{\hskip 1.42271pt\mathbf{3}} , {\underline{\mathrm{B}=1+5}}_{\hskip 1.42271pt\mathbf{4}} , {\underline{\mathrm{B}=6}}_{\hskip 1.42271pt\mathbf{5}} 2 0 0 3 {\underline{\mathrm{A}=1+\mathrm{B}}}_{\hskip 1.42271pt\mathbf{-3}} , {\underline{\mathrm{B}=2+3}}_{\hskip 1.42271pt\mathbf{-2}} ; {\underline{\mathrm{A}=?}}_{\hskip 1.42271pt\mathbf{-1}} {\underline{\mathrm{A}=1+\mathrm{B}}}_{\hskip 1.42271pt\mathbf{0}} , {\underline{\mathrm{B}=2+3}}_{\hskip 1.42271pt\mathbf{1}} , {\underline{\mathrm{B}=5}}_{\hskip 1.42271pt\mathbf{2}} , {\underline{\mathrm{A}=1+\mathrm{B}}}_{\hskip 1.42271pt\mathbf{3}} , {\underline{\mathrm{A}=1+5}}_{\hskip 1.42271pt\mathbf{4}} , {\underline{\mathrm{A}=6}}_{\hskip 1.42271pt\mathbf{5}} 2 1 0 4 {\underline{\mathrm{A}=1+\mathrm{B}}}_{\hskip 1.42271pt\mathbf{-4}} , {\underline{\mathrm{B}=2+3}}_{\hskip 1.42271pt\mathbf{-3}} , {\underline{\mathrm{C}=4+5}}_{\hskip 1.42271pt\mathbf{-2}} ; {\underline{\mathrm{A}=?}}_{\hskip 1.42271pt\mathbf{-1}} {\underline{\mathrm{A}=1+\mathrm{B}}}_{\hskip 1.42271pt\mathbf{0}} , {\underline{\mathrm{B}=2+3}}_{\hskip 1.42271pt\mathbf{1}} , {\underline{\mathrm{B}=5}}_{\hskip 1.42271pt\mathbf{2}} , {\underline{\mathrm{A}=1+\mathrm{B}}}_{\hskip 1.42271pt\mathbf{3}} , {\underline{\mathrm{A}=1+5}}_{\hskip 1.42271pt\mathbf{4}} , {\underline{\mathrm{A}=6}}_{\hskip 1.42271pt\mathbf{5}} 2 1 1 5 {\underline{\mathrm{A}=1+\mathrm{B}}}_{\hskip 1.42271pt\mathbf{-4}} , {\underline{\mathrm{B}=2+\mathrm{C}}}_{\hskip 1.42271pt\mathbf{-3}} , {\underline{\mathrm{C}=1+2}}_{\hskip 1.42271pt\mathbf{-2}} ; {\underline{\mathrm{A}=?}}_{\hskip 1.42271pt\mathbf{-1}} {\underline{\mathrm{A}=1+\mathrm{B}}}_{\hskip 1.42271pt\mathbf{0}} , {\underline{\mathrm{B}=2+\mathrm{C}}}_{\hskip 1.42271pt\mathbf{1}} , {\underline{\mathrm{C}=1+2}}_{\hskip 1.42271pt\mathbf{2}} , {\underline{\mathrm{C}=3}}_{\hskip 1.42271pt\mathbf{3}} , {\underline{\mathrm{B}=2+\mathrm{C}}}_{\hskip 1.42271pt\mathbf{4}} , {\underline{\mathrm{B}=2+3}}_{\hskip 1.42271pt\mathbf{5}} , {\underline{\mathrm{B}=5}}_{\hskip 1.42271pt\mathbf{6}} , {\underline{\mathrm{A}=1+\mathrm{B}}}_{\hskip 1.42271pt\mathbf{7}} , {\underline{\mathrm{A}=1+5}}_{\hskip 1.42271pt\mathbf{8}} , {\underline{\mathrm{A}=6}}_{\hskip 1.42271pt\mathbf{9}} 3 2 0 Table 1: Examples of arithmetic reasoning tasks used in our experiments at each complexity level. #Step indicates the number of required operations to reach the final answer. #Stack indicates how many variables’ values are not immediately determined in their first appearing equation. #Dist. is the number of unnecessary distractor equations. The number (e.g., \mathbf{-3} ) indicated in the lower right corner of each equation represents the equation’s position. This position is used as a reference point for calculating t^{*}_{\mathrm{eq}} in section 3.2 . 2 General settings 2.1 Arithmetic problems We prepared a dataset of multi-step arithmetic problems similarly to Kudo et al. 2023 and Yu 2025 . Each sample is a string of assignments (e.g., A=1 ) and operations (e.g., B=1+3 or B=1+A ) ending with a query for a variable’s value (e.g., B=? ). We also defined five complexity levels, depending on (i) how many equations need to be resolved to reach the answer (#Step in Table 1 ), (ii) how many variables’ values cannot be immediately resolved (and thus pended to a stack) in their first appearance when incrementally reading the problem from left to right (#Stack), (iii) and the number of unnecessary distractor equations (#Dist.). For example, in the Level 5 example in Table 1 , where #Step is three and #Stack is two, calculating A requires at least three steps of reasoning: C(=1+2)=3 , B(=2+3)=5 , and then A(=1+5)=6 , and two variables need to be resolved before reaching A : B and C . Notation. Formally, let v denote a variable name sampled from the 26 letters of the English alphabet \Sigma=\{\text{a},\text{b},\text{c},\dots,\text{z}\} , and d a number sampled from the set of decimal digits D=\{0,1,2,\ldots,9\} . Each instance consists of multiple equations [e_{1},e_{2},\cdots,e_{n}] followed by a final query q . Each equation follows the format v=d , v=d\pm d , or v=d\pm v . We denote i -th variable to appear within an instance from the left as v_{i} . E.g., in the Level 5 example in Table 1 , v_{1}=\texttt{A} , v_{2}=\texttt{B} , and v_{3}=\texttt{C} . 2 2 2 For ease of reading, all examples throughout this paper use the uppercased variables A , B , C , and only the operator + . The value assigned to a variable v_{i} is denoted as \$\{v_{i}\}\in D . Generation rules. We ensure that \$\{v_{i}\} for any v_{i} is also a single-digit number, and \$\{v_{i}\} is constant within the same instance (i.e., we exclude cases such as A=1+2,A=B+2,B=6 ). All samples of the same complexity level follow the exact same format except for the actual numbers, variable names, and operators, meaning that \$\{v_{i}\} in each level can be obtained with exactly the same abstract procedure.