Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models

Peijie Liu, Fengli Xu, Yong Li · 2025

Read Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models by Peijie Liu, Fengli Xu, Yong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06008.txt.

Introduction

Liu, Xu, and Li ask when step-by-step prompts help big chat tools in this twenty twenty-five paper. Such prompts lift scores on hard tasks, yet the lift swings from task to task. The team ties that swing to the shape of the tool's word odds as it writes.

Inside, two marks built from those odds flag tasks where step prompts should pay off. A blend of the marks sorts tasks with truth near nine in ten in their tests. A live switch then picks step prompts or plain replies task by task. That switch cut word use by more than a third at the same truth grade. Closed tools can borrow the pick rule taught on open ones.

Catalog record

CREATORPeijie Liu, Fengli Xu, Yong Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06008

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models Peijie Liu, Fengli Xu, Yong Li Abstract Chain-of-Thought (CoT) technique has proven effective in improving the performance of large language models (LLMs) on complex reasoning tasks. However, the performance gains are inconsistent across different tasks, and the underlying mechanism remains a long-standing research question. In this work, we make a preliminary observation that the monotonicity of token probability distributions may be correlated with the gains achieved through CoT reasoning. Leveraging this insight, we propose two indicators based on the token probability distribution to assess CoT effectiveness across different tasks. By combining instance-level indicators with logistic regression model, we introduce Dynamic CoT, a method that dynamically select between CoT and direct answer. Furthermore, we extend Dynamic CoT to closed-source models by transferring decision strategies learned from open-source models. Our indicators for assessing CoT effectiveness achieve an accuracy of 89.2%, and Dynamic CoT reduces token consumption by more than 35% while maintaining high accuracy. Overall, our work offers a novel perspective on the underlying mechanisms of CoT reasoning and provides a framework for its more efficient deployment. The code can be found at https://github.com/tsinghua-fib-lab/Token_Signature . 1 Introduction Chain-of-Thought (CoT) prompting ( Wei et al. 2022 ) has become a widely adopted technique for enhancing the reasoning capabilities of large language models(LLMs). By incorporating examples of CoT reasoning in a few-shot prompt ( Wei et al. 2022 ) , CoT can be effectively triggered, and the ability of LLMs to solve various complex problems is improved by decomposing the problem step by step ( Wang et al. 2022a ) , while also providing detailed and interpretable explanations ( Lanham et al. 2023 ) . Inspired by CoT reasoning, OpenAI 2024a has introduced the concept of test-time scaling ( Xu et al. 2025 ) , which suggests that the reasoning capabilities of LLMs can be enhanced with more time spent thinking (test-time compute). For many problems such as mathematical word problems and symbolic reasoning, CoT is generally considered to be an effective method ( Chae et al. 2024 ) ( Qi et al. 2024 ) . However, recent studies have shown that CoT prompting does not consistently improve performance across all tasks, and its effectiveness varies depending on the problem domain. As illustrated in Figure 1 , CoT’s performance varies across different task categories. Moreover, in symbolic tasks, which are considered to be tasks where CoT is generally effective ( Sprague et al. 2024 ) , such as ContextHub-abductive and ContextHub-deductive ( Hua et al. 2024 ) , the significance of CoT gain also varies. In non-mathematical fields, for example, CoT has been found to be less effective ( Kambhampati et al. 2024 ) and may even result in negative performance outcomes ( Wang et al. 2024 ) . Sprague et al. 2024 conducted a meta-analysis of CoT-related studies and experiments ( Sprague et al. 2024 ) , revealing that CoT is predominantly effective for mathematical and symbolic reasoning tasks, with limited or no improvements for other types of problems. However, this analysis ( Sprague et al. 2024 ) focused solely on thematic trends and did not fully explore the underlying mechanisms driving CoT’s effectiveness. While the effectiveness of CoT across different problems and models can be generally inferred from the task category, it remains inconsistent and lacks a definitive measure of effectiveness. Consequently, our work is motivated by two main goals: to explore the underlying mechanisms of CoT reasoning and to develop a task-level method for evaluating its effectiveness . Figure 1: Average zero-shot CoT accuracy, direct answer (DA) accuracy, and CoT Gain across five benchmark categories for four open-source models. The benchmarks highlighted in green indicate that CoT is significantly greater than 0 (p<0.05). The categories include: Mathematical, Symbolic, Knowledge, Soft Reasoning, and Commonsense. The results show that CoT does not consistently lead to performance gain. Additionally, CoT’s performance varies across different question types and task categories, and even within the same category, its effectiveness is inconsistent (with varying significance levels). This highlights that the utility of CoT cannot be solely determined by the question category. In this paper, we look at a novel perspective of the LLM decoding process. Instead of focusing on the most probable next token, we look at the token probability distribution and how it changes as the number of tokens scales, which is what we defined as Token Signature . Specifically, we use standard prompts (i.e., questions only) to elicit responses and observe that the probability distribution of the initial token in the model’s greedy decoding path is highly variable and closely correlated with CoT gain. Leveraging this insight, we develop two indicators based on the token probability distribution and Spearman Correlation (SC) ( Wissler 1905 ) : Instance SC and Aggregated SC . These indicators quantify CoT effectiveness at the benchmark level. Secondly, we apply Instance-level SC to individual instances across different models and combine a small number of benchmark samples for classification, which allows us to dynamically select between CoT and direct answer. We refer to this approach as Dynamic CoT . Finally, we determine the best answer type (CoT or direct answer) by ensembling the results of a small open-source model at the question level, which is then transferred to a larger closed-source model for evaluation. We test the effectiveness of our method on 12 well-known benchmarks, including GSM8K ( Cobbe et al. 2021 ) , MultiArith ( Roy & Roth 2016 ) , CommonsenseQA ( Talmor et al. 2018 ) , and LSAT ( Zhong et al. 2023 ) , etc. We demonstrate its generalization across four closed-source models and two open-source models. At the benchmark level, we introduce two indicators that effectively predict the applicability of CoT. The positive and negative values of these indicators are closely correlated with CoT gains. Specifically, Instance SC achieves a prediction accuracy of 69.6%, while Aggregated SC reaches 89.2%. At the question level, the accuracy of our method, Dynamic CoT, is nearly identical to the highest performance between CoT and direct answers. Compared to CoT, Dynamic CoT reduces token consumption by 39.1%. In the transfer experiment, our method still maintains high accuracy and reduces token consumption by 35.8%. Our main contributions are as follows: • We introduce the concept of Token Signature to study the CoT gain across different question types based on the decoded token probability distribution. • We propose two token probability distribution indicators, Instance SC and Aggregated SC , to assess whether a benchmark is suitable for CoT at the benchmark level. • We design Dynamic CoT at the instance level to enable the reasonable selection of either CoT or direct answer. 2 Preliminary Token-level Analysis In this section, we present our initial observation on token probability distribution in open-source language models. We begin by conducting a token-level probability analysis using the publicly available Mistral-7B-Instruct model. The model is prompted with question-only inputs (i.e., without CoT trigger or direct answer trigger) and decoded using a greedy strategy. Figure 3 illustrates the probability distribution of the initially generated tokens across the four benchmarks. Figure 2: Illustration of the proposed method for analyzing CoT features through decoding. We calculate spearman correlation from

Read the complete text