Not All Tokens Are What You Need In Thinking
Hang Yuan, Bin Yu, Haotian Li, Shijun Yang, Christina Dan Wang, Zhou Yu, Xueyin Xu, Weizhen Qi, Kai Chen · 2025
Read Not All Tokens Are What You Need In Thinking by Hang Yuan, Bin Yu, Haotian Li, Shijun Yang, Christina Dan Wang, Zhou Yu, Xueyin Xu, Weizhen Qi, Kai Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17827.txt.
Introduction
Hang Yuan and eight co-authors trim the fat from the long chains of thought that reasoning models produce. Models like o1 and DeepSeek-R1 solve hard problems but overthink, burning time and compute on tokens that barely affect the answer. The challenge is cutting words without cutting correctness.
Their Conditional Token Selection scores each token by its contribution to reaching correct answers, then retrains on the compressed reasoning. Compression levels can vary to suit the budget. On the GPQA benchmark, one model gained accuracy while using fewer reasoning tokens. Pushing compression much further cost only a little accuracy while removing most tokens. The work measures how redundant today's reasoning traces really are.
Catalog record
| CREATOR | Hang Yuan, Bin Yu, Haotian Li, Shijun Yang, Christina Dan Wang, Zhou Yu, Xueyin Xu, Weizhen Qi, Kai Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17827
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Not All Tokens Are What You Need In Thinking Hang Yuan, Bin Yu, Haotian Li, Shijun Yang, Christina Dan Wang, Zhou Yu, Xueyin Xu, Weizhen Qi, Kai Chen Abstract Modern reasoning models, such as OpenAI’s o1 and DeepSeek-R1, exhibit impressive problem-solving capabilities but suffer from critical inefficiencies: high inference latency, excessive computational resource consumption, and a tendency toward overthinking—generating verbose chains of thought (CoT) laden with redundant tokens that contribute minimally to the final answer. To address these issues, we propose Conditional Token Selection (CTS), a token-level compression framework with a flexible and variable compression ratio that identifies and preserves only the most essential tokens in CoT. CTS evaluates each token’s contribution to deriving correct answers using conditional importance scoring, then trains models on compressed CoT. Extensive experiments demonstrate that CTS effectively compresses long CoT while maintaining strong reasoning performance. Notably, on the GPQA benchmark, Qwen2.5-14B-Instruct trained with CTS achieves a 9.1% accuracy improvement with 13.2% fewer reasoning tokens (13% training token reduction). Further reducing training tokens by 42% incurs only a marginal 5% accuracy drop while yielding a 75.8% reduction in reasoning tokens, highlighting the prevalence of redundancy in existing CoT. The code and data are available at github and huggingface . 1 Introduction Large reasoning models such as o1 ( Jaech et al. 2024 ) and R1 ( Guo et al. 2025 ) significantly enhance their reasoning capabilities through reinforcement learning, instructing models to generate thoughtful reasoning steps before producing final answers. Guo et al. 2025 demonstrated that by fine-tuning non-reasoning models like Qwen2.5-14B-Instruct on long Chain of Thought (CoT) data generated by R1, these models can acquire comparable reasoning abilities, even surpassing o1-mini on math and code reasoning tasks. Consequently, numerous distilled R1 reasoning datasets have emerged, including s1K Muennighoff et al. 2025 , SkyThought ( Team 2025 ) , OpenMathReasoning ( Moshkov et al. 2025 ) , and AM-1.4M ( Zhao et al. 2025 ) . Small language models trained on these datasets consistently demonstrate remarkable reasoning capabilities. However, the ever-increasing length of CoT sequences burdens both training and inference, with recent studies ( Sui et al. 2025 ) revealing that models often overthink, expending substantial resources on redundant reasoning steps. This inefficiency raises a critical question: " How can we preserve the accuracy gains of long CoT reasoning while eliminating its computational waste? " Existing solutions, such as TokenSkip’s task-agnostic compression ( Xia et al. 2025 ) , show promise for short CoT sequences but fail to address the unique challenges of reinforcement learning-generated long CoT data (spanning thousands of tokens). Moreover, they overlook contextual signals like questions and answers, which prior work Tian et al. 2025 identifies as key to effective compression. Figure 1: Illustration of Conditional Token Selection. For long CoT datasets, CTS leverages a well-trained reference model to evaluate the importance of each thinking token conditional on the answer, removing less important tokens based on the compression ratio \alpha . The model is then trained on this compressed data, enabling more efficient reasoning capabilities. To bridge this gap, we propose Conditional Token Selection, a framework that dynamically prunes redundant reasoning tokens while preserving those essential for deriving answers. CTS leverages a reference model (RM) trained on high-quality reasoning corpora to score token importance conditioned on critical context (e.g., questions and answers). As shown in Figure 1 , by filtering CoT data at adjustable compression ratios and then fine-tuning model with compressed data, we enable models to learn how to skip unnecessary reasoning tokens during inference. We conducted extensive experiments on models of various sizes, including the Llama-3.1-8B-Instruct ( Grattafiori et al. 2024 ) and the Qwen2.5-Instruct series ( Qwen et al. 2025 ) . The experimental results demonstrate the effectiveness of our method and confirm that there indeed exist many redundant thinking tokens in long CoT data. Specifically, when applied to Qwen2.5-14B-Instruct on the GPQA benchmark ( Rein et al. 2024 ) , CTS demonstrates remarkable efficiency gains: a modest 13% reduction in training tokens results in both a 9.1% accuracy improvement and 13.2% fewer reasoning tokens. When training token reduction is increased to 42% , the method incurs only a minor 5% accuracy decrease while achieving a dramatic 75.8% reduction in reasoning tokens, highlighting an advantageous accuracy-efficiency trade-off. On other benchmarks, such as MATH500 ( Hendrycks et al. 2021 ) and AIME24 ( Mathematical Association of America 2024 ) , as well as with other models, using the compressed training data obtained through CTS resulted in improved accuracy compared to the original data after training. In summary, our key contributions are: • We introduce the Conditional Token Selection framework, which assigns conditional importance scores to tokens within CoT trajectories. This framework selectively preserves essential reasoning tokens, based on critical contextual information, at adjustable compression ratios. • We’ve developed a reference model, trained on high-quality reasoning data, that more accurately assesses the conditional importance of tokens in reasoning CoTs. This model can be applied to other independent tasks, such as prompt compression. • We comprehensively compare token-based compression methods—both conditional and non-conditional—for long CoT data generated by reinforcement learning, thereby validating our token selection strategies. 2 Preliminaries In this section, we introduce some important preliminary concepts. 2.1 Token Compression based on Perplexity For a given context \bm{x}=\{x_{i}\}_{i=1}^{n} , the self-information of a token x_{t} ( 1\leq t\leq n ) can be defined as: I(x_{t})=-\log_{2}P(x_{t}\mid x_{0},x_{1},...,x_{t-1}) (1) Perplexity (PPL) is then defined based on self-information as: \text{PPL}(x_{t})=2^{I(x_{t})} (2) Perplexity is commonly used to measure a language model’s ability to predict the given context. Removing tokens with lower perplexity ( Li et al. 2023 ) has a relatively small impact on the model’s understanding and prediction of the context. Then token compression with a flexible ratio can be achieved by sorting all tokens in the context by perplexity and subsequently removing lower-ranking tokens. 2.2 Conditional and Unconditional Compression To address information redundancy in prompts with long contexts and avoid resource waste during inference, Li et al. 2023 proposed a method that uses a small language model to assess the importance of each lexical unit (such as sentences or tokens) in the original prompts, and then drops the less informative content for prompt compression. Subsequent studies by Jiang et al. 2023 ; Jiang et al. 2024 followed this line of research, proposing more fine-grained compression methods. Pan et al. 2024 utilized a BERT model to transform prompt compression into a classification problem. These compression methods can be categorized into unconditional and conditional approaches, based on whether important information is used as a condition during compression. For instance, in prompts containing instructions, demonstrations, and queries, user queries and instructions can act as crucial conditions for compressing demonstrations. Specifically, during compression, unconditional methods directly compress demonstrations, while conditional methods compress based on essential conditional information \bm{v} using the following formula: I(x_{t}\mid\bm{v},\bm{x}_{<t})=-\log_{2}P(x_{t}\mid\bm{v},\bm{x}_{<t}) (3) In terms of effectiveness, conditional methods generally outperform unconditional methods because they preserve more information ( Pan et al. 2024 ) . 3 Conditional Token Selection Conditional Token Selection builds upon the idea of conditional token compression in prompt. It aims to compress long CoT sequences, such as those generated by harnessing reinforcement learning techniques (e.g., R1), and subsequently fine-tune models on this compressed data.