AdapThink: Adaptive Thinking Preferences for Reasoning Language Model
Xu Wan, Wei Wang, Wenyue Xu, Wotao Yin, Jie Song, Mingyang Sun · 2025
Read AdapThink: Adaptive Thinking Preferences for Reasoning Language Model by Xu Wan, Wei Wang, Wenyue Xu, Wotao Yin, Jie Song, Mingyang Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18237.txt.
Introduction
Xu Wan and five co-authors present a training method that helps reasoning language models think the right amount, in a 2025 paper. Models trained to reflect can waste effort on easy questions. They can also stop too soon on hard ones. Fixed length budgets and set rules adapt poorly to varied problems.
Their method, AdapThink, uses two tools to adjust on the fly. A reward rule weighs the model's confidence and the shape of its answer. That rule steers words linked to reflection without a fixed length target. A sampling tool balances correct answers with varied reasoning paths. Tests on math data sets used distilled open models. The method kept performance while cutting wasted reasoning, the authors report. It also eased the known inefficiencies of slow thinking.
Catalog record
| CREATOR | Xu Wan, Wei Wang, Wenyue Xu, Wotao Yin, Jie Song, Mingyang Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-18237
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AdapThink: Adaptive Thinking Preferences for Reasoning Language Model Xu Wan, Wei Wang, Wenyue Xu, Wotao Yin, Jie Song, Mingyang Sun Abstract Reinforcement Learning (RL)-based post-training has significantly advanced the complex reasoning capabilities of language models, fostering sophisticated self-reflection processes. However, this “slow thinking” paradigm presents a critical challenge to reasoning efficiency: models may expend excessive computation on simple questions and shift reasoning prematurely for complex ones. Previous mechanisms typically rely on static length budgets or predefined rules, lacking the adaptability for varying question complexities and models’ evolving capabilities. To this end, we propose AdapThink , an adap tive post-training framework designed to induce more efficient think ing while maintaining the performance of reasoning language models. Specifically, AdapThink incorporates two key mechanisms: 1) A group-relative reward function that leverages model confidence and response’s characteristic to dynamically adjust the preference of reflection-related transition words without resorting to a fixed length preference. 2) A diversity-aware sampling mechanism that balances the training group’s solution accuracy with reasoning diversity via an entropy-guided score. Experiments on several mathematical reasoning datasets with DeepSeek-distilled models demonstrate AdapThink’s advantages in enabling adaptive reasoning patterns and mitigating the inefficiencies. 1 Introduction Recent breakthroughs in large language models such as OpenAI’s o1 openai2024learning and DeepSeek R1 guo2025deepseek have demonstrated that reinforcement learning (RL)-based post-training methods can substantially improve their reasoning capabilities. This enhancement is primarily attributed to the emergence of models’ sophisticated self-reflection behaviors kumar2025llm ; kazemnejad2024vineppo . However, recent research has highlighted a significant inefficiency associated with this “ slow thinking ” pattern muennighoff2025s1simpletesttimescaling ; han2025tokenbudgetawarellmreasoning ; chen2025think23overthinkingo1like ; wang2025thoughtsplaceunderthinkingo1like ; aggarwal2025l1controllinglongreasoning ; shen2025dastdifficultyadaptiveslowthinkinglarge . Reasoning models frequently overthink simple problems, thereby spending unnecessary computational resources, while conversely underthinking complex challenges, leading to incomplete reasoning and incorrect answers. When presented with a simple problem (see Appendix A ), a model can reach a correct answer with merely 479 tokens. Yet, its self-validation mechanisms—marked by phrases such as “Verify” and “Wait”—triggered unnecessary reflections, resulting in more than a quadrupled token consumption. In contrast, when attempting a complex problem, the same model exhibited frequent and unproductive shifts, marked by phrases “Alternatively” and “Another.” Upon reaching its token limit, the model stopped and arrived at an incorrect answer. Therefore, an ideal chain of thought (CoT) would be capable of adjusting its self-reflection frequency and depth, adaptive to the problem difficulty and its level of confidence. Empirically, ma2025rethinking ; xie2025logic have demonstrated that employing reflection vocabularies does not always guarantee mathematical reasoning improvement. Drawing upon this insight, recent works explored direct control via modifying the input prompt jin2024impactreasoningsteplength ; muennighoff2025s1simpletesttimescaling ; liu2025adaptivestepautomaticallydividingreasoning ; han2025tokenbudgetawarellmreasoning ; chen2025think23overthinkingo1like or providing indirect reasoning-length rewards wang2025thoughtsplaceunderthinkingo1like ; aggarwal2025l1controllinglongreasoning ; shen2025dastdifficultyadaptiveslowthinkinglarge . A common limitation of these approaches is using token budgets governed by rules or offering rewards for adhering to a special length budget, overlooking the critical impact of the changes in models’ capabilities and length preferences. Intuitively, models with limited reasoning abilities benefit from more extended CoT patterns, as redundant self-reflection could serendipitously contribute to reaching correct solutions. In contrast, high-performing models should aim to minimize token consumption to prevent overthinking and to maintain efficient CoT reasoning. Motivated by these insights and our observations in Section 3 , we propose AdapThink , an RL post-training framework for reasoning models. Our post-training empowers models to adjust their preference for reasoning depth, which is adaptive to their current operating capabilities. Instead of directly limiting the budget of reasoning length, our work adjusts the length preference through analyzing the distribution of diverse reasoning patterns observed in groups of generated samples . We find that reasoning length alone does not directly determine model reasoning performance. Instead, strategically regulating reasoning depth is a more effective way to mitigate overthinking and underthinking. Specifically, our main contributions are: 1. First, we introduce a group-relative reward function , a novel mechanism designed to adjust the model’s current reasoning preferences. With it, the model is trained to determine the appropriate preferences for reflection based on intra-group accuracy of generated responses, while measuring reasoning efficiency quantitatively through counting key transition words that appear in these groups of training samples. 2. Furthermore, we propose a diversity-aware sampling mechanism to balance between achieving high reasoning accuracy and fostering rich diversity within the groups of training samples. To achieve this, we begin with an oversampling of reasoning instances. We then employ carefully defined diversity metrics to evaluate both the final answers and the intermediate steps of these instances, before applying diversity-aware downsampling to curate and enhance the overall quality of the instances used for RL post-training. 3. By post-training the DeepSeek-distilled Qwen model subject to a context length limit of only 2K tokens , our method achieves superior performance compared to several length-control baselines when tested within an 8K-token limit, demonstrating AdapThink’s effectiveness in developing powerful and efficient CoT models across multiple reasoning benchmarks. 2 Related Work Induce Longer CoT For inducing longer reasoning length, several works weng2023large ; miao2023selfcheck ; saunders2022self ; renze2024self ; jin2024impactreasoningsteplength have encouraged models to engage in deeper thinking through natural language feedback. For zero-shot CoT, weng2023large ; miao2023selfcheck stimulate model self-reflection by performing backward verification or multiple response voting. For few-shot CoT with demonstrations, jin2024impactreasoningsteplength introduced five general standardized patterns to induce models to simulate human thinking and reshape the CoT. However, these one-size-fits-all approaches ignore the diversity of problem-solving paths, only have a monotonous processing mode. Similarly, muennighoff2025s1simpletesttimescaling designed several budget forcing mechanisms to increase guidance in CoT, such as appending “Wait” multiple times when the model tries to end, forcing it to double-check. Likewise, shen2025satorireinforcementlearningchainofactionthought introduced special meta-action markers like <|continue|> , <|reflect|> , and <|explore|> , enabling the model to restart from intermediate steps, lengthen responses, and correct errors. Induce Shorter CoT Simple prompt methods are also effective for encouraging a shorter CoT nayab2024concise ; xu2025chain ; renze2024benefits . For example, muennighoff2025s1simpletesttimescaling added “Final answer” to terminate the model’s thinking process. Besides, jin2024impactreasoningsteplength ; kang2025c3ot used stronger models to compress long CoTs semantically into shorter ones. yeo2025demystifyinglongchainofthoughtreasoning ; shen2025dastdifficultyadaptiveslowthinkinglarge ; aggarwal2025l1controllinglongreasoning ; yu2025dapoopensourcellmreinforcement design different length-budget signals from the perspective of rule-based reward design, encouraging models to balance accuracy and token efficiency during the thinking process. Although shen2025dastdifficultyadaptiveslowthinkinglarge already takes problem complexity and model confidence into account for the budget, its adoption of a uniformly shorter response preference is less suitable for models with weaker reasoning abilities. Moreover, while yu2025dapoopensourcellmreinforcement ; zhang2025srpo uses dynamic sampling to avoid the zero-advantage issue in group-relative advantage calculation, its sampling strategy focuses solely on outcome diversity metrics and neglects intra-group reasoning process diversity. Moreover, some studies have further focused on addressing overthinking and underthinking in long CoTs. wang2025thoughtsplaceunderthinkingo1like ; sui2025stop introduced thought switching penalties to influence the token decoding probability distribution early in CoT generation, reducing initial thought-switching. Similarly, chen2025think23overthinkingo1like presented an efficiency metric to evaluate each token’s contribution to accuracy and used length preference optimization to achieve more efficient CoT patterns. However, both approaches depend on auxiliary judgments from more powerful reasoning models, making the performance improvements inherently constrained by the capabilities of the reference models. 3 Observations To investigate potential overthinking and underthinking issues in current language models, we first conducted a comprehensive analysis of the generation patterns from DeepSeek-R1-Distill-Qwen-1.5B and DeepSeek-R1-Distill-Qwen-7B models