Thinking-Based Non-Thinking: Solving the Reward Hacking Problem in Training Hybrid Reasoning Models via Reinforcement Learning

Siyuan Gan, Jiaheng Liu, Boyan Wang, Tianpei Yang, Runqing Miao, Yuyao Zhang, Fanyu Meng, Junlan Feng, Linjian Meng, Jing Huo, Yang Gao · 2026

Read Thinking-Based Non-Thinking: Solving the Reward Hacking Problem in Training Hybrid Reasoning Models via Reinforcement Learning by Siyuan Gan, Jiaheng Liu, Boyan Wang, Tianpei Yang, Runqing Miao, Yuyao Zhang, Fanyu Meng, Junlan Feng, Linjian Meng, Jing Huo, Yang Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04805.txt.

Introduction

Large reasoning models answer well but think long, and that long chain of thought costs computing time on every query. Siyuan Gan and 10 co-authors address a training failure behind hybrid reasoning models in a 2026 paper. Such models should choose, per query, whether deep thinking is worth it, and reinforcement learning is the usual way to teach that choice.

The catch is reward hacking: a model may think while appearing not to, collecting the wrong reward and corrupting training. The authors' approach, described as thinking-based non-thinking, aims to detect and block that cheat. Fixing the reward signal matters because hybrid models promise reasoning quality without paying full thinking costs on easy questions.

Catalog record

CREATORSiyuan Gan, Jiaheng Liu, Boyan Wang, Tianpei Yang, Runqing Miao, Yuyao Zhang, Fanyu Meng, Junlan Feng, Linjian Meng, Jing Huo, Yang Gao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-04805

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Thinking-Based Non-Thinking: Solving the Reward Hacking Problem in Training Hybrid Reasoning Models via Reinforcement Learning Siyuan Gan, Jiaheng Liu, Boyan Wang, Tianpei Yang, Runqing Miao, Yuyao Zhang, Fanyu Meng, Junlan Feng, Linjian Meng, Jing Huo, Yang Gao Abstract Large reasoning models (LRMs) have attracted much attention due to their exceptional performance. However, their performance mainly stems from thinking, a long Chain of Thought (CoT), which significantly increase computational overhead. To address this overthinking problem, existing work focuses on using reinforcement learning (RL) to train hybrid reasoning models that automatically decide whether to engage in thinking or not based on the complexity of the query. Unfortunately, using RL will suffer the the reward hacking problem, e.g., the model engages in thinking but is judged as not doing so, resulting in incorrect rewards. To mitigate this problem, existing works either employ supervised fine-tuning (SFT), which incurs high computational costs, or enforce uniform token limits on non-thinking responses, which yields limited mitigation of the problem. In this paper, we propose Thinking-Based Non-Thinking (TNT). It does not employ SFT, and sets different maximum token usage for responses not using thinking across various queries by leveraging information from the solution component of the responses using thinking. Experiments on five mathematical benchmarks demonstrate that TNT reduces token usage by around 50 % 50\% compared to DeepSeek-R1-Distill-Qwen-1.5B/7B and DeepScaleR-1.5B, while significantly improving accuracy. In fact, TNT achieves the optimal trade-off between accuracy and efficiency among all tested methods. Additionally, the probability of reward hacking problem in TNT’s responses, which are classified as not using thinking, remains below 10 % 10\% across all tested datasets. 0 0 footnotetext: {\dagger} Corresponding author. 1 Introduction Large reasoning models (LRMs) ( 25 ) , such as DeepSeek-R1 ( 6 ) and OpenAI o1 ( 9 ) , have recently become a central research topic due to their superior capabilities. However, their performance gains rely on extended chain-of-thought (CoT) ( 24 ) —also referred to as thinking . This reliance leads to protracted and repetitive outputs, resulting in a substantial increase in inference overhead and latency ( 11 ; 3 ) , a phenomenon referred to as the overthinking problem ( 22 ; 17 ) . Figure 1: Overview of TNT. Figure 2: Average accuracy and token usage comparison across different hybrid reasoning model training methods on mathematical benchmarks. We only presented the evaluation results of their open-source checkpoints while some of these methods lack the trained checkpoints based on DeepScaleR-1.5B, and DeepSeek-R1-Distill-Qwen-7B. Adapt- x refers to AdaptThink with \delta=x*0.01 . Auto-S x refers to AutoThink-Stage x . We also follow these abbreviation conventions in the table below. To tackle the overthinking problem, recent works increasingly explore hybrid reasoning models that can dynamically determine whether to activate the thinking mode ( 28 ; 5 ; 23 ; 14 ; 10 ; 2 ; 27 ) . The alternative, where the model directly outputs solution without thinking, is referred to as the non-thinking mode. Among these works, one of the most commonly used method involves employing reinforcement learning (RL), with a higher reward allocated to correct answer of the non-thinking mode than that of the thinking mode ( 28 ; 5 ; 23 ) . Unfortunately, the RL training method suffer from reward hacking problem that reduces the performance. Specially, models embed a significant portion of its reasoning process within the non-thinking response ( 28 ; 23 ) . By doing so, the model leverages the thinking process to arrive at the correct answer while securing additional rewards for following the non-thinking format. This leads to increased rewards, but results in a significant increase in tokens for non-thinking mode responses. One approach to mitigate this reward hacking problem is to use supervised fine-tuning (SFT) with a dataset that is significantly larger than the RL dataset, thereby fixing the model’s output, as did in Thinkless ( 5 ) . However, this also incurs a substantial increase in computational overhead. To avoid such high computational overhead, AdaptThink ( 28 ) set a smaller maximum token usage for the non-thinking mode than that for the thinking mode. This way, the response judged in the non-thinking mode cannot increase token usage to actually operate in thinking mode and obtain a higher reward. Unfortunately, 28 set the maximum token usage for the non-thinking mode to be uniform across queries, resulting in only limited mitigation of reward hacking problem. This is because applying such a uniform maximum may result in poor detection of reward hacking problem because thinking mode tokens for simple queries can be fewer than non-thinking mode tokens for complex ones, making uniform limits ineffective (see details in Section A.2 ). To set the maximum token usage across different queries, we propose Thinking-Based Non-Thinking (TNT). As demonstrated in Figure 1 , TNT employs the solution component of the thinking mode responses (i.e., the tokens following </think> are highlighted in blue as shown in Figure 1 ) to set the maximum token usage for the non-thinking mode. Specifically, LRMs’ thinking mode has been trained to ensure the solution component of the thinking mode responses does not involve thinking, such as DeepSeek-R1. Therefore, this component will not differ significantly from the output of the non-thinking mode as the definition of the latter is that outputting solution without thinking. We can use the token usage of this component to set the maximum token usage for the non-thinking mode. Consistent with previous hybrid reasoning model training methods ( 28 ; 23 ; 5 ) , we employ DeepSeek-R1-Distill-Qwen-1.5B/7B and DeepScaleR-1.5B as base models to evaluate TNT on five standard mathematical benchmarks: AIME24, AIME25, Minerva, AMC23, and Olympiad. The results show that TNT consistently reduces token usage by approximately 50\% relative to base models, while substantially improving accuracy. Compared to other hybrid reasoning model training methods, TNT achieves the optimal balance between accuracy and token usage as Figure 2 . Furthermore, the probability of reward hacking problem in TNT remains below 10\% across all tested datasets. 2 Preliminary Now, we introduce the LRMs. Consider a LRM parameterized by the parameter \theta and denoted by \pi_{\theta} . Given a prompt x=[x_{1},\ldots,x_{n},\text{<think>}] , where [x_{1},\ldots,x_{n}] represents the query and the special token <think> means the start of thinking. For this prompt, this LRM generates a response y=[y_{1},\ldots,y_{\tau},\text{</think>},y_{\tau+2},\ldots,y_{m}] . Here, [y_{1},\ldots,y_{\tau}] corresponds to thinking, which is a long CoT consisting of constant exploration, reflection, and self-verification. The token </think> marks the end of thinking. The remaining sequence, [y_{\tau+2},\ldots,y_{m}] , denotes the final solution, which only includes the correct steps to solve the problem and the final answer. From the perspective of probability theory, the response y is a sample drawn from the conditional probability distribution \pi_{\theta}(\cdot|x) . Since y is generated in an auto-regressive way, the conditional probability \pi_{\theta}(y|x) is: \pi_{\theta}(y|x)=\prod_{t=1}^{m}\pi_{\theta}(y_{t}|x,y_{<t}). (1) In this paper, we follow the approach in 28 ; 23 ; 14 1 1 1 In 23 , they use the ellipsis prompt to make x=[x_{1},\ldots,x_{n},\text{<think>},\text{\textbackslash n},\text{...},\text{\textbackslash n}] , which enables sampling the non-thinking mode without using the off-policy sampling. , referring to [y_{1},\ldots,y_{\tau}]\neq\emptyset as the thinking mode and [y_{1},\ldots,y_{\tau}]=\emptyset as the non-thinking mode. We use this approach to distinguish between thinking and non-thinking modes because it does not require modifications to

Read the complete text