Concise Reasoning via Reinforcement Learning
Mehdi Fatemi † Banafsheh Rafiee † Mingjie (Mike) Tang Kartik Talamadupula Wand AI ∗ † \dagger Equal contribution. Correspondence to: [email protected], [email protected] ⋆ \star This work has been partially done during the authors’ affiliation with Wand AI. · 2025
Read Concise Reasoning via Reinforcement Learning by Mehdi Fatemi † Banafsheh Rafiee † Mingjie (Mike) Tang Kartik Talamadupula Wand AI ∗ † \dagger Equal contribution. Correspondence to: [email protected], [email protected] ⋆ \star This work has been partially done during the authors’ affiliation with Wand AI. (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05185.txt.
Introduction
Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula trace reasoning models' wordiness to a training artifact and cure it with a short second training phase, in a paper from 2025. Long chains of thought cost time and compute. The team argues length grows from how losses treat wrong answers, not from deeper thought.
With unsolvable problems dominating training, incorrect answers push policies toward longer outputs, proven for two standard algorithms. Across models, shorter answers also correlate with correctness. A brief follow-up stage on solvable problems cut response length sharply while holding or improving accuracy. One algorithm showed collapse modes that limit its reliability here.
Catalog record
| CREATOR | Mehdi Fatemi † Banafsheh Rafiee † Mingjie (Mike) Tang Kartik Talamadupula Wand AI ∗ † \dagger Equal contribution. Correspondence to: [email protected], [email protected] ⋆ \star This work has been partially done during the authors’ affiliation with Wand AI. |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-05185
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Concise Reasoning via Reinforcement Learning Mehdi Fatemi † Banafsheh Rafiee † Mingjie (Mike) Tang Kartik Talamadupula Wand AI ∗ † \dagger Equal contribution. Correspondence to: [email protected], [email protected] ⋆ \star This work has been partially done during the authors’ affiliation with Wand AI. Abstract A major drawback of reasoning models is their excessive token usage, inflating computational cost, resource demand, and latency. We show this verbosity stems not from deeper reasoning but from reinforcement learning loss minimization when models produce incorrect answers. With unsolvable problems dominating training, this effect compounds into a systematic tendency toward longer outputs. Through theoretical analysis of PPO and GRPO, we prove that incorrect answers inherently drive policies toward verbosity even when γ = 1 \gamma=1 , reframing response lengthening as an optimization artifact. We further uncover a consistent correlation between conciseness and correctness across reasoning and non-reasoning models. Building on these insights, we propose a two-phase RL procedure where a brief secondary stage, trained on a small set of solvable problems, significantly reduces response length while preserving or improving accuracy. Finally, we show that while GRPO shares properties with PPO, it exhibits collapse modes, limiting its reliability for concise reasoning. Our claims are supported by extensive experiments. Cite this technical report as: @article{concise-reasoning2025, author = {Fatemi, Mehdi and Rafiee, Banafsheh and Tang, Mingjie (Mike) and Talamadupula, Kartik}, title = {Concise Reasoning via Reinforcement Learning}, journal = {arXiv preprint arXiv:2504.05185}, year = {2025}, month = {April}, } Note: This is version 3 of the report (November 2025); however, the April 2025 first version should be cited as provided above. Code: https://github.com/ai-wand/concise-reasoning 1 Introduction Reasoning models have gained significant importance in both research and products, demonstrating remarkable performance in various domains. This success is largely attributed to extensive reinforcement learning (RL) ( Sutton and Barto, 2018 ) post-training applied to a base model, which is initially trained through supervised learning for token completion. During RL training, the model is exposed to a diverse set of reasoning problems, along with their corresponding final answers. Notably, using a complete solution as the training target is not required in RL; instead, the model explores the response space, similar to how an RL agent learns to play a video game. This process fundamentally differs from the supervised training phase for human alignment, commonly referred to as reinforcement learning from human feedback (RLHF), where the objective is to select responses that align with human preferences among multiple model-generated alternatives. A key phenomenon observed during RL post-training is the emergence of an “aha moment” ( Guo et al., 2025 ) . This refers to an inflection point where the model begins exhibiting self-correction behaviors, as seen in responses like “ We must have made a mistake, let’s try again .” Crucially, this behavior is not explicitly programmed but emerges naturally as the model explores the response space. Prior research has found a distinct pattern following this moment: response lengths tend to increase significantly, accompanied by improvements in overall accuracy ( Guo et al., 2025 ; Yeo et al., 2025 ; Zeng et al., 2025 ) . Even with a lack of clear understanding of why this happens, this phenomenon has led many to push for longer responses, leveraging additional training and computational resources in the hope of further enhancing accuracy. We argue that the observed gains in accuracy are distinct from the tendency to generate longer responses. The push for lengthier outputs is not a reasoning strategy but a side effect of minimizing loss. This view also resolves the apparent paradox that, in both reasoning and non-reasoning models, correct responses to reasoning-intensive tasks are often shorter than incorrect ones. Moreover, a critical but often overlooked observation is that RL post-training on moderate-size datasets from domains like mathematics often leads to an initial decrease in response length with no negative impact on the accuracy ( Pan et al., 2025 ; Luo et al., 2025 ) . This suggests that many tokens in the produced chain of thought may not be needed. Additionally, examining the chain of thought in reasoning models, such as DeepSeek’s R1 ( Guo et al., 2025 ) , reveals a significant degree of redundancy, repetition, and irrelevant artifacts. This observation raises a fundamental question: Can reasoning models be further optimized through RL training to produce systematically more concise chains of thought without sacrificing accuracy? To this end, and in light of our theoretical analysis, we propose a two-phase RL training paradigm designed to first enhance the reasoning capabilities of the base model and then enforce conciseness. Both phases can be trained on a small problem set, but the second is key to producing concise responses. Our approach offers a direct alternative to models such as DeepSeek’s R1, yielding significantly shorter responses while maintaining accuracy and in some cases even improving it. It is cost-effective, requires minimal training, and improves computational efficiency, resource use, and response time. Additionally, we present the following key findings. 1. Conciseness and Accuracy Correlation: We show that, during inference of both reasoning and non-reasoning models, concise reasoning strongly correlates with higher accuracy. 2. Analysis of PPO Loss Function Dynamics: We present a mathematical analysis establishing the link between response correctness and PPO’s loss function. Specifically, we show that incorrect answers tend to drive longer responses, while correct ones encourage brevity. 3. Analysis of GRPO Loss Function Dynamics: We show that while GRPO encourages longer responses when facing negative advantage and promotes conciseness when facing positive advantage, it suffers from collapse modes, making it ineffective in enforcing conciseness. 4. Limited Data: We show that RL post-training phases are effective even with remarkably small datasets, a result that defies current trends in the literature and proves viable for resource-constrained scenarios, as confirmed by our experiments. 1.1 Related Work Recent large language models (LLMs) have been fine-tuned with reinforcement learning (RL) to enhance complex reasoning abilities. OpenAI’s o1 model was among the first to use large-scale RL to encourage chain-of-thought (CoT) reasoning, leading to significant gains on challenging math and coding benchmarks. DeepSeek-R1 demonstrated that pure RL post-training (without supervised warm-up) can directly induce strong reasoning capabilities in LLMs. The model exhibited emergent behaviors like self-verification and multi-step planning, and achieved performance competitive with o1 . Similarly, the Kimi k1.5 project scaled RL-based training with extremely long context windows and efficient PPO fine-tuning, enabling the model to backtrack and self-correct ( Team et al., 2025 ) . These models underscore the growing importance of RL for enhancing reasoning capabilities. Beyond large proprietary models, several open research efforts have applied RL to improve reasoning. Han et al. (2023) proposed DialCoT-PPO, which transforms problem solving into a dialogue-based reasoning chain and trains the model to optimize these steps. Reinforced Fine-Tuning (ReFT) has been introduced, combining supervised warm-up with PPO-based exploration of diverse reasoning trajectories, significantly improving accuracy on GSM8K, MathQA, and SVAMP Luong et al. (2024) . Self-Explore was introduced, where the model identifies and learns from its own mistakes in reasoning paths Hwang et al. (2024) . These works demonstrate the versatility of RL in refining reasoning processes across various tasks and model sizes. While it is commonly assumed that longer responses inherently lead to better reasoning accuracy, empirical findings are mixed.