Stable Reinforcement Learning for Efficient Reasoning
Muzhi Dai, Shixuan Liu, Qingyi Si · 2025
Read Stable Reinforcement Learning for Efficient Reasoning by Muzhi Dai, Shixuan Liu, Qingyi Si (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18086.txt.
Introduction
Muzhi Dai, Shixuan Liu, and Qingyi Si work on training language models to reason step by step without rambling. A popular training method rewards only right or wrong final answers, so models learn to think at great length. Adding a penalty for length can fix the bloat, but training then often collapses early, with accuracy falling fast.
Their variant watches how often the model's tries for one question come out right. When few tries pass, it drops the length penalty and guards the quality of thought. When most pass, it keeps the penalty to trim waste. Across five math and science test sets, accuracy rose a bit while the chains of thought shrank by nearly half.
Catalog record
| CREATOR | Muzhi Dai, Shixuan Liu, Qingyi Si |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18086
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stable Reinforcement Learning for Efficient Reasoning Muzhi Dai, Shixuan Liu, Qingyi Si Abstract The success of Deepseek-R1 has drawn the LLM community’s attention to reinforcement learning (RL) methods like GRPO. However, such rule-based 0/1 outcome reward methods lack the capability to regulate the intermediate reasoning processes during chain-of-thought (CoT) generation, leading to severe overthinking phenomena. In response, recent studies have designed reward functions to reinforce models’ behaviors in producing shorter yet correct completions. Nevertheless, we observe that these length-penalty reward functions exacerbate RL training instability: as the completion length decreases, model accuracy abruptly collapses, often occurring early in training. To address this issue, we propose a simple yet effective solution GRPO- λ \lambda , an efficient and stabilized variant of GRPO, which dynamically adjusts the reward strategy by monitoring the correctness ratio among completions within each query-sampled group. A low correctness ratio indicates the need to avoid length penalty that compromises CoT quality, triggering a switch to length-agnostic 0/1 rewards that prioritize reasoning capability. A high ratio maintains length penalties to boost efficiency. Experimental results show that our approach avoids training instability caused by length penalty while maintaining the optimal accuracy-efficiency trade-off. On the GSM8K, GPQA, MATH-500, AMC 2023, and AIME 2024 benchmarks, it improves average accuracy by 1.48% while reducing CoT sequence length by 47.3%. Figure 1: Training process of GRPO+length penalty and our GRPO- \lambda . 1 Introduction Recent advances in large language model (LLM) community have been driven by the development of test-time scaling [ 1 ] , demonstrating a positive correlation between generation length and models’ reasoning capability, which is more effective than model-parameter scaling law [ 2 ] . The open-source releases of DeepSeek-R1 [ 3 ] and Qwen3 [ 4 ] have further stimulated recent research on reinforcement learning (RL) [ 5 , 6 , 7 , 8 , 9 ] for achieving reasoning models [ 10 ] . These models typically generate extended chain-of-thought (CoT) [ 11 ] sequences containing rich and diverse reasoning paths. However, recent studies [ 12 , 13 ] have revealed that reasoning models often suffer from severe overthinking [ 12 , 14 ] issues, characterized by excessive shallow reasoning steps and frequent thought-switching in prolonged CoTs [ 15 , 14 , 16 ] . This occurs because the rule-based outcome rewards in GRPO [ 5 ] cannot effectively regulate intermediate reasoning processes. While longer reasoning chains statistically increase the probability of containing correct reasoning steps (thus improving answer accuracy and rewards during RL training), this GRPO mechanism continuously reinforces the lengthy CoT generation, and results in overthinking problems. To address this issue, representative reasoning models like Kimi-1.5 [ 17 , 18 , 19 ] incorporate length penalty into RL training, constraining the model to generate higher-quality reasoning within shorter sequences, thereby mitigating overthinking while improving inference efficiency. For example, [ 18 ] assigns the highest reward to the shortest correct completion within the group. However, as shown in Figure 1 (left), we reveal that introducing length-aware reward or penalty functions leads to premature RL training collapse: although CoT sequence length decreases as intended, model accuracy abruptly plummets, preventing stable RL training for sufficient iterations. Intuitively, reasoning models require distinct training priorities at different competency stages: when reasoning capability is underdeveloped, reinforcement should prioritize accuracy, whereas efficiency optimization (via length penalty) should only be introduced once the model demonstrates sufficient reasoning capability. Current methods [ 19 , 18 ] overlook this progression, indiscriminately shortening CoT sequences for all samples during RL training, ultimately degrading the model’s inherent reasoning capacity and causing RL training to collapse. Motivated by these insights, we propose a simple yet effective modification to GRPO, namely GRPO- \lambda , that sustainably improves reasoning efficiency without compromising reasoning accuracy, thereby preventing RL training collapse and ensuring sufficient training iterations, as shown in Figure 1(right). Specifically, we sample a set of completions per query following standard GRPO method, then evaluate the group-wise correctness rate, and dynamically switches between optimization modes: applying length penalties once correctness is adequately high (indicating mature reasoning capability to prioritize efficiency) or defaulting to standard GRPO’s 0/1 outcome rewards (to reinforce accuracy fundamentals when below threshold). In this way, our method enables the joint optimization of reasoning efficiency and accuracy while ensuring training stability. Experimental results on GSM8k [ 20 ] , GPQA [ 21 ] , AIME 2024 [ 22 ] , AMC 2023 [ 23 ] , and MATH-500 [ 24 ] benchmarks demonstrate that GRPO- \lambda approach achieves the dual benefit: (1) enhanced training stability (enabling at least 2.5× more viable iterations) and (2) optimal performance-length tradeoffs, with a remarkable 47.3% reduction in sequence length while improving accuracy by 1.48%. 2 Related Work The success of OpenAI-o1 [ 25 , 26 ] reveals that post-training through reinforcement learning serves as a mainstream paradigm for unlocking advanced reasoning capabilities in LLMs. Following the pioneering work of Deepseek-R1 [ 3 ] and Qwen3 [ 4 ] , rule-based outcome reward RL methods [ 3 , 17 , 27 , 28 , 29 , 30 , 31 ] like GRPO [ 5 ] are widely adopted in post-training, encouraging models to produce long CoT outputs, at the cost of inducing overthinking issues [ 12 , 13 , 14 ] . To solve it, recent studies [ 19 , 18 , 17 ] have independently proposed various length penalty mechanisms in reward function design. While these approaches share the common objective of promoting shorter responses and penalizing longer responses among correct ones, they implement distinct strategies. Specifically, Kimi 1.5 [ 12 ] first normalizes the length of sampled responses. For all responses exceeding 0.5 of the normalized length threshold, it assigns negative rewards, whereas those below receive positive rewards. Incorrect responses are restricted to a maximum reward of 0. Similarly, [ 18 ] employs a soft-clip sigmoid function to standardize and smooth length deviations from the group distribution. This maps rewards to the interval (0,1), where shorter and correct responses receive values closer to 1, while incorrect responses are assigned zero reward. S-GRPO [ 19 ] adopts a dual-rollout strategy, performing early-exit interventions at different positions within the first rollout response to construct a serial group, and allocating exponentially decaying rewards based on positional precedence, with zero reward for incorrect ones. Empirical observations reveal that such methods consistently induce premature collapse during RL training. This stems from their unilateral emphasis on length penalization without assessing potential compromises to the model’s reasoning capability. In essence, when the model demonstrates strong pass@1 performance, length optimization should take priority for enhanced efficiency. Conversely, when sampled responses within the group fail to achieve satisfactory accuracy (weak pass@1), the focus should shift to reinforcing reasoning abilities rather than pursuing reasoning efficiently. Our method GRPO- \lambda addresses this limitation by adaptively balancing these objectives, enabling more stable and prolonged RL training that ultimately achieves superior performance-efficiency trade-offs. 3 Methods Figure 2: Framework of GRPO- \lambda . We introduce GRPO- \lambda , a stabilized and efficient variant of GRPO designed to address training instability caused by length-penalty reward. GRPO- \lambda uses batch-wise dynamic adjustment of reward strategies, which selectively applies efficiency-prioritized or accuracy-prioritized optimization for different subsets of groups within a batch.