Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, [3pt] Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, Shengyi Huang, [3pt] Siran Yang, Jiamang Wang, Wenbo Su, Bo Zheng · 2025
Read Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning by Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, [3pt] Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, Shengyi Huang, [3pt] Siran Yang, Jiamang Wang, Wenbo Su, Bo Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08221.txt.
Introduction
Zihe Liu and co-authors sort the real tricks from the traps in reinforcement learning for language model reasoning in a paper from 2025. The field exploded with techniques, each claiming gains. Conflicting reports followed, since studies used different data, settings, and starting models. Practitioners cannot tell which knobs truly matter.
This review re-runs the popular techniques under one roof. Isolated evaluations strip away the confounds. Some celebrated tricks shrink; others prove robust. The authors distill what survives into guidance the field lacked. Part one of a planned series, the paper aims to standardize how reasoning training is judged. Fewer folklore rules would speed everyone's experiments.
Catalog record
| CREATOR | Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, [3pt] Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, Shengyi Huang, [3pt] Siran Yang, Jiamang Wang, Wenbo Su, Bo Zheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08221
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, [3pt] Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, Shengyi Huang, [3pt] Siran Yang, Jiamang Wang, Wenbo Su, Bo Zheng Abstract \definecolor darkmagentargb0.56, 0.0, 1.0 \definecolor softyellowrgb1.0, 0.92, 0.3 \definecolor LightAquamarinergb0.75, 1.0, 0.8 \definecolor FireBrickRGB178,34,34 \definecolor MediumPurpleRGB147,112,219 \definecolor uclabluergb0.15, 0.45, 0.68 † † footnotetext: * Equal Contribution. † Corresponding to: Weixun Wang <[email protected]>. Abstract Reinforcement learning for LLM reasoning has rapidly emerged as a prominent research area, marked by a significant surge in related studies on both algorithmic innovations and practical applications. Despite this progress, several critical challenges remain, including the absence of standardized guidelines for employing RL techniques and a fragmented understanding of their underlying mechanisms. Additionally, inconsistent experimental settings, variations in training data, and differences in model initialization have led to conflicting conclusions, obscuring the key characteristics of these techniques and creating confusion among practitioners when selecting appropriate techniques. This paper systematically reviews widely adopted RL techniques through rigorous reproductions and isolated evaluations within a unified open-source framework. We analyze the internal mechanisms, applicable scenarios, and core principles of each technique through fine-grained experiments, including datasets of varying difficulty, model sizes, and architectures. Based on these insights, we present clear guidelines for selecting RL techniques tailored to specific setups, and provide a reliable roadmap for practitioners navigating the RL for the LLM domain. Finally, we reveal that a minimalist combination of two techniques can unlock the learning capability of critic-free policies using vanilla PPO loss. The results demonstrate that our simple combination consistently improves performance, surpassing strategies like GRPO and DAPO. Figure 1 : Left : The proliferation of RL optimization techniques, coupled with diverse initialized models and data, has raised barriers to practical adoption. Right : We establish detailed application guidelines via dissecting internal mechanisms of widely-used tricks, and introduce Lite PPO, a minimalist two-technique combination that enhances learning capacity in critic-free policies with vanilla PPO loss. The average accuracy is calculated across six mathematical benchmarks. 1 Introduction Recent breakthroughs in large language models (LLMs) such as OpenAI o1 ( Wu et al. 2024 ) and DeepSeek R1 ( Shao et al. 2024 ) have positioned reinforcement learning (RL) as a key driver in unlocking advanced reasoning capabilities within LLMs. This is particularly evident in challenging reasoning tasks like mathematical reasoning ( He et al. 2025a ) and code generation ( Zhuo et al. 2025 ) , where RL has demonstrated the potential to elevate LLM performance beyond what pre-training alone can achieve. Such an emerging trend has sparked widespread interest within the research community in the direction of "RL for LLM" (or RL4LLM). In 2025, RL4LLM experienced a surge in research activity, leading to hundreds of publications across arXiv and major conferences, covering a wide range of topics from algorithmic innovation to practical engineering solutions. However, this rapid progress is shadowed by the lack of usage guidelines for existing RL techniques or tricks ( Huang et al. 2024a ) as well as the absence of in-depth analysis of their underlying mechanisms. Specifically, these limitations can manifest as confusion among practitioners in choosing RL tricks, as different papers provide different solutions to the same problem. For instance, GRPO ( Shao et al. 2024 ) advocates for group-level normalization to enhance policy stability, whereas REINFORCE++ ( Hu et al. 2025 ) argues that batch-level normalization works better. Moreover, GRPO incorporates variance in normalization, yet Dr. GRPO ( Liu et al. 2025a ) explicitly recommends removing variance normalization to prevent bias. Another example: GRPO ( Shao et al. 2024 ) has achieved a breakthrough in performance through the strategy of using response-level loss calculation, while DAPO ( Yu et al. 2025 ) has instead adopted token-level loss calculation. Such contradictory and chaotic phenomena underscore the fragmented understanding and inconsistent recommendations within the RL4LLM community. One possible reason for the above phenomenon is that the experimental settings, training data, and initialization of the existing work all have significant differences, which may also cause deviations in the summary of the conclusions. Apart from the confusion caused by the intrinsic differences of similar techniques, the numerous and seemingly orthogonal techniques, including Normalization, Clip, and Overlong Filtering , also increase the complexity of algorithm application in practice. Practitioners face non-trivial challenges in selecting an appropriate combination from a wide range of techniques to unlock the learning capacity of LLMs in specific scenarios. These ambiguities have naturally triggered a key requirement of practitioners: What scenarios are the existing techniques respectively suitable for? Is there a simple and generalized combination that can be used to enhance policy optimization? Aligned with classic RL mechanism analysis methodologies ( Andrychowicz et al. 2020 ; Engstrom et al. 2020 ; Huang et al. 2024a ) , we systematically review the widely used RL techniques by reproducing them and independently evaluating the actual impact of each technique, based on the same open-source infrastructure framework and policy models. To comprehensively cover practical scenarios, we design extensive experimental settings incorporating datasets of varying difficulty levels, diverse model sizes, and distinct model types. Furthermore, we conduct an in-depth analysis of their theoretical foundations, implementation details, and applicable scenarios as demons. The intuitive contribution is illustrated in Figure 1 . Specifically, ❶ our empirical results reveal that most RL techniques exhibit obvious preferences and sensitivities to the experimental setup, e.g., model type, data distribution, reward mechanism and hyperparameter. ❷ Based on the isolated analysis under our setup, we demonstrate that employing only two techniques, i.e., advantage normalization (group-level mean, batch-level std) and token-level loss aggregation , can unlock the learning capability of critic-free policies using vanilla PPO loss, surpassing mainstream RL4LLM algorithms incorporating redundant components. Our core contributions are selected as: 1. Group-level normalization shows robust efficiency under each reward setting. Batch-level normalization provides more stable improvement under large scale reward setting. (§ 4.1.1 ) 2. Group-level mean and batch-level standard deviation enable further robust normalization. (§ 4.1.3 ) 3. Clip Higher prefers promoting high-quality exploration for aligned models. (§ 4.2.1 ) 4. There appears to be a “scaling law” between the performance and the upper bound of the clipping on the small-sized model. (§ 4.2.3 ) 5. Compared to sequence-level loss aggregation, token-level aggregation is effective on base models, while showing limited improvement on aligned models. (§ 4.3.1 ) 6. Overlong filtering enhances accuracy and clarity for short-to-medium reasoning tasks but provides limited benefits for long-tail reasoning. (§ 4.4.1 ) 7. Two techniques may unlock learning capacity in critic-free policies based on vanilla PPO loss. (§ 5 ) 2 Preliminaries 2.1 Proximal Policy Optimization (PPO) Proximal Policy Optimization (PPO) ( Schulman et al. 2017 ) is a widely used actor-critic algorithm grounded in the policy gradient framework. It improves the stability of policy learning by optimizing a clipped surrogate objective that restricts the divergence between the new and old policies during training. The PPO objective is: \displaystyle\mathcal{J}_{\mathrm{PPO}}(\theta)= \displaystyle\mathbb{E}_{\left[q\sim P(Q),\ o\sim\pi_{\theta_{\mathrm{old}}}(O|q)\right]} (1) \displaystyle\frac{1}{|o|}\sum_{t=1}^{|o|}\min\Bigg(\frac{\pi_{\theta}(o_{t}|q,o_{<t})}{\pi_{\theta_{\mathrm{old}}}(o_{t}|q,o_{<t})}A_{t},\,\mathrm{clip}\left(\frac{\pi_{\theta}(o_{t}|q,o_{<t})}{\pi_{\theta_{\mathrm{old}}}(o_{t}|q,o_{<t})},\,1{-}\epsilon,\,1{+}\epsilon\right)A_{t}\Bigg), where \pi_{\theta} and \pi_{\theta_{\text{old}}} denote the current and old policy models, respectively. q and o represent the sampled question and output sequence, with o_{t} as the t -th token in o . \epsilon is a clipping