Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization

Zhenpeng Su, Leiyu Pan, Xue Bai, Dening Liu, Guanting Dong, Jiaming Huang, Fuzheng Zhang, Kun Gai, Guorui Zhou · 2025

Read Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization by Zhenpeng Su, Leiyu Pan, Xue Bai, Dening Liu, Guanting Dong, Jiaming Huang, Fuzheng Zhang, Kun Gai, Guorui Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07629.txt.

Introduction

Zhenpeng Su and co-authors present Klear-Reasoner, a model trained for long, careful chains of thought. Many strong reasoning models hide their training recipes, which blocks copies. The report opens the full post-training path, from data picking to supervised fine-tuning to reinforcement learning.

Their 2025 paper finds that a few rich data sources beat a wide shallow mix, and that hard items help even without accuracy filters. Two flaws in standard clipping get named: it mutes key exploration signals and wastes poor paths. Their gradient-preserving clipping method softens both faults. Ablations at each stage show what each choice buys, and scores across benchmarks place the model among the leaders.

Catalog record

CREATORZhenpeng Su, Leiyu Pan, Xue Bai, Dening Liu, Guanting Dong, Jiaming Huang, Fuzheng Zhang, Kun Gai, Guorui Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07629

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization Zhenpeng Su, Leiyu Pan, Xue Bai, Dening Liu, Guanting Dong, Jiaming Huang, Fuzheng Zhang, Kun Gai, Guorui Zhou Abstract We present Klear-Reasoner, a model with long reasoning capabilities that demonstrates careful deliberation during problem solving, achieving outstanding performance across multiple benchmarks. Although there are already many excellent works related to inference models in the current community, there are still many problems with reproducing high-performance inference models due to incomplete disclosure of training details. This report provides an in-depth analysis of the reasoning model, covering the entire post-training workflow from data preparation and long Chain-of-Thought supervised fine-tuning (long CoT SFT) to reinforcement learning (RL), along with detailed ablation studies for each experimental component. For SFT data, our experiments show that a small number of high-quality data sources are more effective than a large number of diverse data sources, and that difficult samples can achieve better results without accuracy filtering. In addition, we investigate two key issues with current clipping mechanisms in RL: Clipping suppresses critical exploration signals and ignores suboptimal trajectories. To address these challenges, we propose G radient- P reserving clipping P olicy O ptimization (GPPO) that gently backpropagates gradients from clipped tokens. GPPO not only enhances the model’s exploration capacity but also improves its efficiency in learning from negative samples. Klear-Reasoner exhibits exceptional reasoning abilities in mathematics and programming, scoring 90.5% on AIME 2024, 83.2% on AIME 2025, 66.0% on LiveCodeBench V5 and 58.1% on LiveCodeBench V6 . Figure 1: Benchmark accuracy of Klear-Reasoner-8B on AIME 2024/2025 (avg@64), LiveCodeBench V5 (2024/08/01-2025/02/01, avg@8), and v6 (2025/02/01-2025/05/01, avg@8). 1 Introduction The demonstrated success of OpenAI’s O1 series models ( Jaech et al. 2024 ) and DeepSeek-R1 ( DeepSeek-AI et al. 2025 ) highlights the significant potential of large-scale reinforcement learning for complex reasoning tasks. However, due to the incomplete disclosure of training details, many deep-seated issues remain regarding the reproduction of high-performance reasoning models. Although the community has produced numerous excellent reproductions of small models with 7–8B parameters, such as DeepScaleR ( Luo et al. 2025b , Luo et al. 2025a ) , Light-R1 ( Wen et al. 2025 ) , Ring-7B ( Team et al. 2025 ) , AReal-boba-RL-7B ( Fu et al. 2025 ) and AceReason-Nemotron-1.1-7B ( Liu et al. 2025 ) , their performance in mathematical and coding reasoning tasks still lags behind DeepSeek-R1-0528-Distill-8B ( DeepSeek-AI et al. 2025 ) and Qwen3-8B ( Yang et al. 2025 ) . In this report, we conduct a detailed analytical study of Qwen3-8B-Base ( Yang et al. 2025 ) to advance its reasoning capabilities to state-of-the-art levels for models of comparable scale. We discover that for long Chain-of-Thought supervised fine-tuning (long CoT SFT), a compact set of high-quality data sources proves significantly more effective than larger, more diverse datasets, as high-quality examples ensure consistent learning of accurate reasoning patterns. Then, we find that simple SFT samples that are not filtered for correctness can easily interfere with the model, but difficult samples do not appear to harm the model’s performance even if they are not filtered. In fact, these difficult errors may even promote the model’s exploration and are beneficial to its performance. For reinforcement learning, clipping importance sampling is a commonly used technique. The clipping mechanism primarily controls the magnitude of policy model updates to ensure training stability. Through in-depth analysis, we identify two issues with the current clipping mechanism ( Schulman et al. 2017 ) : • High-entropy token clipping. Among the tokens clipped beyond the upper threshold 1+\epsilon of importance sampling, there exist high-entropy tokens that often correspond to valuable exploratory behaviors at critical decision points. Directly clipping these tokens may lead to premature termination of exploration, adversely affecting the model’s post-convergence performance. Although DAPO ( Yu et al. 2025 ) proposes Clip-Higher to mitigate this issue by adjusting the upper importance sampling threshold to 1+\epsilon_{h} , high-entropy tokens exceeding this threshold still face the same problem. • Delayed convergence of negative samples. When the importance sampling ratio of suboptimal trajectories falls below 1-\epsilon , their gradients are forcibly truncated, preventing the model from updating based on these signals, which in turn slows down convergence. To address the aforementioned two issues, we propose a G radient- P reserving clipping P olicy O ptimization (GPPO) that does not discard the gradients of any tokens. Even for truncated tokens, they are still included in the computational graph of backpropagation and participate in gradient computation. The gradients propagated back from these tokens by GPPO can be proven to be bounded and mild. This mechanism strikes a balance between maintaining training stability and preserving valuable gradient information. Our experimental results demonstrate that, compared to using Clip-Higher, GPPO achieves superior and more stable performance. By integrating long CoT SFT with GPPO RL for math and coding tasks, we derive Klear-Reasoner-8B, a model that outperforms Qwen3-8B 1 1 1 https://huggingface.co/Qwen/Qwen3-8B and DeepSeek-R1-0528-Distill-8B 2 2 2 https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B across key reasoning benchmarks, as shown in Figure 1 . Klear-Reasoner-8B achieves 90.5% on AIME2024, 83.2% on AIME2025, 66.0% on LiveCodeBench V5, and 55.4% on LiveCodeBench V6. Comprehensive ablations validate the efficacy of each component in our training pipeline. 2 Preliminary Before introducing GPPO, we first provide a brief overview of classical RL objectives and key features: 2.1 Proximal Policy Optimization (PPO) PPO ( Schulman et al. 2017 ) is a policy gradient method that stabilizes training via a clipped surrogate objective, constraining policy updates to avoid excessive deviations. Its core objective combines reward maximization with policy constraints through: \displaystyle\mathcal{L}^{\text{PPO}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[\sum_{t=1}^{T}\min\left(r_{t}(\theta)\hat{A}_{t},\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\right)\right] (1) Here, r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\text{old}}(a_{t}|s_{t})} denotes the importance ratio, quantifying the policy change probability for action a_{t} in state s_{t} between the current policy \pi_{\theta} and the previous policy \pi_{\text{old}} . The Generalized Advantage Estimate (GAE) \hat{A}_{t} ( Schulman et al. 2016 ) reduces variance in advantage estimation through discounted temporal differences: \displaystyle\hat{A}_{t}=\sum_{l=0}^{T-t}(\gamma\lambda)^{l}\delta_{t+l},\quad\delta_{t}=r_{t}+\gamma V(s_{t+1})-V(s_{t}) (2) where \gamma is the discount factor controlling future reward weighting, \lambda is the GAE parameter balancing bias-variance trade-offs, V represents the value function, and r_{t} is the immediate reward. The clipping parameter \epsilon , typically 0.2, enforces a hard constraint on policy updates by limiting r_{t}(\theta) to [1-\epsilon,1+\epsilon] . The \min() operator implements a pessimistic bound : when advantages \hat{A}_{t} are positive, it clips overly optimistic updates; for negative advantages, it does not suppress the update magnitude. This mechanism ensures stable policy improvement within a trust region. 2.2 Group Relative Policy Optimization (GRPO) GRPO ( Shao et al. 2024 ) extends PPO by replacing value-based advantage estimation with group-normalized rewards, eliminating the need for a separate value network. For each prompt x and its group of M responses \{y^{(j)}\}_{j=1}^{M} , the objective constrains policy updates through: \displaystyle\mathcal{L}^{\text{GRPO}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[\frac{1}{M}\sum_{j=1}^{M}\frac{1}{|y^{(j)}|}\sum_{t=1}^{T_{j}}\min\left(r_{t}^{(j)}(\theta)\tilde{A}^{(j)},\text{clip}\left(r_{t}^{(j)}(\theta),1-\epsilon,1+\epsilon\right)\tilde{A}^{(j)}\right)\right]. (3) Here, r_{t}^{(j)}(\theta)=\frac{\pi_{\theta}(a_{t}^{(j)}|s_{t}^{(j)})}{\pi_{\text{old}}(a_{t}^{(j)}|s_{t}^{(j)})} remains the importance ratio per token t in response j , while \tilde{A}^{(j)} represents the group-relative advantage computed across all responses: \displaystyle\tilde{A}^{(j)}=\frac{R^{(j)}-\mu_{R}}{\sigma_{R}},\quad\mu_{R}=\frac{1}{M}\sum_{j=1}^{M}R^{(j)},\quad\sigma_{R}=\sqrt{\frac{1}{M}\sum_{j=1}^{M}(R^{(j)}-\mu_{R})^{2}} (4) where R^{(j)} is the cumulative reward for response j , \mu_{R} is the group mean reward, and \sigma_{R} is the reward standard deviation within the response group. This normalization eliminates advantage bias across prompts. The length normalization factor \frac{1}{|y^{(j)}|} , with |y^{(j)}| denoting response length, ensures equitable contribution for variable-length trajectories. 2.3 Raise the Ceiling: Clip-Higher Standard PPO/GRPO employ symmetric clipping bounds 1\pm\epsilon ,

Read the complete text