Flexible Entropy Control in RLVR with Gradient-Preserving Perspective

Kun Chen, Peng Shi, Fanfan Liu, Haibo Qiu, Zhixiong Zeng, Siqi Yang, Wenji Mao · 2026

Read Flexible Entropy Control in RLVR with Gradient-Preserving Perspective by Kun Chen, Peng Shi, Fanfan Liu, Haibo Qiu, Zhixiong Zeng, Siqi Yang, Wenji Mao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09782.txt.

Introduction

Reward learning with checked rewards is a key method for lifting large language model thought. Kun Chen and six co-authors note that steady training often leads to entropy collapse: rapid entropy decay causing early hubris, reduced output range, and fading gradient norms that inhibit learning. Gradient-preserving clipping mainly drives these moves, yet current fixes are largely static and lack a frame linking clipping gears to exact entropy control. Their 2026 paper remakes entropy control from the gradient-preserving clipping view.

The authors first verify in theory and in tests how single importance sampling ratio regions add to entropy growth and fall. A novel control gear using dynamic clipping thresholds then manages entropy exactly. Dynamic entropy control plans are built and tested, including increase-then-decrease, decrease-increase-decrease, and waving decay patterns. Results show these plans ease entropy collapse well and achieve better scores across multiple benchmarks.

Catalog record

CREATORKun Chen, Peng Shi, Fanfan Liu, Haibo Qiu, Zhixiong Zeng, Siqi Yang, Wenji Mao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-09782

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Flexible Entropy Control in RLVR with Gradient-Preserving Perspective Kun Chen, Peng Shi, Fanfan Liu, Haibo Qiu, Zhixiong Zeng, Siqi Yang, Wenji Mao Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a critical method for enhancing the reasoning capabilities of Large Language Models (LLMs). However, continuous training often leads to policy entropy collapse, characterized by a rapid decay in entropy that results in premature overconfidence, reduced output diversity, and vanishing gradient norms that inhibit learning. Gradient-Preserving Clipping is a primary factor influencing these dynamics, but existing mitigation strategies are largely static and lack a framework connecting clipping mechanisms to precise entropy control. This paper proposes reshaping entropy control in RL from the perspective of Gradient-Preserving Clipping. We first theoretically and empirically verify the contributions of specific importance sampling ratio regions to entropy growth and reduction. Leveraging these findings, we introduce a novel regulation mechanism using dynamic clipping threshold to precisely manage entropy. Furthermore, we design and evaluate dynamic entropy control strategies, including increase-then-decrease, decrease-increase-decrease, and oscillatory decay. Experimental results demonstrate that these strategies effectively mitigate entropy collapse, and achieve superior performance across multiple benchmarks. † † affiliationnotice: † Equal correspondence. 1 Introduction Reinforcement Learning with Verifiable Rewards (RLVR) ( 15 ; 9 ) has become an important training paradigm for boosting the reasoning capabilities of Large Language Models (LLMs) across diverse applications. As a representative algorithm of RLVR, the Group Relative Policy Optimization (GRPO) ( 21 ) has been widely adopted and proven effective in enhancing LLM reasoning ( 9 ; 1 ; 30 ; 33 ; 2 ; 34 ) . Nonetheless, a growing body of research also indicates that uncontrolled continuous training in RLVR may cause LLMs towards policy entropy collapse ( 7 ; 22 ; 4 ; 32 ) , manifested as a rapid decay in entropy to near-zero values in model training process, as shown in Figure 1 . Entropy collapse in LLMs poses several challenges to RLVR research. On one hand, a precipitous drop in policy entropy in early stages of RL training causes the model prematurely overconfident in its outputs, at the risk of sacrificing output diversity and thus being trapped in lock optimal solution. On the other hand, 22 has shown that the norm of the model’s training gradients during the RL optimization is constrained by policy entropy, which inhibits the continuous upgrade of the model in later stages of training and ultimately affects model performance. A significant factor influencing policy entropy dynamics during RLVR training is Gradient-Preserving Clipping. Originated from Clipped Proximal Policy Optimization (PPO-Clip) ( 20 ) , this method establishes upper and lower clipping bounds on the importance sampling ratio to optimize the policy within a trust region. Figure 1 : Training dynamics of entropy and gradient norm during GRPO optimization, illustrating entropy collapse and the empirical evidence of the theoretical bound of gradient norm. Related studies on RLVR believe that, although harshly clipping tokens outside this trust region can stabilize the training process to a certain extent, it also impairs the diversity of model outputs due to ignoring some low-probability points, leading to a continuous decline in entropy and even entropy collapse ( 32 ; 17 ; 26 ) . For example, DAPO ( 32 ) introduces the Clip-Higher strategy, which employs a higher upper clipping threshold to raise the probabilities of low-probability ‘exploration’ tokens, thereby increasing policy entropy. However, existing work on policy entropy control is largely confined to understanding the clipping threshold from a static perspective. It lacks a theoretical understanding of the relationship between Gradient-Preserving Clipping and policy entropy control, nor does it tackle entropy control strategy design for effective RL training. To address these limitations, in this paper, we take Gradient-Preserving Perspective to explore the inherent associations of the upper/lower clipping threshold and entropy increase/decrease, and on this basis, aim at flexibly control policy entropy in the training process. Specially, we focus on two core research questions, laying the foundation of mechanism exploration and strategy design in RLVR: Research Questions RQ1 How can we devise the regulation mechanism to precisely control the ebb and flow of entropy? RQ2 Grounded on this mechanism, how can we design the entropy control strategy for effective RL training? The main contributions of our work are three-fold: (1) We theoretically explore the precise representation of different importance sampling ratio regions to entropy increase and decrease in RL training, and provide the empirical support meanwhile; (2) Based on these insights, we devise the regulation mechanism for flexible entropy control, by applying non-linear modulation to these specific ratio regions; and (3) We leverage this regulation mechanism to further explore diverse entropy control strategies, such as an increase-then-decrease (ID) phase, a decrease-increase-decrease (DID) sequence, and an oscillatory decay (OD) within upper and lower clipping thresholds. Extensive experiments demonstrate the effectiveness of our proposed mechanism and strategy design in mitigating entropy collapse, achieving more precise entropy control and markedly enhancing model performance for RLVR. 2 Preliminary 2.1 RL Algorithms of LLMs The PPO-Clip ( 20 ) algorithm addresses the difficulty of determining the appropriate step size in Vanilla Policy Gradient ( 29 ) methods by clipping the policy update ratio, effectively approximating the trust region constraint of Trust Region Policy Optimization (TRPO) ( 18 ) . Specifically, PPO employs an Actor-Critic architecture ( 13 ) . Its core mechanism relies on limiting the divergence between the new and old policies via clipping, thereby preventing drastic policy drift during single updates. The importance sampling ratio r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{old}(a_{t}|s_{t})} is defined as the ratio between the new policy \pi_{\theta} and the old policy \pi_{old} . The objective function of PPO-Clip, denoted as L^{C} , seeks to maximize the advantage function A_{t} while imposing a constraint on r_{t}(\theta) : L^{C}(\theta)=\hat{\mathbb{E}}_{t}\left[\min\left(r_{t}(\theta)\hat{A}_{t},\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\right)\right]\textrm{ .} To more conveniently characterize the clipping threshold in PPO-Clip, we adopt a visualization method similar to that in ( 26 ) , as shown in Figure 2(a) . Different colors represent different sizes of the clipping threshold. Generally, we choose \epsilon_{low}=\epsilon_{high}=0.2 ( 20 ) . In PPO-Clip, \hat{A}_{t} is typically computed using Generalized Advantage Estimation (GAE) ( 19 ) , which relies on an independent value network (Critic) to estimate state values. To enhance training efficiency and stability, GRPO ( 21 ) employs a group-based advantage estimation method that dispenses with the separate Critic. For a given question q , a set of outputs \{o_{1},o_{2},...,o_{G}\} is sampled, yielding corresponding rewards \{r_{1},r_{2},...,r_{G}\} . GRPO calculates the advantage A_{i} for the i -th output as its standardized score relative to the group mean: A_{i}=\frac{r_{i}-\text{mean}(\{r_{1},...,r_{G}\})}{\text{std}(\{r_{1},...,r_{G}\})+\delta}\textrm{ .} (a) (b) (c) Figure 2 : (a) Visualization of PPO clipping threshold regions and probability ratios; (b) Visualization of four entropy-sensitive regions (E1–E4), categorized by the relationship between the old probability ( \pi_{old} ) and the current probability ( \pi_{\theta} ). These regions distinguish between high ( >0.7 ) and low ( \leq 0.3 ) probability states, as well as probability gains and drops; (c) Entropy dynamics curves showing how regions E1/E4 reduce entropy while E2/E3 increase it. 2.2 The Policy Entropy of LLMs The policy entropy of the model characterizes the degree of uncertainty at the current decision point, or equivalently, the ‘flatness’ of the model’s policy. Let V denote the vocabulary space.

Read the complete text