ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models

Song Yu, Li Li · 2026

Read ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models by Song Yu, Li Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28204.txt.

Introduction

Yu and three co-authors fix a flaw in how reinforcement learning credits reasoning tokens. Group methods give every token in a chain the same advantage score, which flattens useful signals and collapses exploration. The team spots critical decision pivots, brief high-entropy moments where the path can branch. Their ERPO method boosts exploration at those pivots and reweights token credit by outcome. On hard math benchmarks, ERPO beat the group baseline and wrote shorter, sturdier solutions.

Catalog record

CREATORSong Yu, Li Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37456
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.28204

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models Song Yu, Li Li Abstract Reinforcement learning from verifiable rewards (RLVR) has significantly advanced the reasoning capabilities of large language models. However, standard Group Relative Policy Optimization (GRPO) typically assigns a uniform, sequence-level advantage to all tokens, thereby overlooking the intrinsic information heterogeneity along reasoning chains. We show that this coarse-grained credit assignment leads to premature entropy collapse and encourages the model to generate redundant, low-quality reasoning paths. Through systematic empirical analysis, we identify Critical Decision Pivots (CDPs): transient high-entropy states where the policy’s trajectory is most sensitive to perturbations. These pivots represent the "forks in the road" where effective multi-path exploration is most crucial yet often suppressed by uniform advantage signals. Building on these insights, we propose Entropy-Regulated Policy Optimization (ERPO), which transitions the optimization focus from coarse sequences to fine-grained token dynamics. ERPO introduces three synergistic components: (i) Entropy-aware Gating, which adaptively amplifies exploration at CDPs to facilitate diverse path discovery; (ii) Bucket-based Implicit Normalization, which mitigates difficulty bias by aligning token progress windows; and (iii) Result-anchored Advantage Synthesis, which re-weights token-level signals via outcome-driven anchors. Extensive experiments on competitive mathematical benchmarks (e.g., MATH, AIME) demonstrate that ERPO significantly outperforms GRPO. Notably, ERPO not only boosts reasoning accuracy but also yields significantly more concise and robust derivation paths, establishing a new efficiency-accuracy frontier for large reasoning models. 1 Introduction (a) Token-level entropy distribution. (b) Causal impact on accuracy. Figure 1 : Token-level entropy distribution and its impact on performance based on Qwen2.5-3B. (a) The LLM exhibits high entropy at high-entropy states (red), indicating that the model is making logical branch decisions, and low entropy at the inference process (yellow), indicating that the model is performing deterministic execution steps. (b) We sampled 50 questions that the model could definitely answer correctly and randomly perturbed the top 5% of high-entropy tokens and the bottom 5% of low-entropy tokens for each sequence. Perturbing these high-entropy hubs resulted in a significant decrease in final accuracy (p<0.001) , confirming their crucial role in inference. With the advancement of large language models (LLMs), reinforcement learning has emerged as a central paradigm for post-training base models in complex agent tasks 12 . OpenAI’s o1 11 demonstrates the capability of solving complex logical problems through Chain-of-Thought (CoT), while the release of DeepSeek-R1 4 marks the significant effectiveness of reinforcement learning with verifiable rewards (RLVR) 7 in enhancing model reasoning abilities, powered by Group Relative Policy Optimization 16 (GRPO). GRPO removes the critic network and adopts intra-group relative advantages for gradient updates, substantially reducing computational overhead and improving training efficiency 20 . Despite the remarkable success of GRPO, its underlying assumptions still exhibit limitations. Its assumes that each token shares the sequence-level advantage, implying an inherent premise that each token in the reasoning sequence contributes equally to the final advantage. However, in actual COT reasoning processes, the informational value of tokens demonstrates significant heterogeneity: when models encounter logical branches or critical decision points, token distributions exhibit high entropy characteristics 19 . At such junctures, models require substantial randomness and exploration to identify correct reasoning paths. During deterministic logical expansion phases, token distributions tend to converge. At this stage, redundant derivations often lead to inefficient generation and may even introduce extraneous computational overhead 10 ; 2 . Existing verifiable reward designs, such as length penalties or KL divergence, are typically token-agnostic. This penalization suppresses exploration when models are hesitating, yet fails to effectively compress redundancy when models are confident. To systematically investigate this heterogeneity, we conducted a multi-stage empirical analysis revealing fundamental differences in lexical importance. We first observed significant heterogeneity in the predictive dynamics of LLMs during complex reasoning processes. By visualizing the lexical entropy of inference trajectories (e.g., on the AMC dataset), we identified specific high entropy token 3 at which the model exhibits higher predictive entropy. This stands in stark contrast to deterministic computational steps that maintain near-zero entropy in Figure 1(a) . We define these high-entropy regions as Critical Decision Pivots (CDPs). Unlike fixed logical keywords, CDPs represent the model’s reasoning frontier where the policy is most uncertain and sensitive to perturbations. We then determined the causal significance of these tokens through a large-scale perturbation study. By performing controlled truncation and random replacement on 2460 samples from the MATH dataset, we revealed a significant sensitivity gap: perturbing high-entropy token leads to a sharp 35.2% drop in inference accuracy 9 , while the model remains highly robust to perturbations of low-entropy steps in Figure 1(b) . This suggests that high-entropy states are not merely ’confused’ steps but represent irreplaceable structural transitions where any deviation leads to a cascading failure of the reasoning chain. However, standard reinforcement learning methods such as GRPO 16 ignore this, imparting a uniform advantage throughout the sequence and diluting the crucial reward signal with gradient noise from redundant labels. Our empirical analysis confirms that disrupting these ’hesitation points’ leads to a disproportionate drop in reasoning accuracy, regardless of whether the token is a formal logical operator or a complex structural transition. Based on these insights, we propose Entropy-Regulated Relative Advantage (ERPO), a novel reinforcement learning paradigm designed to refine credit assignment in complex reasoning tasks. The core of ERPO lies in its ability to transform sparse, sequence-level outcomes into dense, token-level signals by integrating Implicit Process Rewards with a dynamic entropy-gating mechanism. ERPO prioritizes exploration at these CDPs to fortify the most fragile links in the reasoning chain. By selectively amplifying the advantage signal at high-entropy tokens while dampening it during deterministic computations, ERPO ensures that the gradient update is concentrated on the most causal steps of the inference chain. In summary, our main contributions are as follows: • We identify CDPs in LLM reasoning via token-level entropy and propose ERPO, which integrates implicit PRM signals and relative position bucketing for fine-grained credit assignment without additional reward training. • We introduce an adaptive gating function that dynamically adjusts reward density based on local predictive uncertainty, balancing exploration at critical junctions with stability in derivation steps. • Experiments on four datasets show ERPO outperforms GRPO, achieving a superior Pareto frontier between reasoning accuracy and sequence length redundancy, while rivaling or even surpassing models tens of times its size in parameter count. 2 Preliminaries We begin by formalizing the probabilistic framework of LLMs 1 , followed by an overview of RLVR. Then, we delineate the GRPO algorithm, providing the necessary background for our proposed method. LLMs. Specifically, given an input prompt x , an LLM \pi_{\theta} sequentially generates a T -token response y=(y_{1},...,y_{T}) : \pi_{\theta}(\mathbf{y}|\mathbf{x})=\prod_{t=1}^{T}\pi_{\theta}(y_{t}|\mathbf{x},\mathbf{y}_{<t}). (1) RLVR. RLVR 7 is a family of reinforcement learning methods that utilize verifiable reward signals rather than learned reward models. Unlike Reinforcement learning from human feedback (RLHF) 12 , RLVR employs rule-based objective rewards, such as the correctness of a programming output, the correctness of the final answer to a mathematical problem, or compliance with formatting. These rewards originate from tasks with explicit ground truth verification. Consider a dataset \mathcal{D}=\{(x,y)\} where x is the prompt and y is the ground truth. The optimization objective in RLVR is to maximize the expected reward: \mathcal{J}(\theta)=\mathbb{E}_{(x,y)\sim\mathcal{D}}\left[\mathbb{E}_{\hat{y}\sim\pi_{\theta}(\cdot|x)}[R(\hat{y},y)]\right], (2) where R(\hat{y},y) is a verifiable reward function that compares the generated output \hat{y} against

Read the complete text