Distribution-Centric Policy Optimization Dominates Exploration-Exploitation Trade-off

Zhaochun Li, Chen Wang, Jionghao Bai, Shisheng Cui, Ge Lan, Zhou Zhao, Yue Wang · 2026

Read Distribution-Centric Policy Optimization Dominates Exploration-Exploitation Trade-off by Zhaochun Li, Chen Wang, Jionghao Bai, Shisheng Cui, Ge Lan, Zhou Zhao, Yue Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12730.txt.

Introduction

Zhaochun Li and six co authors rethink the exploration and exploitation trade off in reinforcement learning for language models. Training with group relative policy optimization tends to be exploitation driven. Entropy falls steadily, samples converge, and exploration fades away. A model that stops exploring can lock into mediocre habits.

Most existing fixes are sample centric, hunting or rewarding rare samples. They assume exploration comes from novel trajectories and tokens. The team argue for a distribution centric policy optimization instead. Their title claims this view dominates the old trade off framing. Shifting focus from single samples to whole distributions may keep exploration alive. The work targets a core weakness in current language model training.

Catalog record

CREATORZhaochun Li, Chen Wang, Jionghao Bai, Shisheng Cui, Ge Lan, Zhou Zhao, Yue Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-12730

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Distribution-Centric Policy Optimization Dominates Exploration-Exploitation Trade-off Zhaochun Li, Chen Wang, Jionghao Bai, Shisheng Cui, Ge Lan, Zhou Zhao, Yue Wang Abstract The exploration–exploitation (EE) trade-off is a central challenge in reinforcement learning (RL) for large language models (LLMs). With Group Relative Policy Optimization (GRPO), training tends to be exploitation driven: entropy decreases monotonically, samples convergence, and exploration fades. Most existing fixes are sample-centric : they seek or bonus rare samples, assuming exploration comes from novel trajectories and tokens. These heuristics depend on the “luck” of informative samples, lack principled control of the policy, and often yield limited or inconsistent gains. In this work, we are the first to introduce a distribution-centric perspective for RL, in which exploration is always guided by a “better” target distribution, and reveal that a policy’s ability to resist entropy collapse is governed by the distribution itself rather than individual samples. Building on this insight, we propose Distribution-Centric Policy Optimization (DCPO), which reformulates entropy regulation as distribution-level regularization. DCPO achieves controllable entropy fully on-policy without sampling from external distributions, enabling efficient exploration while maintaining training stability. Across multiple models and seven benchmarks, DCPO improves over GRPO by about 20% on average. Overall, DCPO replaces sample-level heuristics with distribution-level principles, offering a theoretically grounded and flexible framework for controllable exploration and a stronger EE trade-off. The code is available in https://github.com/597358816/DCPO . † † affiliationnotice: Equal contribution \svgsetup inkscapearea=page 1 Introduction Reinforcement Learning (RL) has become a cornerstone for enhancing the reasoning capabilities of large language models (LLMs) ( 6 ; 37 ; 27 ; 23 ; 51 ; 42 ) . In complex domains such as mathematics and code generation, recent progress has been driven by Reinforcement Learning with Verifiable Rewards (RLVR), where models learn from outcome-level, automatically checkable rewards ( 14 ; 44 ) . In this context, exploration aims to expose the model to as diverse a set of samples as possible during training, so as to discover better solutions across a broader search space, rather than repeatedly circling within a narrow region of behaviors ( 31 ; 1 ; 30 ; 13 ) . Entropy , as a quantitative measure of model’s uncertainty, has become a practical proxy for a policy’s exploration capacity ( 26 ; 9 ; 19 ) . Among RLVR algorithms, Group-Relative Policy Optimization (GRPO) is particularly attractive due to its efficiency, optimizing directly from final trajectory rewards ( 28 ; 17 ; 7 ) . However, GRPO is widely regarded as exploitation-driven: its training dynamics often reduce entropy monotonically, causing sample convergence and distributional sharpening that progressively suppress exploration ( 48 ; 5 ) . When a policy undergoes entropy collapse, its exploration space becomes severely constrained, reducing subsequent learning to mere distributional sharpening within a narrow region of the solution space, which is consistent with recent findings that RL for LLMs often fails to expand the boundary of reasoning capability ( 49 ) . To mitigate this issue, prior variants introduce explicit entropy encouragement, such as Entropy-Reg ( 22 ; 10 ) and Entropy-Adv ( 3 ) . More recently, AEPO proposes to inject exploration by applying a REINFORCE regularization on high-temperature samples, which alleviates entropy collapse without introducing large optimization bias ( 39 ) . Yet these approaches remain sample-centric : they focus on obtaining or bonusing rarer samples, assuming that exploration arises from individual trajectories and tokens with high novelty. As a consequence, sample-centric optimization relies on the “luck” of drawing sufficiently informative instances, lacks principled control over the policy distribution, and offers limited insight into the overall optimization dynamics, resulting in limited or inconsistent gains. This motivates a distribution-centric perspective, which views exploration as an intrinsic property of the entire policy distribution and leverages a “better” target distribution to guide exploration, rather than relying on fortuitous rare samples. To understand the underlying mechanism, we conduct a series of importance-sampling-based analyses using AEPO as a representative baseline and evaluate policies by their ability to resist entropy collapse. We reveal and prove that the key to avoiding entropy collapse is not the novelty of individual samples, but the expectation gradient induced by the distribution. Building on this insight, we propose Distribution-Centric Policy Optimization (DCPO), which reformulates entropy regulation as a distribution-level regularization problem. The key design components of DCPO are as follows: • Fully Online and On-Policy for Exploration. DCPO performs optimization entirely under the current policy, eliminating off-distribution sampling and improving both training efficiency and stability. • REINFORCE as Regularization. DCPO employs the REINFORCE policy gradient as a regularization term, ensuring entropy regulation toward the target distribution. • Double Importance Sampling. DCPO introduces two layers of importance weighting—one for correcting the deviation between the sampling distribution and the current policy with clipping for stable training, and another for adjusting the gradient expectation toward the target distribution—thereby enabling precise distribution-level regularization. Experiments show that DCPO outperforms GRPO by an average of about 20% across seven widely used reasoning benchmarks, and surpasses the strongest existing entropy-control baseline AEPO. DCPO provides compelling evidence that the essence of exploration lies in the distribution itself, rather than the diversity of sampled instances. In scenarios that demand exploration, seeking a better distribution offers a more efficient path than blindly searching through the vast textual space. Our main contributions are summarized as follows: • Distribution-centric optimization. We reinterpret entropy regulation as a property of the policy’s sampling distribution within the policy gradient expectation, providing a principled explanation for how distributional diversity governs entropy dynamics. • DCPO for entropy regulation. We propose DCPO, which achieves entropy control using only samples from the original distribution. It offers higher optimization performance and efficiency. • A principled framework for EE trade-off. Beyond a single algorithm, we establish a unified, distribution-grounded perspective that formalizes the Exploration–Exploitation balance as a controllable optimization process. Within this framework, we identify the Precision–Prediction (PP) Trade-off , which characterizes how precise sampling ensures stability, while predictive importance sampling enhances optimization effectiveness. This principle provides the foundation for a broader family of EE trade-offs in RL. 2 Preliminary Our work focuses on fine-tuning LLM using RL for tasks with verifiable solutions, such as mathematical reasoning and code generation. Verifiable rewards remove the traditional reward model used in RL and instead assign binary 0/1 rewards. To be specific,for a given query q and its corresponding reference response o^{*} , the reward for any response o sampled from policy \pi_{\theta} is defined as R(q,o)=\textbf{1}[o=o^{*}]. In this paper, an auto-regressive language model parameterized by \theta is defined as a policy \pi_{\theta} . Suppose the LLM is a softmax policy, that is \displaystyle\pi_{\theta}(o_{t}|q_{t})=\frac{\text{exp}(l(q_{t},o_{t}))}{\sum_{o^{\prime}_{t}}\text{exp}(l(q_{t},o^{\prime}_{t}))}, where q_{t} is the concatenation of query q followed by o_{<t} , and l(q_{t},o_{t}) is the logit of token o_{t} given input q_{t} . Furthermore, given a temperature T, we define: \displaystyle\pi_{\theta}^{T}(o_{t}|q_{t})=\frac{\text{exp}(l(q_{t},o_{t})/T)}{\sum_{o^{\prime}_{t}}\text{exp}(l(q_{t},o^{\prime}_{t})T)}. 2.1 Policy optimization REINFORCE is a cornerstone of policy gradient methods that directly optimizes the policy \pi_{\theta} by maximizing the expected reward over sampled trajectories. The objective is formally defined as: \displaystyle\mathcal{J}_{\text{REINFORCE}}(\theta)=\mathbb{E}_{q\sim P(Q),\;o_{i}\sim\pi_{\theta_{\text{old}}}(O|q)}\frac{1}{|o_{i}|} \displaystyle\sum_{t=1}^{|o_{i}|}\min\Big[r_{i,t}(\theta)\,R(q,o),\;\text{clip}\!\big(r_{i,t}(\theta),1-\epsilon,1+\epsilon\big)\,R(q,o)\Big]. A key characteristic of REINFORCE is that the reward R is applied uniformly as a credit to all actions (tokens) within the trajectory. To reduce the high variance inherent in this estimator, a baseline b is typically subtracted from

Read the complete text