Improving DAPO from a Mixed-Policy Perspective
Hongze Tan · 2025
Read Improving DAPO from a Mixed-Policy Perspective by Hongze Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12931.txt.
Introduction
Hongze Tan and Yuchen Li tune a sampling policy method with mixed help in a 2025 paper. The base method, DAPO, trains chat models from reward scores. Sparse rewards make such training shaky and slow to converge.
A first fix leans on a stable guide policy for off path samples. The guide steadies the steps and speeds the climb, with step size set on the fly. A second fix reuses zero reward samples that dynamic sampling throws out. Those samples form their own batch under expert guide. Proofs show both aims still reach the best fix in the frame.
Catalog record
| CREATOR | Hongze Tan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 12 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-12931
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving DAPO from a Mixed-Policy Perspective Hongze Tan Abstract This paper introduces two novel modifications to the Dynamic sAmpling Policy Optimization (DAPO) algorithm [1], approached from a mixed-policy perspective. Standard policy gradient methods can suffer from instability and sample inefficiency, particularly in sparse reward settings. To address this, we first propose a method that incorporates a pre-trained, stable guiding policy ( π ϕ \pi_{\phi} ) to provide off-policy experience, thereby regularizing the training of the target policy ( π θ \pi_{\theta} ). This approach improves training stability and convergence speed by adaptively adjusting the learning step size. Secondly, we extend this idea to re-utilize zero-reward samples, which are often discarded by dynamic sampling strategies like DAPO’s. By treating these samples as a distinct batch guided by the expert policy, we further enhance sample efficiency. We provide a theoretical analysis for both methods, demonstrating that their objective functions converge to the optimal solution within the established theoretical framework of reinforcement learning. The proposed mixed-policy framework effectively balances exploration and exploitation, promising more stable and efficient policy optimization. 1 Introduction Policy gradient methods are a cornerstone of modern reinforcement learning. However, on-policy algorithms like vanilla Policy Gradient often exhibit high variance and sample inefficiency. Methods like Proximal Policy Optimization (PPO) [2] and Generational Reinforcement Policy Optimization (GRPO) [3] mitigate this by constraining the update step, ensuring the new policy does not deviate excessively from the old one. Yet, they still require the policies to be very close. The Dynamic sAmpling Policy Optimization (DAPO) [1] algorithm introduces a unique approach but can be inefficient, especially in early training phases when the policy is not yet well-trained. The samples collected can be of low quality. Furthermore, DAPO’s dynamic sampling may discard a significant number of zero-reward samples, which could contain valuable exploratory information. In this work, we propose to enhance DAPO by introducing a mixed-policy training paradigm. Our key idea is to leverage a well-trained guiding policy, \pi_{\phi} , which can be “not very close” to the policy being trained, \pi_{\theta} , but is bounded by the importance sampling weights. This guiding policy provides stable, high-quality experience to accelerate and stabilize training. We present two modifications based on this idea: 1. A mixed-policy method that combines on-policy samples from \pi_{\theta} with off-policy samples from a guiding policy \pi_{\phi} . 2. An extension that re-incorporates the zero-reward samples, previously discarded by DAPO, by treating them as a third set of data guided by \pi_{\phi} . We demonstrate theoretically that these modifications are well-founded, leading to convergent algorithms that find theoretically optimal solutions. 2 Theoretical Foundation Our analysis builds upon the convergence properties of policy gradient algorithms. We start with a foundational theorem [4]. Theorem 1 . Suppose the objective function of the policy gradient algorithm J\in\mathcal{J}_{n} , where \mathcal{J}_{n} is the class of finite-sum Lipschitz smooth functions, has \sigma -bounded gradients, and the importance weight w=\pi_{\theta}/\pi_{\phi} is clipped to be bounded by [\underline{w},\bar{w}] . Let the learning rate \alpha_{k}=\alpha=c/\sqrt{K} , where c=\sqrt{\frac{2(J(\theta^{*})-J(\theta^{0}))}{L\sigma^{2}\bar{w}}}, and \theta^{*} is an optimal solution. Then, the iterates of our algorithm of J(\theta) satisfy \min_{0\leq k\leq K-1}\E\left[\|\nabla J(\theta^{k})\|^{2}\right]\leq\sqrt{\frac{2(J(\theta^{*})-J(\theta^{0}))L\bar{w}}{K\underline{w}}}\sigma. This theorem provides a convergence guarantee for policy gradient algorithms under certain conditions, which forms the basis for our proposed methods. 3 Proposed Modifications We introduce two modifications to the DAPO algorithm from a mixed-policy perspective. 3.1 Method 1: Mixed Policy with Off-policy Guidance In the initial stages of training, the target policy \pi_{\theta} is often suboptimal. To improve sample efficiency, we introduce a well-trained guiding policy \pi_{\phi} to “guide” the training of \pi_{\theta} . Unlike PPO, \pi_{\theta} and \pi_{\phi} do not need to be very close, as long as their ratio is bounded as per Theorem 1. • On-policy Sampling: We first generate N_{\text{on}} sequences of samples using the current policy \pi_{\theta} . Let this set be \{u_{i}\}_{i=1}^{N_{\text{on}}} . • Off-policy Sampling: We then generate N_{\text{off}} sequences using the guiding policy \pi_{\phi} . Let this set be \{v_{j}\}_{j=1}^{N_{\text{off}}} . Following DAPO, the reward for any token within a sequence is defined as the final reward of that sequence. Let R(u_{i}) and R(v_{j}) be the rewards. We define two separate advantage functions based on this principle: \hat{A}_{i,t}=\hat{A}_{i}=\frac{R(u_{i})-\mean(\{R(u_{k})\}_{k=1}^{N_{\text{on}}})}{\std(\{R(u_{k})\}_{k=1}^{N_{\text{on}}})} (1) \hat{B}_{j,t}=\hat{B}_{j}=\frac{R(v_{j})-\mean(\{R(v_{k})\}_{k=1}^{N_{\text{off}}})}{\std(\{R(v_{k})\}_{k=1}^{N_{\text{off}}})} (2) Next, we define the objective functions. The on-policy objective is: J_{\text{on}}(\theta)=\frac{1}{\sum_{i=1}^{N_{\text{on}}}|u_{i}|}\sum_{i=1}^{N_{\text{on}}}\sum_{t=1}^{|u_{i}|}\clip(r^{(1)}_{i,t}(\theta),\hat{A}_{i},\epsilon),\quad\text{where}\quad r^{(1)}_{i,t}(\theta)=\frac{\pi_{\theta}(u_{i,t}|q,u_{i,<t})}{\pi_{\theta_{\text{old}}}(u_{i,t}|q,u_{i,<t})}. (3) The off-policy objective, guided by \pi_{\phi} , is: J_{\text{off}}(\theta)=\frac{1}{\sum_{j=1}^{N_{\text{off}}}|v_{j}|}\sum_{j=1}^{N_{\text{off}}}\sum_{t=1}^{|v_{j}|}f(\hat{r}_{j,t}(\theta,\phi))\cdot\hat{B}_{j}, (4) where f(x)=\frac{x}{x+\gamma} is a scaling function with a small constant 0<\gamma\ll 1 , and \hat{r}_{j,t}(\theta,\phi)=\frac{\pi_{\theta}(v_{j,t}|q,v_{j,<t})}{\pi_{\phi}(v_{j,t}|q,v_{j,<t})} . The final objective function combines both parts. Assuming N_{\text{on}}\approx N_{\text{off}} , we can use equal weights: J(\theta)=\frac{1}{2}J_{\text{on}}(\theta)+\frac{1}{2}J_{\text{off}}(\theta). (5) 3.1.1 Gradient Analysis and Convergence The gradient of the off-policy objective is: \nabla_{\theta}J_{\text{off}}(\theta)=\E_{v\sim\pi_{\phi}}\left[f^{\prime}\left(\frac{\pi_{\theta}(v)}{\pi_{\phi}(v)}\right)\frac{\pi_{\theta}(v)}{\pi_{\phi}(v)}\nabla_{\theta}\log\pi_{\theta}(v)\cdot\hat{B}_{v}\right]. The gradient of the on-policy objective is the standard importance-sampling gradient estimator: \nabla_{\theta}J_{\text{on}}(\theta)=\E_{u\sim\pi_{\theta_{\text{old}}}}\left[\frac{\pi_{\theta}(u)}{\pi_{\theta_{\text{old}}}(u)}\nabla_{\theta}\log\pi_{\theta}(u)\cdot\hat{A}_{u}\right]. Since f^{\prime}(x)=\frac{\gamma}{(x+\gamma)^{2}}>0 , the gradient of J_{\text{off}}(\theta) has a similar form to that of J_{\text{on}}(\theta) , differing only by a positive scaling factor. Based on Theorem 1, J_{\text{off}}(\theta) will converge. If we use the same optimization algorithm for both gradients, they will both approach their respective theoretical optima. This implies: \nabla_{\theta}J(\theta)=0\iff\nabla_{\theta}J_{\text{off}}(\theta)=0\iff\nabla_{\theta}J_{\text{on}}(\theta)=0. This result guarantees that the extremum we find by optimizing J(\theta) is a valid optimum in the reinforcement learning sense. 3.1.2 Benefits of the Modification • Adaptive Step Size: In early training, when \pi_{\theta} is far from producing correct answers, the ratio \pi_{\theta}/\pi_{\phi} is small. This makes f^{\prime}(\pi_{\theta}/\pi_{\phi})\approx 1/\gamma\gg 1 , leading to a larger gradient step for the off-policy component. This accelerates learning by strongly guiding \theta towards the expert policy’s behavior. • Variance Reduction: As training progresses and \pi_{\theta} approaches \pi_{\phi} , the ratio \pi_{\theta}/\pi_{\phi} approaches 1. In this regime, f^{\prime}(\pi_{\theta}/\pi_{\phi})\approx\gamma is small. The variance of the off-policy gradient, \Var(\nabla_{\theta}J_{\text{off}}) , becomes approximately \gamma^{2}\Var(\nabla_{\theta}J^{*}) , where \nabla_{\theta}J^{*} is the traditional off-policy gradient. The scaling function f(\cdot) thus stabilizes training by reducing gradient variance when the policies are close. 3.2 Method 2: Reusing Zero-Reward Samples DAPO’s dynamic sampling discards trajectories with zero reward, which can be wasteful. We propose to utilize these samples by treating them as a separate batch guided by the expert policy \pi_{\phi} . The sampling process is now threefold: • On-policy (non-zero reward): N_{\text{on}} samples \{u_{i}\}_{i=1}^{N_{\text{on}}} from \pi_{\theta} with R(u_{i})>0 . • On-policy (zero reward): N_{\text{zero}} samples \{w_{k}\}_{k=1}^{N_{\text{zero}}} from \pi_{\theta} with R(w_{k})=0 . • Off-policy: N_{\text{off}} samples \{v_{j}\}_{j=1}^{N_{\text{off}}} from the guiding policy \pi_{\phi} . We define three corresponding advantage functions. \hat{A}_{i} is the same as before. \hat{B}_{j} and \hat{C}_{k} are now normalized over the combined pool of off-policy and zero-reward samples to share a common baseline: \hat{B}_{j,t}=\hat{B}_{j}=\frac{R(v_{j})-\mean(\{R(v)\}_{v\in V\cup W})}{\std(\{R(v)\}_{v\in V\cup W})} (6) \hat{C}_{k,t}=\hat{C}_{k}=\frac{R(w_{k})-\mean(\{R(v)\}_{v\in V\cup W})}{\std(\{R(v)\}_{v\in V\cup W})} (7) where V=\{v_{j}\}_{j=1}^{N_{\text{off}}} and W=\{w_{k}\}_{k=1}^{N_{\text{zero}}} . The on-policy objective J_{\text{on}}(\theta) remains the same. We introduce a new mixed objective J_{\text{mix}}(\theta) that combines the zero-reward and off-policy samples: J_{\text{mix}}(\theta)=\underbrace{\sum_{j,t}f(\hat{r}_{j,t}(\theta,\phi))\cdot\hat{B}_{j}}_{\text{off-policy part}}+\underbrace{\sum_{k,t}\clip(r^{(2)}_{k,t}(\theta),\hat{C}_{k},\epsilon)}_{\text{zero-reward part}} (8) where r^{(2)}_{k,t}(\theta)=\frac{\pi_{\theta}(w_{k,t}|q,w_{k,<t})}{\pi_{\phi}(w_{k,t}|q,w_{k,<t})} . Note the importance sampling for the zero-reward samples is also done with respect to \pi_{\phi} . The final objective is a combination of the on-policy and mixed objectives: J(\theta)=\frac{1}{2}J_{\text{on}}(\theta)+\frac{1}{2}J_{\text{mix}}(\theta). (9) The gradient of the mixed term is a weighted sum: \nabla_{\theta}J_{\text{mix}}(\theta)=l_{1}\cdot\nabla_{\theta}J_{1}(\theta)+l_{2}\cdot\nabla_{\theta}J_{2}(\theta), (10) where \nabla_{\theta}J_{1}(\theta) corresponds to the off-policy gradient and \nabla_{\theta}J_{2}(\theta) corresponds to the zero-reward sample gradient, with l_{1}+l_{2}=1 . Similar to the first method, the analysis holds. The gradients for all three components ( J_{\text{on}},J_{1},J_{2} ) are structurally similar up to positive scaling