Positive-Only Drifting Policy Optimization

Qi Zhang · 2026

Read Positive-Only Drifting Policy Optimization by Qi Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16519.txt.

Introduction

Zhang proposes PODPO, a generative method for online reinforcement learning that learns only from positive examples. Standard Gaussian and flow policies suffer from narrow expressiveness, complex gradient clipping and strict trust-region rules, and they correct bad actions only by penalizing negative samples after the fact. PODPO instead updates policy through advantage-weighted local contrastive drifting, steering actions toward high-return regions while using the generative model's local smoothness to prevent errors proactively. The approach needs neither likelihoods nor gradient clipping, opening a new path for generative policy learning.

Catalog record

CREATORQi Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26927
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16519

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Positive-Only Drifting Policy Optimization Qi Zhang Abstract In the field of online reinforcement learning (RL), traditional Gaussian policies and flow-based methods are often constrained by their unimodal expressiveness, complex gradient clipping, or stringent trust-region requirements. Moreover, they all rely on post-hoc penalization of negative samples to correct erroneous actions. This paper introduces Positive-Only Drifting Policy Optimization (PODPO), a likelihood-free and gradient-clipping-free generative approach for online RL. By leveraging the drifting model, PODPO performs policy updates via advantage-weighted local contrastive drifting. Relying solely on positive-advantage samples, it elegantly steers actions toward high-return regions while exploiting the inherent local smoothness of the generative model to enable proactive error prevention. In doing so, PODPO opens a promising new pathway for generative policy learning in online settings. I Introduction This paper introduces Positive-Only Drifting Policy Optimization (PODPO). Drawing on the core idea of Drifting Models [ 1 ] , PODPO reframes policy optimization as advantage-weighted drifting vector learning. Unlike existing methods, PODPO performs explicit drifting updates exclusively on positive-advantage samples. It leverages the inherent local smoothness of the generative model to achieve indirect error correction and proactive error prevention on near-boundary samples. At the same time, it suppresses extreme gradients at their source through a multi-temperature non-normalized accumulation mechanism, completely eliminating the need for explicit gradient clipping or trust-region constraints. The approach requires no policy likelihood computation and employs a lightweight contrastive drifting loss, making it fully compatible with the standard PPO framework while remaining straightforward to implement and deploy. The main contributions of this paper are as follows: 1. We propose the PODPO algorithm, which seamlessly integrates Drifting Models with online reinforcement learning to enable likelihood-free, single-step generative policy optimization. 2. We design a per-observation local contrastive drifting mechanism that eliminates cross-sample interference, achieving higher drifting precision while preserving computational efficiency. 3. We show that positive-advantage samples alone are sufficient to achieve proactive error prevention in policy optimization, and we introduce a novel paradigm for generative reinforcement learning: “optimize only the recoverable errors while ignoring irrecoverable states. 4. Through advantage weighting and gradient analysis, we demonstrate the method’s simplicity and training stability, showing that it can be directly embedded into existing PPO training pipelines. 5. We validate the effectiveness of PODPO on robotic continuous control tasks, providing a concise and highly efficient new solution for generative online reinforcement learning. II Background II.A PPO PPO [ 2 ] typically optimizes a unimodal Gaussian policy, which struggles to capture the multimodal characteristics inherent in complex action spaces. Moreover, it relies on post-hoc penalization of negative samples for error correction, thereby limiting the policy’s exploration capability and expressive power in challenging scenarios. II.B FPO/FPO++ FPO/FPO++ [ 3 , 4 ] introduce generative models into the policy learning framework, effectively breaking through the bottleneck of unimodal Gaussian policies. However, existing flow-based strategies still fully inherit PPO’s explicit gradient clipping and complex trust-region designs, and they continue to depend on bidirectional updates from both positive and negative samples. II.C Diffusion Reinforcement Learning and Offline Applications Diffusion models possess powerful distribution modeling capabilities and have been widely adopted in reinforcement learning. Current research on diffusion-based RL focuses primarily on offline settings. While diffusion models excel at learning complex multimodal policies from offline datasets, they require multi-step iterative generation during inference, incurring substantial computational overhead. This makes them ill-suited for the real-time demands of online reinforcement learning [ 5 , 6 , 7 ] . II.D Drifting Models In 2026, the team of Kaiming He introduced Drifting Models, a novel single-step generative paradigm that fundamentally rethinks the “iterative inference” paradigm of traditional diffusion and flow-matching models by shifting the entire iterative process into the training phase. The core idea of Drifting Models is to introduce a drifting field that governs the movement direction of generated samples during training. When the generated distribution deviates from the target data distribution, the drifting field guides the samples toward the target; once the two distributions match perfectly, the drifting field becomes zero and equilibrium is reached. The training objective is remarkably simple: compute the drifting vector for each generated sample and train the network to output exactly that drifted target [ 1 ] . Compared with traditional generative models, Drifting Models offer three compelling advantages: • True single-step generation: At inference time, only a single forward pass is required to produce high-quality samples, eliminating any iterative steps and delivering exceptional computational efficiency. • Extremely simple implementation: Training requires nothing more than a plain MSE loss, with no need for complex time-step embeddings, ODE solvers, or adversarial training. • Inherent contrastive learning property: The drifting field naturally performs distribution matching through an “attract positive samples, repel negative samples” contrastive mechanism, effectively capturing both local and global structures in the data. Drifting Models have achieved breakthrough performance in image generation tasks. However, their application in reinforcement learning remains largely unexplored. This work builds upon this latest advance by integrating Drifting Models with online reinforcement learning, resulting in the proposed Positive-Only Drifting Policy Optimization (PODPO) algorithm. III Positive-Only Drifting Policy Optimization III.A Gradient-Based Policy Optimization In online policy gradient optimization, PPO is typically employed to jointly optimize both the value function and the policy. The surrogate objective of standard PPO is given by: \max_{\theta}\mathbb{E}_{a_{t}\sim\pi_{\theta_{\text{old}}}}\left[\min\left(r(\theta)\hat{A}_{t},\ \text{clip}\left(r(\theta),1-\epsilon_{\text{clip}},1+\epsilon_{\text{clip}}\right)\hat{A}_{t}\right)\right] where r(\theta) denotes the likelihood ratio between the new and old policies [ 2 ] . Recent flow-based policy gradient methods (such as FPO and FPO++) further extend this framework. FPO++ optimizes and clips gradients via per-sample ratio and the ASPO trust region [ 3 , 4 ] : \hat{\rho}^{(i)}_{\text{FPO++}}(\theta)=\exp\left(l_{\text{old}}^{(i)}-l_{\theta}^{(i)}\right) \psi_{\text{ASPO}}\bigl(\hat{\rho},\hat{A}_{t}\bigr)=\begin{cases}\psi_{\text{PPO}}\bigl(\hat{\rho},\hat{A}_{t}\bigr)&\hat{A}_{t}\geq 0,\\[4.0pt] \psi_{\text{SPO}}\bigl(\hat{\rho},\hat{A}_{t}\bigr)&\hat{A}_{t}<0\end{cases} Both approaches achieve stable optimization by explicitly clipping gradients. In contrast, PODPO follows an entirely different path: it directly employs an advantage-weighted MSE drifting loss that requires no likelihood computation, yet still achieves implicit gradient control through advantage weighting, thereby maintaining high compatibility with the standard PPO training framework. III.B Drifting Loss The theoretical foundation of PODPO is directly inspired by the recently proposed Drifting Models (generative drifting models) [ 1 ] . This work introduces a novel generative paradigm: during training, a drifting field guides the pushforward distribution to gradually evolve toward the target data distribution, thereby achieving true single-step generation. Its core drifting loss is given by: L_{j}=\text{MSE}\left(\tilde{\phi}_{j}(x)-\text{sg}\left(\tilde{\phi}_{j}(x)+\tilde{V}_{j}\right)\right) where \tilde{V}_{j} denotes the drifting field and sg is the stop-gradient operation. This loss enables the network to directly learn a single-step ”drift-toward-the-target” mapping during training, completely avoiding the multi-step iterative process required by diffusion models. PODPO naturally extends this elegant drifting-model paradigm to online reinforcement learning policy optimization. III.C PODPO Drifting Loss The core innovation of PODPO lies in performing per-observation local contrastive drift independently for each observation within a batch. For every positive-advantage sample (\hat{A}>0) , we generate G dedicated candidate actions exclusively for that observation. These candidates are independently sampled from a standard Gaussian \mathcal{N}(0,1) at each step, forming a completely isolated 1-positive + G -negative contrastive set. This design eliminates cross-observation noise interference and thereby guarantees maximum drifting efficiency for each individual state. Formally, for each positive-advantage sample b=1,\dots,B_{\rm pos} in the mini-batch: x^{(b)}\in\mathbb{R}^{G\times D},\ y_{\text{pos}}^{(b)}\in\mathbb{R}^{1\times D},\ y_{\text{neg}}^{(b)}=x^{(b)} where x^{(b)} denotes the G candidate actions independently generated for the b -th observation, y_{\text{pos}}^{(b)} is the true rollout action of that observation, and the drifting vector

Read the complete text