One-Way Policy Optimization for Self-Evolving LLMs
Shuo Yang, Jinda Lu, Kexin Huang, Chiyu Ma, Shaohang Wei, Yuyang Liu, Guoyin Wang, Jingren Zhou, Li Yuan · 2026
Read One-Way Policy Optimization for Self-Evolving LLMs by Shuo Yang, Jinda Lu, Kexin Huang, Chiyu Ma, Shaohang Wei, Yuyang Liu, Guoyin Wang, Jingren Zhou, Li Yuan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22156.txt.
Introduction
Yang and co-authors present OWPO, one-way policy optimization for evolving language models trained with checked rewards. Sparse binary verifier rewards cause low efficiency and wobble, and existing token constraints relative to a reference policy penalize deviations blindly. OWPO applies one-sided treatment, with gain locking where the policy beats the reference. Iterative reference updates create a ratchet effect locking in gains. Tests show OWPO beats strong baselines including DAPO, OPD, and MOPD, breaking the fixed-prior bottleneck for evolution without external reference models.
Catalog record
| CREATOR | Shuo Yang, Jinda Lu, Kexin Huang, Chiyu Ma, Shaohang Wei, Yuyang Liu, Guoyin Wang, Jingren Zhou, Li Yuan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55353 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22156
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
One-Way Policy Optimization for Self-Evolving LLMs Shuo Yang, Jinda Lu, Kexin Huang, Chiyu Ma, Shaohang Wei, Yuyang Liu, Guoyin Wang, Jingren Zhou, Li Yuan Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has become a promising paradigm for scaling reasoning capabilities of Large Language Models (LLMs). However, the sparsity of binary verifier rewards often leads to low efficiency and optimization instability. To stabilize training, existing methods typically impose token-level constraints relative to a reference policy. We identify that such constraints penalize deviations indiscriminately; this can flip verifier-determined direction when the policy attempts to outperform the reference, thereby suppressing gains. To resolve this, we propose One-Way Policy Optimization (OWPO) , a method based on the principle of decoupling optimization direction from update magnitude. In OWPO, the verifier dictates the update direction, while the reference policy serves only to adjust the magnitude. Specifically, OWPO applies asymmetric reweighting: it performs Accelerated Alignment for inferior deviations (where the policy lags behind the reference) and Gain Locking for superior deviations (where the policy surpasses the reference). Furthermore, by incorporating iterative reference updates, OWPO creates a “Ratchet Effect” that continuously consolidates gains. Experimental results demonstrate that OWPO outperforms strong baselines, including DAPO, OPD, and MOPD, breaking the bottleneck of fixed priors to enable continuous self-evolution without reliance on external reference models. † † affiliationnotice: Equal contribution 1 Introduction As large language models (LLMs) ( 26 ; 20 ; 3 ) are increasingly trained for complex reasoning ( 22 ; 14 ) , Reinforcement Learning with Verifiable Rewards (RLVR) ( 5 ; 9 ; 15 ) has become a promising post-training paradigm. RLVR leverages feedback from external verifiers to provide automatically checkable rewards, and has shown strong potential on verifiable reasoning tasks ( 18 ) . However, verifier rewards are typically sparse and binary, and are often only available at the sequence level. This provides a weak training signal, reducing sample efficiency and frequently resulting in unstable optimization and slow convergence ( 6 ; 23 ) . Figure 1 : Comparison of RLVR paradigms. (A) Pure RLVR ensures the correct optimization direction via verifier signals but suffers from instability due to sparsity. (B) RLVR with Reference Policy (e.g., KL regularization) improves stability but introduces a direction conflict : the reference constraint can forcibly reverse reward-improving updates (Force Reversal) when the policy attempts to deviate from the prior, thereby capping performance. To address this, a common way to stabilize RLVR under sparse, binary verifier rewards is to introduce a reference policy and impose token-level guidance or constraints during optimization ( 1 ; 26 ; 11 ; 7 ) . For example, on-policy distillation aligns the student distribution to a reference model on the student’s own trajectories, while KL-regularized RL constrains policy updates by penalizing deviations from the reference distribution ( 25 ; 24 ) . By providing dense token-level signals and keeping updates close to a known prior, these methods substantially improve training stability and efficiency. However, a key issue is direction : the token-level constraint pushes the policy toward the reference even when the reward-improving direction is away from it. In particular, KL regularization penalizes deviations from the reference regardless of whether they increase reward. For example, when a trajectory has positive advantage ( A>0 ) over the reference, the reward term would encourage moving further in that direction, but the KL term still exerts a strong pull-back toward the reference. If the KL pressure is large enough, it can flip the effective update direction , turning an improving step into a step back toward the reference and limiting gains beyond it (Figure 1 (B)). To resolve this conflict, we propose that the guidance from the reference policy should not interfere with the optimization direction determined by the verifier. Instead, we introduce a simple principle: optimization direction and update magnitude should be decoupled . Specifically, the verifier dictates the direction (the gradient sign), while the reference policy serves only to adjust the magnitude of the update. Building on this, we propose One-Way Policy Optimization (OWPO) . OWPO keeps the reward-improving direction unchanged, but uses the reference policy to scale the step size at the token level. This preserves stability while avoiding unnecessary suppression of superior deviations. Table 1 : Categorization of deviations. We classify policy behaviors based on the verifier’s signal (Advantage A_{t} ) and the deviation from the reference ( \pi_{\theta} vs. \pi_{\text{ref}} ). Superior deviations (Cases I & IV) indicate the policy outperforms the reference, whereas Inferior deviations (Cases II & III) indicate lag. Case Adv. Relation Intuition Deviation I A_{t}>0 \pi_{\theta}>\pi_{\text{ref}} Correct & Confident Superior II A_{t}>0 \pi_{\theta}<\pi_{\text{ref}} Correct but Hesitant Inferior III A_{t}<0 \pi_{\theta}>\pi_{\text{ref}} Wrong & Over-confident Inferior IV A_{t}<0 \pi_{\theta}<\pi_{\text{ref}} Wrong & Cautious Superior OWPO realizes the decoupling of optimization direction and update magnitude through token-level advantage reweighting. Specifically, based on the verifier’s directional signal (the sign of the advantage) and the deviation from the reference model, we categorize the four interaction scenarios into two distinct regimes and apply specific weighting rules for each (as summarized in Table 1 ): (1) Accelerated Alignment for Inferior Deviations (Cases II \& III): We define a deviation as ’Inferior’ when the current policy \pi_{\theta} lags behind the reference \pi_{\text{ref}} along the verifier-determined direction—when \pi_{\theta} is less confident than \pi_{\text{ref}} on correct tokens or more confident on wrong ones. To counter this, OWPO increases the gradient weights ( w>1 ), leveraging the reference prior to accelerate correction. (2) Gain Locking for Superior Deviations (Cases I \& IV): We define a deviation as ’Superior’ when \pi_{\theta} surpasses \pi_{\text{ref}} along the verifier-determined direction—exhibiting higher confidence on correct tokens or stronger suppression on wrong ones. To preserve these advanced gains, OWPO reduces the weights ( w<1 ), shielding them from high-variance interference while maintaining verifier’s direction. However, relying on a fixed \pi_{\text{ref}} eventually creates a bottleneck. Since OWPO performs a one-way trust region update anchored to the reference, the optimization scope becomes constrained once \pi_{\theta} significantly outperforms the initial prior. To transcend this limit, we extend OWPO into an iterative framework ( 30 ; 4 ) . By periodically updating \pi_{\text{ref}} to the optimized \pi_{\theta} , we establish a ’Ratchet Effect’: each iteration consolidates recent gains into a new baseline, enabling sustained performance improvement across stages. We propose OWPO , which decouples update magnitude from direction to prevent flipping the update direction. We show that its asymmetric reweighting preserves beneficial deviations, allowing OWPO to outperform strong baselines (e.g., MOPD) and break the “prior ceiling” for continuous self-evolution without reliance on external teacher models. Conflict of Interest Disclosure. Several authors are employees of Alibaba Group, which developed the Qwen series of models used in this study. This work strictly utilizes the publicly accessible open-source weights of Qwen. 2 Preliminaries In this section, we briefly review the frameworks that form the basis of our study, progressing from RLVR methods to reference-guided strategies. Group Relative Policy Optimization (GRPO) ( 18 ) . GRPO eliminates value networks by estimating advantages via group sampling, typically incorporating a KL penalty towards \pi_{\theta_{\mathrm{old}}} . Given a group of outputs \{y_{i}\}_{i=1}^{G} sampled from \pi_{\theta_{\mathrm{old}}} , the advantage is normalized against group statistics: \hat{A}_{i}=\frac{R_{i}-\text{mean}(\{R_{k}\}_{k=1}^{G})}{\text{std}(\{R_{k}\}_{k=1}^{G})},\,\text{where }R_{i}=\mathbb{I}(\text{Verify}(x,y_{i})). (1) Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) ( 29 ) refines GRPO