Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients
Mingwei Xu, Hao Fang · 2026
Read Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients by Mingwei Xu, Hao Fang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06650.txt.
Introduction
Mingwei Xu and Hao Fang propose POPO, a training method that improves language model reasoning by learning only from successful attempts. Standard verifiable-reward training also penalizes failed rollouts, but failures carry no grading of severity, and sampling a few negatives barely dents the vast space of wrong answers under sparse binary rewards. POPO therefore trains exclusively on positive rollouts.
Bounded importance sampling over the positive set lets implicit negative pressure emerge as good answers gain probability. A Siamese network with momentum updates and a bounded similarity penalty stabilizes training. Across math benchmarks POPO beats the GRPO baseline, including a clear margin on the AIME 2025 contest set.
Catalog record
| CREATOR | Mingwei Xu, Hao Fang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06650
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients Mingwei Xu, Hao Fang Abstract Reinforcement learning with verifiable rewards (RLVR), due to the deterministic verification, becomes a dominant paradigm for enhancing the reasoning ability of large language models (LLMs). The community witnesses the rapid change from the Proximal Policy Optimization (PPO) to Group Relative Policy Optimization (GRPO), in which GRPO reduces the complicated advantage estimation with simple estimation over grouped positive and negative rollouts. However, we note that negative rollouts may admit no gradation of failure severity, and the combinatorial vastness makes penalizing a few sampled negatives unlikely to cover a meaningful reward signal under sparse binary rewards. In this work, we propose Positive-Only Policy Optimization (POPO), a novel RLVR framework in which learning can occur exclusively via online positive rollouts. Specifically, POPO utilizes bounded importance sampling over the positive rollout set. Thus, no disjoint negative rollouts are used for the gradient guidance during the policy post-training. We show that implicit negative gradients can emerge naturally through reinforcing the positive probability via rollouts redistribution. Next, POPO stabilizes the policy optimization through two mechanisms. First, it applies a siamese policy network with a momentum-based adaptation law for stabilized policy evolution. Second, we replace the KL-divergence with a bounded similarity penalty term in the siamese representation space. We conduct extensive experiments using publicly available, well-established text-LLM models, e.g., the Qwen family, across all-level mathematical benchmarks (MATH-500, AMC23, AIME 2024/2025, and Olympiad). Our experiment demonstrates that POPO achieves performance comparable to, or even superior to GRPO. Notably, we show that POPO can achieve 36.67 % \% in AIME 2025 with Qwen-Math-7B, outperforming GRPO 30.00 % \% . Our ablation and sweep studies further illustrate the necessity and robustness of POPO components. Our results suggest a new RLVR direction without negative rollouts. 1 Introduction Figure 1: (A) Scheme of RLVR for mathematical question-solving tasks. A general on-policy RL algorithm will reinforce the positive rollouts, while penalizing the negative rollouts. However, the negative rollouts may not contain a meaningful reasoning chain-of-thought in addressing the graduate-level mathematical questions, which motivates the POPO algorithm design. (B) A comparison of POPO and GRPO on the reward trace (upper panel) and mean output completion length (lower panel) on a simple GSM8K dataset. The emergence of reasoning capabilities in large language models (LLMs) through reinforcement learning (RL) makes a significant advance in the capability enhancement and human feedback (RLHF) alignment ( 36 ) . Well-known work, such as DeepSeek-R1 ( 15 ) and OpenAI’s o1 series ( 22 ) , demonstrates that LLMs can surge intelligent chain-of-thought (COT) ( 48 ) reasoning ability through RL with Verifiable Rewards (RLVR) ( 24 ) . In general, RLVR works by giving a sampled question x and the corresponding LLM-generated response y containing reasoning steps and a final answer, a verifier provides deterministic feedback such as R(x,y)=\mathbf{1}[\texttt{verify}(y)=\texttt{True}] ( 7 ) . The simplicity of the verification mechanism, together with various RL algorithms, has achieved substantial improvements in mathematical problem-solving and coding tasks ( 25 ) , where LLMs show a strong ability of self-reflection and generating sophisticated reasoning chains. Despite these successes, current RLVR approaches face a fundamental challenge: the treatment of incorrect responses. Take the mathematical reasoning task, for example. At the cold-start stage, the vast majority of sampled responses are incorrect, e.g., typical pass@k rates are usually below 50% for the AMC 23 dataset, which happens due to diverse situations such as computational errors, problem misinterpretations, and inappropriate solution strategies (see figure 1 ). Thus, we begin to reconsider do incorrect responses provide meaningful learning signal under the sparse reward signal setting? and thus ask the question: can a policy improve by reinforcing only from its successes? This seems counterintuitive at first glance, given the conventions in RL, which reinforce the policy with the good reward while penalizing it for the bad rewards. However, a few insightful observations motivate it. First, like GRPO ( 40 ) , all correct responses share the same advantage over all incorrect ones. Thus, there is no gradation of “how good or how wrong” a response is. Second, the space of incorrect responses is combinatorially vast ( 54 ) for mathematical tasks, making it unlikely that penalizing a few sampled failures meaningfully covers the failure modes. Third, past work in contrastive self-supervised learning (SSL) demonstrates that effective representations are feasible using only positive samples, works such as ”Bootstrap your own latent”(BYOL) ( 14 ) and ”Exploring simple siamese representation learning” (SimSiam) ( 5 ) , achieving powerful latent representations. Inspired by the above, we propose Positive-Only Policy Optimization (POPO) , a new on-policy optimization approach for RLVR that learns exclusively on positive reward responses. Specifically, POPO has the following contributions: • POPO introduces bounded importance sampling weights, which normalize over the positive rollout set, to preferentially reinforce positive responses, while introducing implicit negative penalties. • POPO uses an exponential moving average (EMA) to update the siamese policy network as an adaptive policy anchor, combining the benefits from SSL insights. • POPO replaces KL divergence with representation-space similarity, providing a soft bounded constraint that preserves semantic structure in the predictor space. To evaluate the POPO algorithm, we use the DeepScaleR-Preview-Dataset for policy training. Then, we conduct extensive test experiments using publicly available, well-established LLM models, including the Qwen2.5 Math family, DeepSeek-R1 distilled models, and general text-LLM models, across all-level mathematical reasoning benchmarks (MATH-500, AMC23, AIME 2024, AIME 2025, and Olympiad). Our experiment demonstrates that POPO still achieves comparable or even superior performance compared with GRPO and its variants. Especially, in AIME 2025, we show that POPO can achieve 36.67 \% using Qwen-Math-7B, outperforming GRPO 30.00 \% . Last, our ablation and sweep studies illustrate the necessity and robustness of POPO components and hyperparameters. Our results have implications for future sparse RLVR beyond negative rollouts. 2 Related Work RLHF and RLVR. RLHF addresses a critical limitation that the next-token prediction objective optimized during pretraining and supervised finetuning (SFT) phases does not directly capture the intentions of human users. RLHF aligns LLM behavior to be close to human intent by training a reward model on human preference and optimizing the policy via RL ( 6 ; 43 ) . However, learning reward modeling is an imperfect proxy for human intent and thus may be sensitive to the reward hacking issue ( 12 ) . Offline preference learning, which removes the reward modeling by purely using the contrastive preference, also has widespread usage. For example, Direct Preference Optimization (DPO) ( 37 ) has been applied for aligning the chatbot output to user intent ( 44 ) . On the other hand, RLVR replaces the learnable reward modeling process with deterministic verification pipelines, where correctness can be easily assessed in domains such as mathematical reasoning and code generation ( 7 ; 56 ) . Recent technical reports ( 22 ; 15 ) demonstrate that simple RLVR elicits sophisticated COT. Policy Optimization Methods for RLVR. Following the well-established adoption in GRPO ( 40 ) , it has become a widely used approach by removing the value network and estimating advantages through group-level reward normalization. A series of GRPO extensions have been proposed ( 45 ; 58 ; 57 ; 55 ;