PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF
Doo Hwan Hwang, Kee-Eung Kim · 2026
Read PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF by Doo Hwan Hwang, Kee-Eung Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29758.txt.
Introduction
Hwang and Kim propose PS-PPO, a cheaper way to run critic-free reinforcement learning from human feedback for large language models. Critic-free methods are popular for their simplicity, but a trajectory-level signal spread across all tokens makes every update costly.
Prefix-Sampling PPO exploits that temporal redundancy. A prompt-conditioned cutoff distribution picks a cutoff point for each trajectory. The update then backpropagates only through the sampled prefix. An importance-weighting correction keeps the truncated gradient estimator unbiased relative to the full-trajectory objective. Experiments show big reductions in training compute and in peak GPU memory, with accuracy comparable to strong baselines.
Catalog record
| CREATOR | Doo Hwan Hwang, Kee-Eung Kim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29758
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF Doo Hwan Hwang, Kee-Eung Kim Abstract Reinforcement Learning from Human Feedback (RLHF) for Large Language Models increasingly relies on critic-free methods as a practical alternative to actor–critic training. Despite their simplicity, existing critic-free approaches propagate a trajectory-level learning signal uniformly across all tokens in a trajectory. This requires full-trajectory policy updates for every rollout, leading to substantial optimization cost for long reasoning traces, even though intermediate prefixes often contain enough information to largely determine the final outcome. We propose Prefix-Sampling Proximal Policy Optimization (PS-PPO), a compute-efficient critic-free method for RLHF that exploits this temporal redundancy. PS-PPO introduces a prompt-conditioned cutoff distribution and samples a cutoff timestep for each trajectory. During the update pass, PS-PPO backpropagates only through the sampled prefix of each trajectory and applies an importance-weighting correction so that the resulting truncated gradient estimator remains unbiased with respect to the full-trajectory objective. Experiments on mathematical reasoning and RLHF benchmarks show that PS-PPO achieves large reductions in training compute and peak GPU memory, while maintaining accuracy comparable to strong critic-free baselines. 1 Introduction As Large Language Models (LLMs) are increasingly deployed for complex reasoning and decision-making tasks, the quality of a generated response is determined not merely by linguistic fluency, but by its alignment with human preferences and task objectives ( 23 ; 6 ; 2 ; 29 ; 13 ) . Since such objectives are difficult to encode directly in a token-level likelihood objective, Reinforcement Learning from Human Feedback (RLHF) has become a standard method for aligning model behavior with desired outcomes ( 22 ; 29 ) . In practice, RLHF typically assigns a single scalar reward after a full completion is generated. This delayed feedback provides little information about which parts of a long trajectory were responsible for success or failure, making temporal credit assignment a central challenge ( 3 ; 11 ; 19 ) . Figure 1 : Prefix-conditioned success rate on AIME 2024 and MATH-500 versus prefix progress, where prefix progress denotes the percentage of the full completion length. We estimate the success rate using Qwen2.5-Math-7B with 32 suffix rollouts per prefix. The success rate often stabilizes well before the end of the full completion. A classic remedy is to introduce a critic network to estimate token-level value functions and provide the value estimates to support policy-gradient optimization, such as Proximal Policy Optimization (PPO) ( 24 ) . However, training a critic at LLM scale incurs substantial compute and memory overhead and is often a major source of optimization instability ( 25 ; 10 ; 14 ) . These limitations have motivated critic-free alternatives such as GRPO ( 25 ) and RLOO ( 1 ) , which avoid explicit value learning by estimating advantages from multiple trajectories sampled per prompt. However, they commonly broadcast the reward signal uniformly across all timesteps ( 15 ) , requiring full-trajectory updates that lead to substantial compute cost for long reasoning traces. Our motivation for addressing this full-trajectory bottleneck comes from the observation that, in many reasoning tasks, intermediate prefixes already contain substantial information about the final outcome. For example, in step-by-step mathematical reasoning, early decisions such as the choice of decomposition or intermediate derivation often strongly constrain whether the final answer can be correct. Later tokens may still be useful, especially when a model revises or corrects its earlier reasoning. However, our late-recovery analysis in Appendix A suggests that such cases are not the dominant pattern in the reasoning traces we study. We empirically support this observation by analyzing success rates conditioned on intermediate prefixes, as shown in Figure 1 . Across tasks, prefix-conditioned success rates often saturate before the end of the completion, indicating that intermediate prefixes can be highly predictive of the final outcome. Building on this observation, we propose Prefix-Sampling Proximal Policy Optimization (PS-PPO) , a compute-efficient critic-free method. PS-PPO defines a prompt-dependent cutoff distribution of timesteps from a prefix uncertainty signal, retaining only the prefix up to the sampled cutoff during the policy gradient update. To correct the randomness introduced by truncation, we derive a reweighted estimator whose expectation over cutoff sampling recovers the corresponding full critic-free broadcast update. Our main contributions are as follows: • We propose PS-PPO, a compute-efficient critic-free method that samples a prompt-dependent cutoff and computationally truncates forward and backward computation beyond the cutoff. • We derive an inclusion-probability reweighted estimator that corrects for stochastic truncation and recovers the full broadcast update in expectation over cutoff sampling, without additional rollouts or auxiliary models. • We demonstrate that PS-PPO reduces training-time compute and peak GPU memory while maintaining accuracy relative to strong critic-free baselines on mathematical reasoning and RLHF benchmarks. 2 Related Work 2.1 Post-Training for LLM Alignment Post-training of pretrained LLMs is typically conducted via Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL) ( 9 ; 28 ) . While SFT relies on imitating static datasets provided by humans, RL offers a distinct advantage by optimizing for reward maximization. This approach reduces the dependency on extensive ground-truth demonstrations and encourages the model to explore diverse solution paths beyond the training data distribution ( 25 ; 13 ) . This capability has established a new paradigm for reasoning models, enabling them to solve complex problems through broad exploration. 2.2 Critic-Free Policy Optimization in LLMs Standard RLHF pipelines commonly employ actor–critic algorithms to optimize a policy with respect to scalar rewards produced by a reward model. While effective, scaling actor–critic training to large language models often requires substantial memory and compute, and the resulting updates can be sensitive to value estimation accuracy, potentially affecting training stability ( 8 ) . Motivated by these challenges, critic-free alternatives such as RLOO ( 1 ) , GRPO ( 25 ) , Dr.GRPO ( 20 ) , and DAPO ( 32 ) have been proposed. In these methods, the parameterized value network is removed and advantages are computed from multiple samples generated for the same prompt. They broadcast a single trajectory-level learning signal across all tokens, which requires backpropagating through the entire completion. This makes the update increasingly expensive as completions grow longer, motivating token-selective policy optimization. 2.3 Token-Selective RL for LLMs To mitigate the inefficiency of full-sequence updates, recent work has explored token-selective policy optimization by applying learning signals only to a selected subset of tokens. 30 identify a small fraction of high-entropy tokens in chain-of-thought trajectories as decision points and restrict policy updates to those positions. S-GRPO ( 16 ) keeps a heuristically chosen fixed prefix and subsamples the remaining suffix tokens using Bernoulli sampling. PPPO ( 26 ) updates only prefix tokens by progressively increasing the prefix length according to a predefined schedule during training, and evaluates each prefix by sampling multiple continuations conditioned on that prefix and aggregating their rewards. These methods are commonly implemented via gradient masking, which can yield limited computational savings when forward and backward passes are still executed over the full sequence. In contrast, PS-PPO avoids fixed or heuristic token-selection rules by deriving a prompt-dependent cutoff distribution in a principled way. It then samples a cutoff position from this distribution and backpropagates only through the sampled prefix, directly reducing backpropagation cost. 3 Method 3.1 Background Policy optimization ( 22 ) is a primary framework