Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias

Rahaf Abu Hara, Vaibbhav Murarri, Claudio Zito · 2026

Read Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias by Rahaf Abu Hara, Vaibbhav Murarri, Claudio Zito (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08315.txt.

Introduction

Rahaf Abu Hara, Vaibbhav Murarri, and Claudio Zito let a language model revise a policy after watching it fail. Standard prompted optimizers see only a single reward number, which hides loops, early falls, and the one catastrophic rollout among nineteen good ones.

Their R2PO framework separates two jobs: a search model proposes candidate policy parameters, and a critic model studies the actual trajectories and suggests targeted fixes grounded in observed states, actions, and rewards. Across ten environments, ablations show the gains need that separation plus selection to filter wild edits. The authors also name a failure mode, salience bias, where the critic overweights the most dramatic rollout.

Catalog record

CREATORRahaf Abu Hara, Vaibbhav Murarri, Claudio Zito
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08315

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias Rahaf Abu Hara, Vaibbhav Murarri, Claudio Zito Abstract Existing LLM-based policy optimizers see only scalar rewards: that a policy scored 0.45, but not whether the agent got stuck in a loop, fell into a hole on the third step, or performed well on 19 out of 20 rollouts and failed catastrophically on one. We propose Reflective Prompted Policy Optimization (R2PO), a two-stage LLM framework for policy search over compact policy classes that augments scalar reward feedback with trajectory-level behavioral evidence. A Search-LLM acts as a global policy optimizer and proposes candidate policy parameters; the environment executes them; a Critic-LLM then inspects the resulting rollouts and proposes targeted parameter revisions grounded in observed states, actions, and rewards. Across ten environments, ablations show R2PO’s gains arise from a design that explicitly separates global search from behavior-grounded revision and uses selection to filter high-variance edits. We further identify a dominant failure mode, salience bias . When presented with multiple rollouts, the Critic-LLM fixates on improving a single failure even when most trajectories succeed. In a three-trajectory variant, where the Critic-LLM is shown the best, worst, and median rollout from each evaluation, this behavior explains 76.6% of regressions on CartPole. R2PO mitigates this by reasoning over aggregate rollout statistics, median-trajectory selection, and a revision rule. Using a relatively small open-weight 20B-parameter model, R2PO achieves the highest mean best reward across all ten environments, while reaching near-optimal performance substantially earlier in training (e.g., near-maximum CartPole reward within ∼ \sim 500 episodes), and training far more stably than both deep RL and prior LLM-based methods. Together, these results show that treating trajectories as first-class in-context evidence, rather than background artifacts reduced to scalar returns, fundamentally changes how even comparatively small LLMs can search over policy spaces, enabling them to learn faster, diagnose more precisely, and reliably improve external controllers rather than tune them by trial and error. 1 Introduction Reinforcement learning has made substantial progress on policy optimization, but standard algorithms require careful hyperparameter tuning and millions of environment interactions to converge, with performance sensitive to implementation details. Recent work has shown that large language models can serve as black-box optimizers for policy search, consuming histories of parameter-reward pairs and proposing new candidate policies through in-context learning [ 22 , 23 ] . These methods treat the LLM as an optimizer over the policy parameter space, requiring no task-specific algorithmic design. However, existing LLM-based policy optimizers operate on scalar reward feedback alone. The optimizer sees that a policy scored 0.45, but not why : whether the agent got stuck in a loop, fell into a hole on the third step, or performed well on 19 out of 20 rollouts but catastrophically failed on one. Scalar feedback provides no diagnostic signal about what went wrong, limiting the optimizer to trial-and-error tuning rather than targeted correction. We introduce Reflective Prompted Policy Optimization (R2PO) , a two-stage LLM framework for policy search over compact policy classes (linear policies for continuous control and tabular policies for discrete tasks). A Search-LLM acts as a global policy optimizer: given the history of evaluated policies and their scalar rewards, it proposes a candidate policy parameter vector. The candidate is executed in the environment, producing rollout trajectories that capture states, actions, and rewards. A Critic-LLM then inspects this evidence and proposes a targeted parameter revision. Finally, a selection step compares the original and revised candidates and keeps whichever achieves the higher reward, limiting damage from over-corrections or LLM hallucinations. Trajectory-grounded diagnosis is a powerful tool that enables better decisions than scalar-only feedback, but it can also cause sharp regressions when the trajectories used for diagnosis are unrepresentative. We identify a recurring failure pattern, salience bias : when the Critic-LLM is shown multiple trajectories from the same policy, it treats the worst-case trajectory as representative, even when the policy is working well. We mitigate salience bias by feeding the Critic-LLM a more representative view of policy behavior: (i) aggregate rollout statistics that summarize overall performance, (ii) a median trajectory that reflects typical rather than worst-case behavior, and (iii) a revision rule that preserves policies that already perform well. Our contributions are: • R2PO , a two-stage LLM policy-optimization framework that combines scalar reward search with trajectory-grounded behavioral diagnosis, treating rollouts as first-class in-context evidence. • A systematic ablation across ten environments over search budget, prompt design, two-stage architecture, selection, and trajectory evidence, showing that R2PO’s gains cannot be explained by more LLM calls or richer prompts alone, but require the combination of trajectory-grounded revision, two-stage design, and keep-best selection. • A mechanistic analysis identifying and quantifying a dominant failure mode of trajectory-grounded LLM revision, salience bias , in which the Critic-LLM overweights vivid worst-case rollouts when multiple trajectories are available, accounting for 76.6% of regressions in a three-trajectory setting on CartPole. • Empirical validation using a 20B open-weight LLM: R2PO achieves the highest mean reward on 9 of 10 environments and ties on the 10th, matches or exceeds all baselines on mean best reward, reaches near-optimal performance substantially earlier in training, and trains more stably than both deep RL and prior LLM-based baselines. 1 1 1 Code is available at: https://github.com/HWUDLabAIRoboticsResearch/R2PO 2 Related Work Recent work integrates LLMs into RL pipelines in two main ways. One line uses LLMs to shape components of the RL loop, such as reward functions, action selection, or auxiliary policy guidance, while leaving optimization to conventional RL algorithms [ 6 , 8 , 4 , 21 , 24 ] . In these approaches, the LLM acts as an auxiliary module rather than as the optimizer itself. A second line places the LLM at the center of optimization. OPRO [ 22 ] uses LLMs to optimize arbitrary objective functions through iterative prompting; ProPS [ 23 ] extends this to RL by consuming parameter-reward histories and proposing policy updates directly. Our work operates in this regime, but differs fundamentally: rather than optimizing from scalar feedback alone, we introduce a second LLM stage that reasons over the proposed policy’s actual rollout behavior, enabling targeted diagnosis of failure modes that scalar rewards cannot reveal. Self-improvement and reflection methods for LLMs are also related. Self-Refine [ 9 ] performs iterative generate-critique-revise loops for single outputs; Reflexion [ 18 ] extends this to agents by converting task feedback into verbal reflections stored in episodic memory. In Reflexion, the LLM is the policy and iteratively retries the same task using verbal reflections from prior attempts. In R2PO, the LLM is an optimizer of separate policy parameters: the Critic-LLM diagnoses an external policy’s rollout behavior and produces a concrete parameter vector evaluated against the original. A recent direction uses trajectories as supervision for LLM training via gradient updates: TreeRL [ 5 ] adds on-policy tree search with process supervision, and TSR [ 1 ] constructs higher-quality training trajectories through lightweight tree-style search. R2PO instead uses rollout trajectories as in-context evidence to revise external policies, with no gradient computation or fine-tuning. 3 Methodology 3.1 Problem setting We consider episodic policy search with parameterized policies \pi_{\theta} and the objective of maximizing expected return J(\theta)=\mathbb{E}[R(\tau)] , where \tau is a trajectory generated by \pi_{\theta} . We focus on compact policy classes whose

Read the complete text