EXPO: Stable Reinforcement Learning with Expressive Policies

Perry Dong, Qiyang Li, Dorsa Sadigh, Chelsea Finn · 2025

Read EXPO: Stable Reinforcement Learning with Expressive Policies by Perry Dong, Qiyang Li, Dorsa Sadigh, Chelsea Finn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07986.txt.

Introduction

Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn stabilize online reinforcement learning for expressive robot policies, in a 2025 paper. Diffusion and flow policies can represent rich behavior, but training them against a value function is unstable: gradients must travel a long denoising chain. Simple Gaussian policies avoid that pain and lose the richness.

Their algorithm, EXPO, never optimizes the big policy against value directly. The expressive base policy trains by imitation, a stable objective. A small Gaussian edit policy then nudges the base policy's sampled actions toward higher value. At each step the better of the base and edited action is used, for both acting and learning targets. Across benchmarks, the method improved sample efficiency two to three fold on average. It worked both for fine-tuning pretrained policies and for learning online with offline data mixed in.

Catalog record

CREATORPerry Dong, Qiyang Li, Dorsa Sadigh, Chelsea Finn
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-07986

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EXPO: Stable Reinforcement Learning with Expressive Policies Perry Dong, Qiyang Li, Dorsa Sadigh, Chelsea Finn Abstract We study the problem of training and fine-tuning expressive policies with online reinforcement learning (RL) given an offline dataset. Training expressive policy classes with online RL present a unique challenge of stable value maximization. Unlike simpler Gaussian policies commonly used in online RL, expressive policies like diffusion and flow-matching policies are parameterized by a long denoising chain, which hinders stable gradient propagation from actions to policy parameters when optimizing against some value function. Our key insight is that we can address stable value maximization by avoiding direct optimization over value with the expressive policy and instead construct an on-the-fly RL policy to maximize Q-value. We propose EX pressive P olicy O ptimization (EXPO), a sample-efficient online RL algorithm that utilizes an on-the-fly policy to maximize value with two parameterized policies – a larger expressive base policy trained with a stable imitation learning objective and a light-weight Gaussian edit policy that edits the actions sampled from the base policy toward a higher value distribution. The on-the-fly policy optimizes the actions from the base policy with the learned edit policy and chooses the value maximizing action from the base and edited actions for both sampling and temporal-difference (TD) backup. Our approach yields up to 2-3x improvement in sample efficiency on average over prior methods both in the setting of fine-tuning a pretrained policy given offline data and in leveraging offline data to train online. † † footnotetext: Correspondence to [email protected] 1 Introduction Robotics has seen significant progress on challenging real-world tasks by training expressive policies on large datasets via imitation learning ( Black et al. 2024 ) . Despite promising results, imitation learning methods often struggle to achieve the high reliability and performance needed for real world use-cases, even when scaled to large datasets. Fine-tuning these policies with reinforcement learning (RL) can in principle address this problem by enabling high performance through online self-improvement. Yet, existing online reinforcement learning methods are typically designed for simple Gaussian policy ( Schulman et al. 2017 ; Fujimoto et al. 2018 ) and do not effectively leverage expressive pre-trained policies, such as diffusion or flow-matching policies ( Chi et al. 2023 ) typically used in imitation learning. Can we design an efficient and effective RL fine-tuning method for expressive policy classes? Fine-tuning expressive policies with online RL comes with a unique challenge not present in fine-tuning simpler Gaussian policies – expressive policies like diffusion or flow-matching policies are parameterized by a long chain of denoising steps, which hinders stable gradient propagation from the action output to the policy parameters whenever we want to optimize their actions against some value functions ( Ding & Jin 2024 ; Park et al. 2025 ) . In the adjacent purely offline or purely online settings, many approaches have sought to avoid the gradient propagation instability by incorporating losses at intermediate denoising steps to guide the denoising process towards high-value actions ( Psenka et al. 2023 ; Fang et al. 2024 ) , but it is still not obvious how to best perform stable value maximization for efficient online fine-tuning. In this work, we make the key observation that value maximization of expressive policy classes can be made much more effective and stable by avoiding direct optimization over value of the expressive policy itself. Instead, we can train the base expressive policy using a stable supervised learning objective and construct an on-the-fly policy to maximize value through two steps — (1) a light-weight, one-step edit policy that refines the action samples from the base expressive policy, and (2) a non-parametric post-processing step that takes multiple action candidates from the base and edit policy and selects the highest-value action among base and edited actions. We impose an edit distance constraint on the edit policy such that the edited actions remain close to the original actions from the base policy. This restricts the edit policy to solve a simpler, local optimization problem, allowing it to be much smaller than the base expressive policy and enabling efficient and stable optimization. The local edits can be viewed as refining actions within modes of the base policy’s action distribution, which is complemented by the second on-the-fly, non-parametric post-processing step, which considers multiple pairs of base and edited actions potentially from different modes and selects the best actions. We instantiate these insights as EXPO , a sample-efficient online RL algorithm that enables stable online fine-tuning of expressive policies. EXPO consists of two parameterized policies: a base expressive policy that is initialized from offline pre-training and then online fine-tuned with an imitation learning objective, and a small Gaussian edit policy that is trained with standard policy loss in reinforcement learning to maximize the Q -value of the edited action. The base policy is never trained to explicitly maximize value. Instead, we construct an on-the-fly policy to maximize Q -value by optimizing the actions from the base policy with the learned edit policy and selecting the best action from the base and edited actions according to their Q -values. The on-the-fly extraction has the advantage that any changes in the Q -function are more immediately reflected in both the agent’s behavior and the TD Q -value target, unlike standard policy extraction methods that require slow parameter updates to align the policy to the Q -function. In addition, the edit policy can be trained with entropy regularization, offering a convenient way to add state-dependent action noises for online exploration beyond the behavior distribution, which is often challenging to do with expressive policies alone. Our main contribution is a simple yet effective method for online RL fine-tuning of expressive policy classes, EXPO. Our method is stable to train and unlike many prior works that focus on a particular class of policies (e.g., diffusion, flow-matching), our method is agnostic to policy parameterization and can fine-tune from any pre-trained policies. We evaluate our method on 12 tasks across 4 domains and find that our approach achieves strong performance in both online RL and offline-to-online RL setting with up to 2-3x improvement in sample efficiency on average. Figure 1: Left: Expressive Policy Optimization (EXPO) is a stable, sample efficient method for training expressive policies with reinforcement learning by avoiding direct optimization over the value function with the expressive policy. Right: Average performance over tasks of EXPO and prior methods. 2 Related Works Sample-efficient reinforcement learning. Online sample efficiency is a long-standing challenge in reinforcement learning ( Chen et al. 2021 ; D’Oro et al. 2023 ) . Recent works have focused on leveraging prior data to better help learning ( Li et al. 2023 ; Ball et al. 2023 ; Hu et al. 2023 ; Dong et al. 2025 ) . However, these methods still mostly rely on simple Gaussian policies, and as such may not be able to derive benefits from the ability to learn multi-modal behavior via expressive policy classes. Furthermore, they are not directly applicable to fine-tuning base policies trained with policies such as diffusion. Various other works have focused on architectural design choices aimed to improve sample efficiency ( Yarats et al. 2021 ; Espeholt et al. 2018 ; Schwarzer et al. 2023 )

Read the complete text