Opinion: Towards Unified Expressive Policy Optimization for Robust Robot Learning

Haidong Huang, Haiyue Zhu, Jiayu Song, Xixin Zhao, Yaohua Zhou, Jiayi Zhang, Yuze Zhai, Xiaocong Li · 2025

Read Opinion: Towards Unified Expressive Policy Optimization for Robust Robot Learning by Haidong Huang, Haiyue Zhu, Jiayu Song, Xixin Zhao, Yaohua Zhou, Jiayi Zhang, Yuze Zhai, Xiaocong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10087.txt.

Introduction

Haidong Huang and six co-authors write an opinion piece proposing UEPO, a unified approach for training robot policies that start offline and then adapt online. That offline-to-online path promises safer deployment, but behavior coverage is thin and distributions shift during adaptation. The authors borrow the pretrain-then-fine-tune rhythm of large language models and rebuild it around generative diffusion policies.

Three components carry the design: a multi-seed dynamics-aware diffusion policy that captures varied behaviors in one model, a divergence regularizer that keeps that diversity physically meaningful. A diffusion-based augmentation module that hardens the dynamics model. On the D4RL benchmark the system improved over a leading baseline on locomotion and dexterous manipulation tasks, which the authors offer as evidence for the unified approach.

Catalog record

CREATORHaidong Huang, Haiyue Zhu, Jiayu Song, Xixin Zhao, Yaohua Zhou, Jiayi Zhang, Yuze Zhai, Xiaocong Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-10087

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Opinion: Towards Unified Expressive Policy Optimization for Robust Robot Learning Haidong Huang, Haiyue Zhu, Jiayu Song, Xixin Zhao, Yaohua Zhou, Jiayi Zhang, Yuze Zhai, Xiaocong Li Abstract Offline-to-online reinforcement learning (O2O-RL) has emerged as a promising paradigm for safe and efficient robotic policy deployment but suffers from two fundamental challenges: limited coverage of multimodal behaviors and distributional shifts during online adaptation. We propose UEPO, a unified generative framework inspired by large language model pretraining and fine-tuning strategies. Our contributions are threefold: (1) a multi-seed dynamics-aware diffusion policy that efficiently captures diverse modalities without training multiple models; (2) a dynamic divergence regularization mechanism that enforces physically meaningful policy diversity; and (3) a diffusion-based data augmentation module that enhances dynamics model generalization. On the D4RL benchmark, UEPO achieves +5.9% absolute improvement over Uni-O4 on locomotion tasks and +12.4% on dexterous manipulation, demonstrating strong generalization and scalability. 1 INTRODUCTION Offline-to-Online Reinforcement Learning (O2O-RL) has emerged as a key paradigm for safe and efficient robot deployment. It leverages static offline datasets to pretrain base policies so as to more accurately capture physical dynamics, mitigating real-world trial-and-error risks, and further fine-tunes policies via environmental interaction to adapt to dynamic scenarios, forming a complete “offline initialization to online fine-tuning” framework. In practice, the perception stack and camera configuration materially affect data quality in the wild: camera motion can confound evaluation, and large camera arrays require stable calibration and efficient refinement ( 10 ; 17 ; 9 ) . In adjacent agentic systems, LLM-based multi-agent orchestration and self-evolving teamwork demonstrate scalable planning capabilities ( 2 ) , while systematic safety audits reveal risk accumulation in multi-agent decision loops and motivate conservative online interaction ( 3 ) ; furthermore, learnable tool capability memory helps agents choose and parameterize tools more reliably ( 14 ) . However, existing O2O-RL methods still face significant challenges, including inefficient offline initialization and a weak interface between generative models and online adaptation ( 18 ; 7 ) . Traditional Behavior Cloning (BC) ( 1 ) relies heavily on large amounts of expert data and struggles to cover multi-modal action distributions. While mainstream generative models such as Diffusion Policy ( 4 ) excel at offline modeling, their fixed noise schedules and the lack of environmental feedback often lead to policy degradation and distribution shift during online fine-tuning ( 11 ) . Recent frameworks such as Off2On ( 5 ) , BPPO ( 19 ) , and Uni-O4 ( 8 ) have made notable progress in integrating offline and online learning by jointly optimizing objectives without additional regularization. Nevertheless, Uni-O4 ( 8 ) still exhibits limitations in offline pre-training, generative model adaptation, and scalability, leading to high computational costs for the integrated strategy, insufficient diversity at the physical execution level, and poor data efficiency and generalization. Moreover, current pipelines rarely leverage advances in upstream data selection and preference aggregation that are commonplace in multimodal recommendation, such as multi-level self-supervised alignment for retrieval and group-consensus modeling ( 16 ; 15 ) . These shortcomings render existing algorithms less suitable for high-dimensional dynamics and scenarios with scarce real-world data. To address these issues, we propose a unified generative O2O-RL framework. Our approach includes a data-efficient generative offline module that: (1) employs a dynamics-aware diffusion policy combining U-Net ( 12 ) and Transformer ( 13 ) to model long-horizon action sequences, while using different noise seeds to generate diverse sub-policies without training multiple models, significantly reducing the need for expert demonstrations; (2) integrates divergence regularization with diffusion sampling diversity to enhance behavioral differences among sub-policies through dynamic discrepancy measurement, noise perturbation, and sequence-level constraints; (3) expands training data by synthesizing trajectories through diffusion and then combines them with real data to train dynamics models. This effectively bridges offline generative strategies with online fine-tuning. Our framework enhances policy representation, diversity, and generalization, improving O2O-RL for complex robotic tasks. On the D4RL benchmark, it surpasses state-of-the-art baselines, especially in dexterous manipulation and quadruped locomotion, showing strong stability and adaptability. 2 METHOD In this section, we present our method, UEPO, as shown in Fig. 1 . The three core innovations of this framework are seamlessly integrated into the learning pipeline spanning the offline and online phases. Figure 1: UEPO employs a multi-seed diffusion sampling strategy to initialize components for the subsequent phase. During the offline optimization stage (middle), the strategy enhances diversity through a regularization mechanism that amplifies policy divergence, whilst simultaneously training the dynamics model \hat{T} using a joint approach of real data and synthetic trajectories to improve its generalization capability. Finally, a qualifying policy is selected as the initialization for online fine-tuning. 2.1 Conditional Action Sequence Generation via Diffusion Model To address limitations of BC, we adopt a state-conditional diffusion policy to model the entire action sequence distribution p(a_{1:T}\mid s_{1:T}) , capturing long-horizon dependencies and multi-modal behaviors in offline data. We then construct an ensemble policy via multi-seed sampling to enhance sequence rationality and behavioral diversity. Traditional ensemble methods incur high computational costs due to the training of multiple independent models. Instead, we construct an ensemble of n sub-policies \{\pi_{\theta}^{i}\}_{i=1}^{n} from a single trained diffusion model by varying initial noise seeds during reverse sampling. For each sub-policy \pi_{\theta}^{i} , we condition on the same state sequence s_{1:T} but initialize the reverse process with a distinct random seed \epsilon_{i}\sim\mathcal{N}(0,\mathbf{I}) . Each unique seed generates an action sequence a_{1:T}^{i} corresponding to a distinct behavioral modality, thus reducing training costs while naturally promoting sub-policy diversity. 2.2 Divergence Regularization Enhancement Guided by Diffusion Sampling The ensemble policy constructed in Section 2.1 provides initial diversity. However, to ensure sub-policies exhibit divergence during dynamic execution, we introduce dynamic divergence constraints directly into the diffusion sampling process. This contrasts with Uni-O4’s ( 8 ) approach, which applies a KL divergence penalty to the distribution of single-step actions and may result in policies that are statically distinct but dynamically similar or mutually conflicting. 2.2.1 Dynamic Divergence Constraint When generating action sequence a_{i} for the i -th sub-policy, we measure its divergence from other sub-policies \{a_{j}\mid j<i\} that have already been generated within the same sampling round. We introduce a divergence reward based on dynamics-level discrepancies to adjust the sampling path. • Dynamic Divergence Metric: We define the divergence between two action sequences a_{i} and a_{j} by measuring the dynamics difference between first-order (velocity) and second-order (acceleration): \text{div}(a_{i},a_{j})=\frac{1}{T}\sum_{t=1}^{T}\left(\|\dot{a}_{i,t}-\dot{a}_{j,t}\|_{2}+(1-\cos(\ddot{a}_{i,t},\ddot{a}_{j,t}))\right) where velocity \dot{a}_{t}=a_{t}-a_{t-1} and acceleration \ddot{a}_{t}=\dot{a}_{t}-\dot{a}_{t-1} , to ensure that discrepancies are meaningful at the level of physical execution. • Adaptive Perturbation: If the divergence \text{div}(a_{i},a_{j}) falls below a threshold \tau , we interpret the paths as being too similar. To encourage exploration, we perturb the current denoised estimate a_{t}^{i} in the reverse process: a_{t}^{i}\leftarrow a_{t}^{i}+\delta,\quad\delta\sim\mathcal{N}(0,\sigma_{\text{div}}^{2}\mathbf{I}),\quad\text{where }\sigma_{\text{div}}=\eta\cdot\frac{\tau-\text{div}(a_{i},a_{j})}{\tau} The scaling factor \sigma_{\text{div}} increases as the divergence decreases, and \eta is a hyperparameter controlling the perturbation strength. This adaptive noise injection forces the sub-policy to explore distinct dynamic modes. 2.2.2 Synergy with Sequence-Level KL Regularization We retain the KL divergence penalty from Uni-O4 ( 8 ) to ensure distributional diversity at a global level. However, we redefine its application from the single-step action distribution to the entire action-sequence distribution, which aligns naturally with our sequence-based diffusion policy. The overall objective for each sub-policy \hat{\pi}^{i} is: J(\hat{\pi}^{i})=\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[\log p_{\theta}(a\mid s)\right]+\alpha\,\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[\log\left(\frac{p_{\theta}(a\mid s)}{\max_{j}p_{\theta}(a\mid s)}\right)\right] Here, p_{\theta}(a\mid

Read the complete text