WOMBET: World Model-based Experience Transfer for Robust and Sample-efficient Reinforcement Learning

Mintae Kim, Koushil Sreenath · 2026

Read WOMBET: World Model-based Experience Transfer for Robust and Sample-efficient Reinforcement Learning by Mintae Kim, Koushil Sreenath (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08958.txt.

Introduction

Kim and Sreenath address a robotics bottleneck, where gathering real-world training data is costly and risky, by moving experience from a source task to a new target task. Their WOMBET framework learns a world model in the source task and generates prior data from it. That data moves to the target task under an uncertainty-penalized objective that lower-bounds the true return. Tests on continuous control benchmarks show better sample efficiency and final performance than strong baselines, with theory separating mismatch from approximation error.

Catalog record

CREATORMintae Kim, Koushil Sreenath
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30619
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08958

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

WOMBET: World Model-based Experience Transfer for Robust and Sample-efficient Reinforcement Learning Mintae Kim, Koushil Sreenath Abstract Reinforcement learning (RL) in robotics is often limited by the cost and risk of data collection, motivating experience transfer from a source task to a target task. Offline-to-online RL leverages prior data but typically assumes a given fixed dataset and does not address how to generate reliable data for transfer. We propose World Model-based Experience Transfer (WOMBET), a framework that jointly generates and utilizes prior data. WOMBET learns a world model in the source task and generates offline data via uncertainty-penalized planning, followed by filtering trajectories with high return and low epistemic uncertainty. It then performs online fine-tuning in the target task using adaptive sampling between offline and online data, enabling a stable transition from prior-driven initialization to task-specific adaptation. We show that the uncertainty-penalized objective provides a lower bound on the true return and derive a finite-sample error decomposition capturing distribution mismatch and approximation error. Empirically, WOMBET improves sample efficiency and final performance over strong baselines on continuous control benchmarks, demonstrating the benefit of jointly optimizing data generation and transfer. \coltauthor Hybrid Robotics, UC Berkeley Berkeley, CA 94720, USA keywords Offline-to-Online RL, Experience Transfer, World Models 1 Introduction Reinforcement learning assumes abundant interaction and frequent resets–conditions rarely satisfied in real-world robotics, where data collection is expensive and unsafe 25 ; 20 ; 8 . As a result, RL methods achieve strong asymptotic performance but remain sample-inefficient. Like biological systems that continually reuse past experience through internal world models, we seek to enable experience transfer : leveraging data collected in a source environment to improve sample efficiency and robustness in a target environment 33 ; 17 . Existing paradigms address this only partially. Online RL adapts to new tasks but requires extensive interaction and faces distributional shift between source and target experiences 6 ; 37 ; 13 . Offline RL avoids unsafe exploration but but degrades under limited or mismatched data 35 ; 16 ; 15 ; 34 . Applying it to real-world robotics often requires high-quality expert demonstrations or extensively tuned model-based controllers–assumptions that are unrealistic in many cases 4 ; 12 ; 14 . Offline-to-online RL combines these by pretraining offline and fine-tuning online 22 ; 19 ; 26 ; 29 ; 28 , but assumes access to high-quality offline data without addressing how to obtain or safely reuse it. Model-based RL (MBRL) provides a principled way to generate data by learning a predictive world model. Planning-based approaches use the world model for control via model predictive control (MPC) 5 ; 32 , while Dyna-style methods use the model to generate synthetic rollouts for policy learning 11 ; 31 . However, both faces limitations for transfer: MPC-based methods tend to generate low-diversity trajectories due to exploitation, and model-generated data can introduce bias when used without reliability control. Consequently, existing approaches struggle to provide reliable and adaptable data for transfer. We propose World Model-based Experience Transfer (WOMBET), a unified framework that couples uncertainty-aware model-based offline data generation with online fine-tuning through iterative model-policy co-evolution . In each iteration, a world model generates trajectories in the source task via uncertainty-penalized MPC, and a dual criterion–high return and low epistemic uncertainty–filters reliable rollouts to form an offline dataset D_{S} . WOMBET then performs off-policy policy optimization in the target task, to maximize a provable lower bound on the true return, while adaptive data mixing balances bias and variance between offline ( D_{S} from the source task) and online ( D_{T} collected in the target task) experience. The world model is continuously refined using the aggregated dataset D=D_{S}\cup D_{T} , improving prediction accuracy and expanding the reliable planning region. This alternating process unifies the exploitation strength of MBRL with the exploration capability of model-free RL. Our contributions are threefold: (1) Coupled data generation and transfer. We introduce a framework that jointly generates and utilizes prior data for transfer, instead of assuming a fixed offline dataset. WOMBET uses uncertainty-aware planning to construct data and iteratively refines both the world model and policy. (2) Reliable data curation and adaptive transfer. We propose (i) a dual-criterion filter selecting trajectories with high return and low epistemic uncertainty, and (ii) an adaptive sampling strategy that balances offline and online data based on TD error. (3) Theoretical and empirical validation. We show that uncertainty-penalized planning maximizes a lower bound on the true return and derive a finite-sample error decomposition capturing distribution mismatch and approximation error. Experiments demonstrate improved sample efficiency and final performance over strong baselines on continuous control benchmarks. Figure 1: WOMBET pipeline: uncertainty-aware model-based data generation in the source task, followed by adaptive offline-to-online learning with iterative model updates. 2 Related Work Offline-to-online RL. This paradigm combines offline data efficiency with online adaptability to accelerate learning when interaction is costly 21 ; 30 ; 26 . Early methods stabilize fine-tuning via advantage-weighted updates or constrained policy improvement 22 ; 3 , while later work improves robustness through fixed offline critics, ensemble critics, and transition weighting 19 . Architectural components such as symmetric replay buffers, ensemble critics, and layer normalization (LayerNorm) can further stabilize training without explicit regularization 2 ; 3 . Other studies explore pretraining on related tasks to enhance exploration or show that keeping the offline dataset is unnecessary when pre-trained rollouts suffice for initialization 36 . Behavior cloning and conservative value regularization are also used, but often require delicate tuning and may limit long-term performance 16 ; 34 . Learning-from-demonstration methods differ in relying on expert trajectories for imitation rather than addressing distributional shift during online fine-tuning 10 ; 23 . Overall, most methods treat offline data as static and independent of the target task, leaving data generation, critic stability, and robust exploration unresolved. MBRL for data generation and offline learning. Model-based approaches improve sample efficiency by learning a predictive world model and leveraging it either for planning or data generation. Planning-based methods perform control via MPC 5 ; 32 , while Dyna-style methods such as MBPO generate short synthetic rollouts to train model-free policies 11 . MOPO introduces uncertainty-based reward penalties to mitigate model exploitation 35 , and COMBO extends this with conservative value estimation 34 . The main difficulty in model-based offline RL is model bias : compounding prediction errors can degrade value estimation and lead to unsafe policies 26 . To address this, ensemble-based uncertainty penalizes unreliable or out-of-distribution (OOD) rollouts, often combined with behavior regularization or conservative value estimation. However, most methods use the learned model only once for rollout generation or regularization, without iterative refinement. WOMBET instead alternates between uncertainty-penalized data generation and online fine-tuning, updating both model and policy using aggregated experience. This iterative process unifies model-based offline RL and online adaptation for experience transfer. 3 Preliminaries We consider an MDP \mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\mu_{0},\gamma) with state space \mathcal{S} , action space \mathcal{A} , transition kernel P(s^{\prime}|s,a) , reward r(s,a) , initial distribution \mu_{0} , and discount factor \gamma\in[0,1) . A policy \pi(a|s) aims to maximize the discounted return J(\pi)=\mathbb{E}_{\pi,P}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})] . We study two tasks sharing dynamics but differing in reward and initial state: the source task \mathcal{M}_{S}=(\mathcal{S},\mathcal{A},P,r_{S},\mu_{0}^{S},\gamma) and the target task \mathcal{M}_{T}=(\mathcal{S},\mathcal{A},P,r_{T},\mu_{0}^{T},\gamma) . The objective is to maximize the target return J_{T}(\pi)=\mathbb{E}_{\pi,P}[\sum_{t=0}^{\infty}\gamma^{t}r_{T}(s_{t},a_{t})] , using

Read the complete text