MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning

Yihong Guo Yu Yang Pan Xu Anqi Liu · 2025

Read MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning by Yihong Guo Yu Yang Pan Xu Anqi Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08460.txt.

Introduction

Yihong Guo, Yu Yang, Pan Xu, and Anqi Liu present MOBODY, a reinforcement learning method for when the world changes. Their twenty twenty-five paper studies offline learning in a new domain whose dynamics differ from the source data. Past methods either dock the reward or drop source moves from high-shift zones, which confines the policy to safe regions.

MOBODY instead learns a model of the target dynamics and plans with it. Since the same next state may need distinct actions in each domain, it gives each domain its own action encoder while sharing the state and transition parts. A cloning loss, weighted by target Q-values, keeps the policy near good target actions. On standard benches it beat past methods.

Catalog record

CREATORYihong Guo Yu Yang Pan Xu Anqi Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08460

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning Yihong Guo Yu Yang Pan Xu Anqi Liu Abstract We study the off-dynamics offline reinforcement learning (RL) problem, where the goal is to learn a policy from offline datasets collected from source and target domains with mismatched transition dynamics. Existing off-dynamics offline RL methods typically either filter source transitions that resemble those of the target domain or apply reward augmentation to source data, both constrained by the limited transitions available from the target domain. As a result, the learned policy is unable to explore target domain beyond the offline datasets. We propose MOBODY, a Model-Based Off-Dynamics offline RL algorithm that addresses this limitation by enabling exploration of the target domain via learned dynamics. MOBODY generates new synthetic transitions in the target domain through model rollouts, which are used as data augmentation during offline policy learning. Unlike existing model-based methods that learn dynamics from a single domain, MOBODY tackles the challenge of mismatched dynamics by leveraging both source and target datasets. Directly merging these datasets can bias the learned model toward source dynamics. Instead, MOBODY learns target dynamics by discovering a shared latent representation of states and transitions across domains through representation learning. To stabilize training, MOBODY incorporates a behavior cloning loss that regularizes the policy. Specifically, we introduce a Q-weighted behavior cloning loss that regularizes the policy toward actions with high target-domain Q-values, rather than uniformly imitating all actions in the dataset. These Q-values are learned from an enhanced target dataset composed of offline target data, augmented source data, and rollout data from the learned target dynamics. We evaluate MOBODY on standard MuJoCo benchmarks and show that it significantly outperforms state-of-the-art baselines, with especially pronounced improvements in challenging scenarios where existing methods struggle. 1 Introduction Reinforcement learning (RL) ( Kaelbling et al. 1996 ; Li 2017 ) aims to learn a policy that maximizes cumulative reward by interacting with an environment and collecting the corresponding rewards. While RL has led to impressive successes in many domains, real-world applications, such as autonomous driving ( Kiran et al. 2021 ) and healthcare ( Lee et al. 2023 ) , face significant constraints on interaction with the environment due to safety or cost concerns. One solution is to learn a policy from a pre-collected offline dataset ( Levine et al. 2020 ) . Still, when the offline dataset is insufficient, data from another environment, such as a simulator with potentially mismatched dynamics, may be needed but requires further domain adaptation. In our paper, we study a specific type of domain adaptation in RL, called off-dynamics offline RL ( Liu et al. 2022 ; Liu et al. 2024a ; Lyu et al. 2024b ) , where the simulator (source) and real/deployed (target) environments differ in their transition. The agent is not allowed to interact with the environment but only has access to offline data that are pre-collected from the two domains with mismatched dynamics and trains a policy with the offline data. Figure 1 : Comparison between DARA ( Liu et al. 2022 ) (a model-free reward augmentation method), MOPO ( Yu et al. 2020 ) (a vanilla model-based offline RL), and MOBODY on two MuJoCo environments with different dynamics shift type and level. We find that MOBODY substantially outperforms DARA and MOPO by exploring the target environment. For experiment details, see Section 5 . Existing works on off-dynamics offline RL solve the problem by 1) reward shaping that accounts for the dynamics shift ( Liu et al. 2022 ; Wang et al. 2024 ; Lyu et al. 2024a ) or 2) filtering the source transition that is similar to the target transition ( Xu et al. 2023 ; Wen et al. 2024 ; Liu et al. 2024a ) . All of them are model-free methods which lack the exploration in the target domain and thus might theoretically be unable to find the optimal policy if the offline dataset has poor coverage of the optimal policy ( Zhan et al. 2022 ) . Model-based methods have been proven effective in both offline and online RL, which can be more sample-efficient and allow for more exploration of the environment with learned dynamics. In this paper, we propose a Model-Based Off-Dynamics RL algorithm (MOBODY) that first learns target domain dynamics and rollouts new data from the learned dynamics during policy optimization. However, learning the target dynamics using a mixed dataset with dynamics shift is harder compared to learning dynamics for a single domain in existing model-based offline RL ( Yu et al. 2020 ; Yu et al. 2021 ; Rigter et al. 2022 ) . First, the target data are limited, so we cannot directly learn the target dynamics only using the target data. Second, the existing dynamics learning methods learn the dynamics for a single domain and cannot be directly applied or adapted to other domains. Though straightforwardly we can learn dynamics by directly combining the source and target data, the learned dynamics will not work for the target domain due to the dynamics mismatch. As shown in Figure 1 , MOPO that directly learns dynamics with combined source and target data significantly underperforms MOBODY which is specifically optimized to learn the target dynamics. This justifies the need of a novel dynamics learning algorithm that learns target dynamics with both source data and limited target data for model-based off-dynamics RL. The proposed MOBODY algorithm first learns a target dynamics with source data and limited target data, and then learns a policy with enhanced target domain data consisting of reward augmented offline source data, offline target data and rollout data from the learned dynamics. For dynamics learning, MOBODY learns shared latent representations of states and transitions respectively across domains through representation learning by novel representation encoder loss and a novel cycle transition loss, which takes advantage of the data from both source and target domains. After we learn the target dynamics, we roll out new data with the learned dynamics during the policy optimization and optimize the policy with enhanced target data consisting of reward augmented source data, target data and new rollout data. To stabilize the policy optimization training especially when data are out of distribution, we utilize the behavior cloning loss to regularize the policy ( Fujimoto and Gu 2021 ) . However, vanilla behavior cloning loss will push the policy to favor the action in the source data, but the action in the source data might not perform well in the target domain due to the dynamics shift. Inspired by the advantage-weighted regression ( Peters and Schaal 2007 ; Kostrikov et al. 2021a ) , MOBODY incorporates a Q-weight behavior cloning loss regularization, which will up-weight action with the high target Q value, while the target Q function is learned with the aforementioned enhanced target data . Our contribution can be summarized as follows: • We propose a novel paradigm for off-dynamics offline RL, called model-based off-dynamics offline RL, that can explore the target domain with the learned target transitions. • We propose a novel framework for learning the target dynamics with source data and limited target data by learning a shared state representation and the transition in the latent space. For the offline RL, we

Read the complete text