Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learning

Zhenghai Xue, Lang Feng, Jiacheng Xu, Kang Kang, Xiang Wen, Bo An, Shuicheng YAN · 2025

Read Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learning by Zhenghai Xue, Lang Feng, Jiacheng Xu, Kang Kang, Xiang Wen, Bo An, Shuicheng YAN (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06893.txt.

Introduction

Zhenghai Xue and co-authors improve robot learning across changed settings in a 2025 paper. A common route copies expert state paths gathered under other conditions. Copying has a flaw: the learner can never beat the expert it copies. A second flaw is that some expert states cannot be reached at all once conditions change.

The new frame joins reward seeking with copying, guided by a distance based penalty. It holds the learner close only on states that can be reached in any setting. States that are out of reach in the new setting no longer drag the learner down. The aim is policies that adapt when the world shifts, rather than mimic a fixed path.

Catalog record

CREATORZhenghai Xue, Lang Feng, Jiacheng Xu, Kang Kang, Xiang Wen, Bo An, Shuicheng YAN
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06893

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learning Zhenghai Xue, Lang Feng, Jiacheng Xu, Kang Kang, Xiang Wen, Bo An, Shuicheng YAN Abstract To learn from data collected in diverse dynamics, Imitation from Observation (IfO) methods leverage expert state trajectories based on the premise that recovering expert state distributions in other dynamics facilitates policy learning in the current one. However, Imitation Learning inherently imposes a performance upper bound of learned policies. Additionally, as the environment dynamics change, certain expert states may become inaccessible, rendering their distributions less valuable for imitation. To address this, we propose a novel framework that integrates reward maximization with IfO, employing ℱ \mathcal{F} -distance regularized policy optimization. This framework enforces constraints on globally accessible states—those with nonzero visitation frequency across all considered dynamics—mitigating the challenge posed by inaccessible states. By instantiating ℱ \mathcal{F} -distance in different ways, we derive two theoretical analysis and develop a practical algorithm called A ccessible S tate O riented Policy R egularization (ASOR). ASOR serves as a general add-on module that can be incorporated into various RL approaches, including offline RL and off-policy RL. Extensive experiments across multiple benchmarks demonstrate ASOR’s effectiveness in enhancing state-of-the-art cross-domain policy transfer algorithms, significantly improving their performance. 1 Introduction Imitation Learning (IL) and Reinforcement Learning (RL) ( Sutton & Barto, 1998 ) facilitates large-scale policy optimization using datasets that are either static ( Wu et al., 2019 ; Fujimoto et al., 2019 ) or continuously updated during training ( Lillicrap et al., 2016 ; Haarnoja et al., 2018 ) . When applied to non-stationary environments with evolving dynamics, obtaining effective policies necessitates extensive training datasets with sufficient dynamics coverage ( Liu et al., 2022 ; Li et al., 2023 ) . This challenge arises because optimal policies and state-action distributions are inherently dynamics-specific and do not generalize well. As a result, trajectory data collected under one dynamics cannot be directly utilized for policy learning in another. To address inefficient data exploitation, IfO algorithms ( Wu et al., 2019 ; Torabi et al., 2018b ; Jiang et al., 2020 ) propose a dynamics-agnostic approach that enables learning from data with varying dynamics. It imitates stationary state distributions of expert policies and relies on the idea that expert state distributions are similar across different dynamics ( Gangwani & Peng, 2020 ; Desai et al., 2020 ; Radosavovic et al., 2021 ) . Building on this idea, Xue et al. (2023a) further shows that the learning policy can achieve strong cross-dynamics performance if it recovers the expert state distribution in at least one of the dynamics. Instead, we highlight a key limitation of this idea: the similarity of state distributions does not always hold, particularly in tasks where state accessibility varies with environmental dynamics. In such cases, certain expert states become inaccessible as the environment changes. For example, an autonomous vehicle may safely navigate intersections at high speed under low traffic densities, but will face a high risk of collisions in dense traffic. Consequently, states representing “safe driving at high speed” become inaccessible in certain dynamics, leading to distinct stationary state distributions. In such scenarios, expert trajectories with dynamics shift can be misleading. To deal with the issue of distinct state accessibility, expert states that are not visited in some dynamics should be excluded during training. We define globally accessible states which maintains the same accessibility across different dynamics. By restricting imitation to these states, the policy can still leverage expert state trajectories from various dynamics while avoiding misleading information from inaccessible states. Meanwhile, IL cannot be naturally integrated with datasets containing reward signals and requires demonstrations to be optimal. To overcome this limitation, we propose a policy regularization method that incorporates imitation as constraint while optimizing the reward maximization objective in RL. Specifically, the constraint ensures that the \mathcal{F} -distance ( Arora et al., 2017 ) between the accessible state distributions of the current and expert policies remains upper-bounded. This approach offers two key advantages: By instantiating the \mathcal{F} -distance with JS divergence and network distance, we establish lower-bound performance guarantees for policy regularization under dynamics shift; By instantiating the \mathcal{F} -distance with a GAN-like distance measure, we transform policy regularization into a practical reward augmentation algorithm which can be a general add-on module to existing cross-dynamics RL algorithms ( Chen et al., 2021 ; Luo et al., 2022 ) . In empirical evaluations, we access the proposed algorithm across various environments, including Minigrid ( Chevalier-Boisvert et al., 2023 ) , the simulated robotics environment MuJoCo ( Todorov et al., 2012 ) , the simulated autonomous-driving environment MetaDrive ( Li et al., 2023 ) , and a large-scale fall guys-like game environment. The proposed algorithm exhibits superior performance when integrated with multiple state-of-the-art algorithms. Our contributions can be summarized as follows: 1) We identify a common limitation of existing IfO methods under dynamics shift and propose state distribution imitation restricted to globally accessible states; 2) We design an \mathcal{F} -distance regularized policy optimization framework that combines expert imitation with reward maximization; 3) By instantiating the \mathcal{F} -distance in different ways, we conduct theoretical analyses and introduce a practical algorithm, both validating the effectiveness of policy regularization on globally accessible states. 2 Backgroud 2.1 Preliminaries To model a set of decision-making tasks with different environment dynamics, we consider the Hidden Parameter Markov Decision Process (HiP-MDP) ( Doshi-Velez & Konidaris, 2016 ) defined by a tuple (\mathcal{S},\mathcal{A},\Theta,T,r,\gamma,\rho_{0}) , where \mathcal{S} is the state space and \mathcal{A} is the bounded action space with actions a\in[-1,1] . \Theta is the space of hidden parameters. T_{\theta}(s^{\prime}|s,a) is the transition function conditioned on (s,a) , as well as a hidden parameter \theta sampled from \Theta . r(s,a,s^{\prime}) is the environment reward function. By taking all s,a,s^{\prime} into account, the reward function inherently includes the transition information and does not change in different dynamics. We also assume r(s,a,s^{\prime}) w.r.t. the action a is \lambda -Lipschitz. Discussions on these Lipschitz properties can be found in Appendix A.3 . s^{\prime} is termed as accessible from s under dynamics T 1 1 1 T without subscript \theta refers to the transition function under any of the hidden parameter \theta . if \sum_{a\in\mathcal{A}}T(s^{\prime}|s,a)>0 . \gamma\in(0,1) is the discount factor and \rho_{0}(s) is the initial state distribution. Policy optimization under dynamics shift aims at finding the optimal policy that maximizes the expected return under all possible \theta\in\Theta : \pi^{*}=\argmax_{\pi}\eta(\pi)=\mathbb{E}_{\theta}\mathbb{E}_{\pi,T_{\theta}}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t},s_{t+1})\right] , where the expectation is under s_{0}\sim\rho_{0} , a_{t}\sim\pi(\cdot|s_{t}) , and s_{t+1}\sim T_{\theta}(\cdot|s_{t},a_{t}) . The Q-value Q_{T}^{\pi}(s,a) denotes the expected return after taking action a at state s : Q_{T}^{\pi}(s,a) =E_{\pi,T}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t},s_{t+1})|s_{0}=s,a_{0}=a\right] . The value function is defined as V_{T}^{\pi}(s)=\mathbb{E}_{a\sim\pi(\cdot|s)}Q_{T}^{\pi}(s,a) . The optimal policy \pi^{*} under T is defined as \pi_{T}^{*}=\argmax_{\pi}\mathbb{E}_{s\sim\rho_{0}}V_{T}^{\pi}(s) . We also intensely use the stationary state distribution (also referred to as the state occupation function) d_{T}^{\pi}(s)=(1-\gamma) \sum_{t=0}^{\infty}\gamma^{t}p\left(s_{t}=s\mid\pi,T\right) . The stationary state distribution under the optimal policy is denoted as d^{*}_{T}(s) , which is the shorthand for d^{\pi_{T}^{*}}_{T}(s) . d^{*}_{T}(s) will be briefly termed as optimal state distribution in the rest of this paper. 2.2 Related Work Cross-domain Policy Transfer Cross-domain policy transfer ( Niu et al., 2024 ) focuses on training policies in source domains and testing them in the target domain. In this paper, we

Read the complete text