Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning
Chi Zhang, Ziying Jia, George K. Atia, Sihong He, Yue Wang · 2025
Read Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning by Chi Zhang, Ziying Jia, George K. Atia, Sihong He, Yue Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18447.txt.
Introduction
Chi Zhang, Ziying Jia, George Atia, Sihong He, and Yue Wang built a transfer learning framework for reinforcement learning in two thousand twenty-five. Transfer aims to solve a new task with little data by borrowing from related tasks. Transferred policies can act badly, and poor sources can hurt rather than help.
Their framework rests on pessimism: it builds and optimizes a conservative estimate of performance in the target setting. That gives an optimized lower bound, which supports safe decisions. Performance also improves steadily as source quality rises, avoiding negative transfer. They build two kinds of conservative estimates, prove their value, and give distributed algorithms with convergence guarantees.
Catalog record
| CREATOR | Chi Zhang, Ziying Jia, George K. Atia, Sihong He, Yue Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 139 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18447
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning Chi Zhang, Ziying Jia, George K. Atia, Sihong He, Yue Wang Abstract Transfer reinforcement learning aims to derive a near-optimal policy for a target environment with limited data by leveraging abundant data from related source domains. However, it faces two key challenges: the lack of performance guarantees for the transferred policy, which can lead to undesired actions, and the risk of negative transfer when multiple source domains are involved. We propose a novel framework based on the pessimism principle, which constructs and optimizes a conservative estimation of the target domain’s performance. Our framework effectively addresses the two challenges by providing an optimized lower bound on target performance, ensuring safe and reliable decisions, and by exhibiting monotonic improvement with respect to the quality of the source domains, thereby avoiding negative transfer. We construct two types of conservative estimations, rigorously characterize their effectiveness, and develop efficient distributed algorithms with convergence guarantees. Our framework provides a theoretically sound and practically robust solution for transfer learning in reinforcement learning. 1 Introduction Reinforcement learning (RL) aims to learn a policy that optimizes an agent’s performance when interacting with its environment. RL has achieved remarkable success in areas such as game playing ( Silver et al., 2016 ; Silver et al., 2017 ; Mnih et al., 2013 ) , robotics ( Nguyen & La, 2019 ; Liu et al., 2021 ) or natural language processing ( Sharma & Kaushik, 2017 ; Uc-Cetina et al., 2023 ; Kirk et al., 2023a ) . These successes, however, often depend on access to vast amounts of data, which are critical for enabling agents to understand their environments and develop effective policies. In domains with limited data—due to high exploration costs or other constraints—RL performance typically degrades. A promising solution is transfer learning (TL), which leverages more data from related environments (source domains) to train policies that can be transferred to target environments. We focus on zero-shot transfer , where no target domain data is available prior to deployment, and multi-domain transfer , where multiple source domains are utilized. Despite its potential, TL is often hindered by the Sim-to-Real Gap ( Salvato et al., 2021 ; Zhao et al., 2020 ) , which refers to performance degradation when transferring learned policies from source to target domains. TL typically relies on constructing a proxy to approximate target domain performance and transferring the proxy’s optimal policy. While successful when source domains closely resemble the target domain (e.g., high-fidelity simulations), significant divergence between the source and target domains—caused by modeling errors, non-stationarity, or perturbations—leads to inaccurate proxies and degraded transfer performance. In multi-domain transfer, the presence of multiple source domains introduces additional challenges, particularly negative transfer. Source domains that differ significantly from the target domain can skew the proxy, resulting in overly pessimistic or suboptimal policies that hinder the transfer process if they cannot be identified and treated properly. These challenges are critical in practical applications, where ineffective policies may have severe consequences, such as mechanical damage in robotics or traffic accidents in autonomous driving. Existing TL methods, such as domain randomization (DR) or imitation learning, lack systematic guarantees for transferred performance, as they fail to establish a robust connection between the proxy and target domain outcomes. To address these limitations, and inspired by the effectiveness of pessimism principle in various RL settings, particularly in scenarios where conservativeness is advantageous, such as offline RL ( Jin et al., 2020 ; Shi et al., 2022 ) , we propose a novel framework that incorporates the pessimism principle in proxy construction for TL. By using a pessimistic estimation of target domain performance, our approach ensures that transferred policies achieve satisfactory outcomes while avoiding severe consequences from overly optimistic actions. Our framework also mitigates negative transfer in multi-domain settings. We rigorously characterize the effectiveness of this approach and design concrete algorithms with convergence guarantees. Our contributions are summarized as follows. Introducing pessimism principle in transfer learning. We introduce the pessimism principle to transfer learning, enabling the transfer of conservative yet effective policies to the target domain. We characterize the framework’s effectiveness, showing that the target domain’s optimality gap depends on the degree of pessimism used in proxy construction. This motivates the design of proxies with minimal but sufficient pessimism, ensuring robust lower bounds on transferred performance and avoiding the severe consequences common in other TL methods. Construction of pessimistic proxy and design of concrete algorithms. Leveraging the inherent pessimism in robust RL frameworks, we construct pessimistic proxies by incorporating prior knowledge of domain similarities, such as upper bounds on the distance between source and target domains. We develop several proxy types, demonstrate their effectiveness, and propose a distributed algorithm for multi-domain transfer. This algorithm ensures convergence and yields conservative but effective policies across diverse source domains. Design of proxy and algorithm to avoid negative transfer. In multi-domain transfer, some source domains may significantly diverge from the target, leading to overly pessimistic policies if treated equally. We show that the pessimism principle inherently mitigates negative transfer by improving the proxy’s value without overly optimistic assumptions. Building on this insight, we propose a refined proxy that avoids negative transfer and design a convergent algorithm to ensure effective policy transfer in such settings. 2 Preliminaries and Problem Formulation Markov Decision Process. A Markov Decision Process (MDP) serves as the standard framework for formulating RL. An MDP can be represented by a tuple \mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma) , where \mathcal{S} and \mathcal{A} denote the state and action spaces, respectively, P:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S}) 1 1 1 \Delta(\cdot) denotes the probability simplex defined on the space. denotes the transition kernel, and we denote the probability vector under (s,a) -pair by P^{a}_{s} , r:\mathcal{S}\times\mathcal{A}\rightarrow[0,1] is the deterministic reward function, and \gamma\in[0,1] is the discount factor. At the t -th step, the agent selects an action a^{t} at state s^{t} , transitions to the next state s^{t+1} according to the transition probability P_{s^{t}}^{a^{t}} , and receives a reward r(s^{t},a^{t}) . A stationary policy \pi:\mathcal{S}\rightarrow\Delta(\mathcal{A}) maps each state to a probability distribution over \mathcal{A} , and \pi(a|s) gives the probability of selecting action a at state s . The performance of the agent following a policy is measured by the expected cumulative reward, defined as the value function V_{P}^{\pi} of policy \pi with transition kernel P : V_{P}^{\pi}(s)\triangleq\mathbb{E}_{P}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s^{t},a^{t})\Big|s^{0}=s,\pi\right], for all s\in\mathcal{S} . Alternatively, the cumulative reward can also be characterized by the Q-function Q_{P}^{\pi} for all (s,a)\in\mathcal{S}\times\mathcal{A} defined as Q_{P}^{\pi}(s,\!a)\!\triangleq\!\mathbb{E}_{P}\!\!\left[\sum_{t=0}^{\infty}\!\gamma^{t}r(s^{t},\!a^{t})\Big|s^{0}=s,a^{0}=a,\pi\right] . The goal of an MDP is to learn the optimal policy \pi^{*} : \pi^{*}\triangleq\arg\max_{\pi\in\Pi}V^{\pi}_{P}(s),\forall s\in\mathcal{S}, which exists and can be restricted to the set of deterministic policies ( Puterman, 2014 ) . The optimal value functions are V_{P}^{\ast}\triangleq\max_{\pi}V_{P}^{\pi}=V^{\pi^{*}}_{P} and Q_{P}^{\ast}\triangleq\max_{\pi}Q_{P}^{\pi}=Q^{\pi^{*}}_{P} . Robust Markov Decision Process (RMDP). Robust RL aims to optimize the performance under the worst case, when the environment is uncertain, and it is generally formulated as a robust MDP. The transition kernel in a robust MDP is not fixed but lies in an uncertainty set \mathcal{P} . In this paper, we focus on (s,a) -rectangular uncertainty sets ( Nilim & El Ghaoui, 2004 ; Iyengar, 2005 ) . Given a nominal kernel P_{0} , the uncertainty set \mathcal{P} is defined as \displaystyle\mathcal{P}\triangleq\otimes_{(s,a)\in\mathcal{S}\times\mathcal{A}}\mathcal{P}_{s}^{a}, (1) where