Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning

Yassine Chemingui, Aryan Deshwal, Honghao Wei, Alan Fern, Janardhan Rao Doppa · 2024

Read Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning by Yassine Chemingui, Aryan Deshwal, Honghao Wei, Alan Fern, Janardhan Rao Doppa (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18946.txt.

Introduction

Yassine Chemingui, Aryan Deshwal, and three colleagues help a safety-trained agent adapt when the safety budget changes after training, in a paper published in two thousand twenty-four. Offline safe reinforcement learning learns from a fixed log of past behavior, maximizing reward while respecting cost constraints. Retraining for every new budget is impractical.

Their framework, CAPS, wraps around existing offline algorithms. In training, it learns several policies that share one representation but strike different reward-versus-cost trade-offs. At deployment, it switches between them state by state. In each state it picks the policy promising the best future reward among those that still satisfy the current cost limit. Tests on thirty-eight tasks from a standard safety benchmark show the switcher consistently beating rival methods. The trick is to prepare a portfolio of caution levels in advance. Deployment then becomes a matter of choosing, not relearning.

Catalog record

CREATORYassine Chemingui, Aryan Deshwal, Honghao Wei, Alan Fern, Janardhan Rao Doppa
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18946

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning Yassine Chemingui, Aryan Deshwal, Honghao Wei, Alan Fern, Janardhan Rao Doppa Abstract Offline safe reinforcement learning (OSRL) involves learning a decision-making policy to maximize rewards from a fixed batch of training data to satisfy pre-defined safety constraints. However, adapting to varying safety constraints during deployment without retraining remains an under-explored challenge. To address this challenge, we introduce constraint-adaptive policy switching (CAPS), a wrapper framework around existing offline RL algorithms. During training, CAPS uses offline data to learn multiple policies with a shared representation that optimize different reward and cost trade-offs. During testing, CAPS switches between those policies by selecting at each state the policy that maximizes future rewards among those that satisfy the current cost constraint. Our experiments on 38 tasks from the DSRL benchmark demonstrate that CAPS consistently outperforms existing methods, establishing a strong wrapper-based baseline for OSRL. The code is publicly available at https://github.com/yassineCh/CAPS . 1 Introduction Online reinforcement learning (RL) has shown great successes in applications where the agent can continuously interact with the environment to collect new feedback data to improve its decision-making policy ( Mnih et al. 2015 ; Silver et al. 2018 ; Li 2017 ) . However, in many real-world domains such as agriculture, smart grid and healthcare, executing exploratory decisions is costly and/or dangerous, but we have access to pre-collected datasets. Offline RL ( Levine et al. 2020 ) is an emerging paradigm to learn decision policies solely from such offline datasets, eliminating the need for additional interaction with the environment. The key challenge in offline RL is addressing the distributional shift between the state-action distribution of the offline dataset and the learned policy, as this shift results in extrapolation error. Existing offline RL methods typically employ the principle of pessimism in the face of uncertainty (i.e., some form of penalization to deal with out-of-distribution states/actions) to address this challenge ( Levine et al. 2020 ) . In many safety-critical domains (e.g., agriculture and smart grid), the agent behavior needs to also satisfy some cost constraints in addition to maximizing reward. Most of the prior work on offline RL focuses on the unconstrained setting where the policy is optimized to maximize rewards. There is relatively less work on offline constrained/safe RL and the majority of these methods assume that the cost constraint is known during the training phase (as elaborated in the related work section). However, in many applications, the cost constraints can change at deployment depending on the use-case scenario. For example, in agricultural drones, the reward is the area covered during spraying, and the cost is the battery/power consumption. The goal is to maximize efficiency for a given agricultural field by optimizing coverage based on the available battery power. This paper studies the problem of offline safe RL (OSRL) where the safety/cost constraints can vary after deployment. We propose Constraint-Adaptive Policy Switching (CAPS) , an algorithm that can be wrapped around existing offline RL methods and is easy to implement. CAPS handles the challenge of unknown cost constraint threshold as follows. At training time, we learn multiple policies with a shared representation with the goal of achieving different reward and cost trade-offs: one centered on reward maximization, one on cost minimization, and other policies optimized for varied linear scalarizations of costs and rewards. At testing time, CAPS switches between the decisions from the trained policies at each state, selecting the one that maximizes future rewards from the ones that satisfy the cost constraint. We also theoretically analyze the conditions under which CAPS decision-making process has safety guarantees. We perform experiments on 38 tasks from the DSRL benchmark ( Liu et al. 2024 ) to compare CAPS with state-of-the-art OSRL baselines and our main findings are as follows. First, CAPS wrapped around two qualitatively different offline RL algorithms exhibits safe behavior (first-order objective) on a larger fraction of tasks compared to baselines across different cost threshold constraints. Prior methods struggle even with known cost constraints, potentially because of estimation errors in value functions and/or instability in the Lagrange multiplier. Second, the implicit Q-learning instantiation of CAPS demonstrates safe performance in 34/38 tasks (89%) and achieves the highest rewards in 18 of those tasks compared to the baselines.. Third, CAPS using shared representation performs significantly better than CAPS with independently trained policies. Finally, increasing the number of policies improves the overall performance of CAPS, but CAPS with two policies (one for reward maximization and one for cost minimization) is competitive, has the advantage of removing extra hyperparameters, and stands as a strong minimalist approach for effective offline safe RL. Contributions. The key contribution is the development and evaluation of CAPS framework for offline safe RL with varying cost constraints. Specific contributions include: • Development of CAPS to handle any test-time cost threshold constraint. CAPS can be wrapped around existing offline RL algorithms and easy to implement. • Theoretical analysis to understand the conditions under which CAPS decision policy has safety guarantees. • Experiments and ablations on 38 DSRL tasks, demonstrate the effectiveness of CAPS over prior methods. 2 Problem Setup RL problems with safety constraints are naturally formulated within the Constrained Markov Decision Process (CMDP) framework. A CMDP is defined by a tuple M=(S,A,P,r,c,\mu_{0}) , where S is the state space, A is the action space, P:S\times A\times S\to[0,1] is the state transition probability function, r:S\times A\to\mathbb{R} defines the reward function, and c:S\to[0,C_{\text{max}}] quantifies the costs associated with states, where C_{\text{max}} is the maximum possible cost, and \mu_{0}:S\rightarrow[0,1] is the initial state distribution. For simplicity of notation, we define cost function on states noting that it is easy to define an equivalent CMDP with costs associated with state-action pairs. In offline safe RL problems, we are given a fixed pre-collected dataset \mathcal{D} = \{(s,a,r,c,s^{\prime})_{i}\}_{i=1}^{n} from one or more (unknown) behavior policies, where each training example i contains the action a taken at state s , reward received r , cost incurred c , and the next state s^{\prime} . The goal is to learn a policy \pi:S\to A from the offline dataset \mathcal{D} to maximize the expected reward while satisfying a specified cost/safety constraint. This problem is mathematically formulated as: \displaystyle\max_{\pi}\mathbb{E}_{\tau\sim\pi}[R(\tau)]\quad\mathrm{subject\ to}\quad\mathbb{E}_{\tau\sim\pi}[C(\tau)]\leq\kappa, Here, \kappa\in[0,+\infty) is the cost threshold for safety constraint, \tau=\{s_{1},a_{1},r_{1},c_{1},\ldots,s_{T},a_{T},r_{T},c_{T}\} denotes a trajectory sampled by executing the policy \pi , T is the length of episode, R(\tau)=\sum_{t=1}^{T}r_{t} is the total accumulated reward, and C(\tau)=\sum_{t=1}^{T}c_{t} is the total incurred cost. The majority of prior work on OSRL assumes that the cost threshold \kappa for safety constraint is known at the training time and remains same at deployment. However, this assumption is violated in many real-world applications where the safety requirements can change during deployment based on the operating conditions or environmental factors. For example, in agricultural crop management, the goal may be to maximize yield under constraints on water and/or nitrogen application depending on time-varying regulations. Additionally, in a decision-support scenario, it can be useful for decision makers to quickly explore the impact of different cost thresholds without requiring retraining. Other application domains where this problem setting arises include healthcare, power systems, and robotics. Consequently, OSRL methods that learn a single policy tailored to a specific cost constraint are ill-equipped to handle varying

Read the complete text