Safe Reinforcement Learning with Minimal Supervision Alexander Quessy, Thomas Richardson, Sebastian East Abstract Reinforcement learning (RL) in the real world necessitates the development of procedures that enable agents to explore without causing harm to themselves or others. The most successful solutions to the problem of safe RL leverage offline data to learn a safe-set, enabling safe online exploration. However, this approach to safe-learning is often constrained by the demonstrations that are available for learning. In this paper we investigate the influence of the quantity and quality of data used to train the initial safe learning problem offline on the ability to learn safe-RL policies online. Specifically, we focus on tasks with spatially extended goal states where we have few or no demonstrations available. Classically this problem is addressed either by using hand-designed controllers to generate data or by collecting user-generated demonstrations. However, these methods are often expensive and do not scale to more complex tasks and environments. To address this limitation we propose an unsupervised RL-based offline data collection procedure, to learn complex and scalable policies without the need for hand-designed controllers or user demonstrations. Our research demonstrates the significance of providing sufficient demonstrations for agents to learn optimal safe-RL policies online, and as a result, we propose optimistic forgetting, a novel online safe-RL approach that is practical for scenarios with limited data. Further, our unsupervised data collection approach highlights the need to balance diversity and optimality for safe online exploration. 1 Introduction Reinforcement learning (RL) is a powerful technique and has enabled robotic systems to complete a variety of tasks ( Lee et al., 2021 ; Haarnoja et al., 2018c ; Kalashnikov et al., 2018 ) in high-dimensional environments ( OpenAI et al., 2018 ) under uncertainty. Typically, modern RL agents learn through extensive unconstrained interaction with their environment, aiming to maximize a reward based objective function. However, unconstrained exploration is hazardous in the real-world, posing harm to both the agent and its environment. This presents a fundamental challenge to the real-world use of RL: understanding the effects of our actions to guarantee safety whilst not knowing the consequences of unsafe transitions until they are explored ( Dulac-Arnold et al., 2019 ) . Critically, we need to infer how our agent’s actions will impact its behavior before we deploy it into the real-world. Planning with LMPC ( Rosolia & Borrelli, 2016 ) offers a useful approach to predict the effect of a controller’s actions without explicitly sampling a possibly dangerous state. Generally, LMPC methods rely upon dense reward functions, to learn policies efficiently, but complex real-world tasks rarely offer an intuitive representation of task success with a simple dense reward function representation. Instead, we often perceive complex tasks as problems with temporally and/or spatially extended goal-states with a binary success criteria. For example an aerial robot landed or crashed. Directly encoding complex tasks with a simple binary reward function upon reaching the goal state offers little supervision during the learning process, degrading sample efficiency and requiring greater exploration. Often reward shaping ( Mataric, 1994 ) is used to address this problem, improving the density of the reward function at the cost of generalization and scalability. However, reward shaping masks the true problem, which is that fundamentally we need to provide more information to the agent if we want to learn efficiently, but naturally this will be task specific. This is often evident from the ineffectiveness of reward shaping in tasks with complex state-spaces, such as image based observations ( Tian et al., 2020 ; Levine et al., 2015 ; Nair et al., 2018 ) : they are simply too complicated for the designer to provide useful information through the reward function. A logical next step, when the task is too complex to learn from reward shaping alone, is to learn from demonstrations in the form of offline reinforcement learning ( Levine et al., 2020 ) . The combination of offline RL, safe-sets, and LMPC presents a promising method for safe-RL ( Gros et al., 2020 ; Richards et al., 2018 ) . This approach enables learning a sub-optimal control policy that can be refined online, with exploration restricted by the safe-set ( Rosolia & Borrelli, 2016 ) . To learn an effective safe online policy we require a dataset containing: • Goal reaching demonstrations , where the agent reaches the goal, to learn how to reach the goal state. • Constraint violating demonstrations , where the agent intentionally violates the task constraints to learn the bounds of the safe-set. In previous work demonstrations were generated using: discrete hand-designed task specific controllers ( Wilcox et al., 2021 ; Thananjeyan et al., 2019 ) , trained online stochastic controllers ( Gülçehre et al., 2020 ; Fu et al., 2020 ) : using reward shaping, or user demonstrations ( Hoque et al., 2021 ; Dasari et al., 2019 ) . Whilst effective, all these approaches require task specific information provided by the practitioner to generate useful demonstrations, limiting their ability to scale and generalize to new tasks and environments. An alternative, previously unexplored, approach to this safe-RL problem is to use unsupervised RL ( Laskin et al., 2021 ) to automatically generate a task specific dataset. A technique that does not rely upon the practitioner to design a task specific controller. For safe-RL this is a problem of exploration, gathering sufficient demonstrations to learn a complex representation online and classification, encoding a representation of goal-reaching and constraint violating behavior. In this work we focus on which elements of offline learning are critical for safe online learning and how to effectively make use of unsupervised RL for offline RL. We make the following contributions: • In section 4 we analyze the impact of data quantity on training using demonstrations from a discrete controller on a complex goal-oriented visual motor task. We then offer an improvement called optimistic forgetting which helps to mitigate the binary failure mode that can occur when learning from few offline demonstrations online. • In section 5 we analyze the impact of data quality on training making use of unsupervised learning to generate datasets automatically without the need for a hand-designed controller. This allows us to learn generalizable safe-RL policies, agnostic of task specific objectives. We found one of the largest challenges in this area is working out how to balance data quality over quantity, as it is easy to generate plenty of data with unsupervised learning, but providing a balance between sub-optimal demonstrations and constraint violating behavior is important for successful adaption online. 2 Related Work 2.1 Optimizing for Performance and Safety Rosolia & Borrelli (2016) first introduced Learning Model Predictive Control (LMPC) with the aim to iteratively improve controller performance from an initial feasible trajectory within a reference free setting. They found by iteratively learning a value function and safe-set from task roll-outs, theoretic guarantees can be made concerning stability, optimality and solution feasibility for non-linear or stochastic linear systems ( Rosolia et al., 2018 ; Rosolia & Borrelli, 2019 ) . Thananjeyan et al. (2019) extend LMPC to higher order visuomotor tasks and LS 3 Wilcox et al. (2021) improved upon this by optimizing a task specific cost function, rather than uniformly expanding across the state-space. We build upon this work, investigating tasks in higher order state-spaces, offering algorithmic improvements that prove critical when learning from few offline demonstrations. 2.2 Learning Safe Behaviors from Demonstrations Using demonstrations to learn safe-sets offline has proven to be an effective method to safely constraint exploration in online RL. In contrast to previous approaches where exploration is restricted by hard constraints Dalal et al. (2018) or a safety critic Bharadhwaj et al. (2020) , our approach restricts exploration with a safe-set. Previous LMPC based approaches include LS 3 ( Wilcox et al., 2021 ) and SAVED ( Thananjeyan et al., 2019 ) where they aim to predict the likelihood of constraint violation or leaving the safe-set. Similarly, Thananjeyan et al. (2020) Recovery RL queries actions over a task-policy distribution and is used with model-free or model-based RL methods. All of these previous approaches obtained demonstrations either from hand-designed controllers or user demonstrations. We aim to understand the impact of these demonstrations for online learning. We explore the importance of controlling implicit bias within the dataset used for offline learning, and how quantity and diversity of demonstrations effects task-specific performance ( Gulcehre et al., 2022 ) . 2.3 Unsupervised Offline Learning Levine (2021) and Riedmiller et al. (2021) argue decoupling exploration and exploitation into independent separate components should help to improve scalability of RL; recent work by Yarats et al. (2022) ; Lambert et al. (2022) in offline learning provide evidence of this. Similar arguments can be made for safe-RL and to some extent the offline to online learning setup already separates out exploration and exploitation. In safe-RL the problem is more complicated as we need to not only explore the environment to find suitable behaviors, but classify them under constraint-violating or goal-reaching categories. Laskin et al. (2021) provide a useful comparison of different unsupervised approaches using the DeepMind Control Suite ( Tassa et al., 2020 ) . They found there is still significant algorithmic work to do in order to reach state-of-the-art performance with unsupervised learning and that there is a large gap in performance when using state and image based observations. In the unsupervised numerical experiments presented in this paper, we focused on using state-based observations because the results are easier to interpret than image-based observations. We also found competence based approaches ( Eysenbach et al., 2018 ; Lee et al., 2019 ; Liu & Abbeel, 2021a ) useful due to their expressive latent skill vector encoding allowing for behavioral classification. It is important to note that these unsupervised RL methods are relatively weak ( Laskin et al., 2021 ) , and we expect that as advancements are made in this area, the procedures outlined in this work will also improve. 3 Problem Statement We consider episodic finite time-horizon goal-conditioned tasks, characterized by a Constrained Markov Decision Process (CMDP) and described with a tuple: \mathcal{M}:=(\mathcal{S},\mathcal{G},\mathcal{A},P,R,C,\mu,T) . • \mathcal{S} , \mathcal{G} and \mathcal{A} are the state, goal and action spaces respectively. We consider image states with pixel width W and height H and direct state representations where \mathcal{S}=\mathbb{R}^{n} . The state variable at time t is denoted s_{t}\in\mathcal{S} . • P(s_{t+1}|s_{t},a_{t}) maps states and actions to a probability distribution over subsequent states, P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R} . • R(s_{t+1},s_{t},a_{t}) is the CMDP reward function, R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R} . • C(s_{t}) is a constraint indicator function, C:\mathcal{S}\to\{0,1\} , 1 if violated. • \mu is the initial state distribution ( s_{0}\sim\mu ) and T , the time horizon. The agent’s objective is to reach a goal state \mathcal{G}\subset\mathcal{S} in the fewest number of steps whilst avoiding task specific constraints. To represent the difficulty of reward shaping in real-world finite horizon tasks, similar to Thananjeyan et al. (2019) and Wilcox et al. (2021) , we use a discrete reward: R(s,a,s^{\prime})=\begin{cases}0&\text{$s^{\prime}\in\mathcal{G}$}\\ -1,&\text{$s^{\prime}\notin\mathcal{G}$}.\end{cases} For all tasks, the goal state was defined as a ball with radius \epsilon centered on a single state value, s_{g}\in\mathcal{S} : \mathcal{G}:=\{s:\|s-s_{g}\|\leq\epsilon\}. Given a policy \pi:\mathcal{S}\rightarrow\mathcal{A} , we define total return over a CMDP \mathcal{M} as R^{\pi}=\mathbb{E}_{\pi,\mu,P}[\sum_{t=0}^{T}R(s_{t},a_{t})] . Similar to Wilcox et al. (2021) , we define P_{\mathcal{C}}^{\pi} as the probability of future constraint violation over the CMDP time-horizon under policy \pi from state s . The objective then becomes to maximize the expected return R^{\pi} , whilst maintaining a constraint violation probability less than \delta_{\mathcal{C}} , equation 1 : \displaystyle\pi^{*}=\argmax_{\pi\in\Pi} \displaystyle\mathbb{E}_{\pi,\mu,P}\Bigl[\sum_{t=0}^{T}R(s_{t},a_{t})\Bigr] (1) \displaystyle\textrm{s.t.} \displaystyle\mathbb{E}_{s_{0}\sim\mu}[P_{\mathcal{C}}^{\pi}(s_{0})]\leq\delta_{\mathcal{C}}. 4 Safe Learning from Few Demonstrations Obtaining high quality goal-reaching demonstrations can be challenging and expensive, requiring demonstrations from task/domain experts or for the practitioner to spend time hand-designing a controller. To understand the influence of data quantity on online safe-RL we conducted experiments using three navigation tasks with varying complexity, and a range of dataset sizes. The following navigation tasks were investigated and are explained in detail in appendix A : • SimplePointBot (SPB), the agent needs to navigate around a block, by controlling its velocity. • Bottleneck , the agent needs to navigate down a pipe between 2 chambers without hitting the walls, by controlling its velocity. • SimpleVelocityBot (SVB), the agent needs to navigate around a block, by controlling its acceleration. To learn an optimal policy online we use an LMPC approach similar to those outlined in LS 3 ( Wilcox et al., 2021 ) and SAVED ( Thananjeyan et al., 2019 ) . To learn from the RGB 64\times 64 pixel image observations used in this section, we employed a \beta -variational auto-encoder, f_{enc} and f_{dec} , as described in Higgins et al. (2017) . The encoder, f_{enc} transforms states from the dataset \mathcal{D} into a d -dimensional latent space \mathcal{Z} , whilst the decoder, f_{dec} , reconstructs the original images. We then used a probabilistic dynamics model, similar to PETS ( Chua et al., 2018 ) , to estimate the likelihood of the agent: reaching the goal, violating constraints, or generating high reward. These likelihoods are estimated by querying learned deep neural network based representations for the safe-set f_{\mathcal{S}} , value-function V_{\pi} , goal-indicator f_{\mathcal{G}} and constraint estimator f_{\mathcal{C}} . This training procedure is described in more detail in Appendix B . When the quantity of demonstrations used to train the LMPC approach outlined above is too low the agent is unable to learn a goal-reaching policy online. As depicted in figure 1 , when fewer than 125 demonstrations are used, SPB experiences a failure to generate any reward, and when fewer than 50 demonstrations are used, Bottleneck encounters the same issue. Additionally, SVB is unable to learn a reliable policy online using any number of demonstrations. The presence of this binary-failure mode online highlights the importance of using a sufficient quantity of demonstrations to train safe-RL agents. Figure 1: Episodic return from the 3 navigation environments using the LMPC procedure outlined in LS 3 . SPB requires a minimum of 125 demonstrations to learn a goal-reaching policy and SVB is unable to learn a goal-reaching policy at all. We discovered the reason for this binary-failure mode is that when too few demonstrations are provided, the agent frequently fails to reach the goal state because the learned policy is not robust enough to reach the goal state. This leads to the value function and safe-set shrinking in size during online training, resulting in the cost of exploration becoming excessively high and the agent rarely venturing beyond the initial starting state. As shown in figure 2 , after only 50 demonstrations, the safe-set is effectively null as the agent expects all transitions to be unsafe, causing it to remain at the initial starting position. Figure 2: Heatmap of the Safe-Set f_{\mathcal{S}}(s) for SPB. Left: initial offline training. Top right: after 50 updates with optimistic forgetting. Bottom right: after 50 updates without optimistic forgetting. To address this failure mode, we propose a modification to the LMPC procedure described above called optimistic forgetting . This approach involves dropping the last N_{\text{forget}} episodes from the replay buffer \mathcal{D} if the normalized return R is less than the minimum return R_{\text{min}} over the past N_{\text{forget}} episodes. Similar to accept-reject sampling ( Bishop, 2007 ) , this technique reinforces confidence in goal-reaching behavior by keeping the safe-set and goal-indicator open to exploration towards the goal, while also improving the accuracy of the learned dynamics, value function and constraint estimator, effectively introducing an optimism bias ( Sharot, 2011 ) . This approach assumes that the dataset used for offline learning includes some demonstrations of goal-reaching behavior. Algorithm 1 describes the safe-learning procedure used in this work based on LS 3 with optimistic forgetting shown in orange . Algorithm 1 Safe-Set LMPC w/ Optimistic Forgetting Input: offline dataset \mathcal{D} , number of episodes N_{\text{episodes}} , forgetting frequency N_{\text{forget}} , minimum return R_{\text{min}} if state s\in\mathcal{D} of type images then Train: VAE encoder f_{enc} and decoder f_{dec} using data from \mathcal{D} end if Train: dynamics f_{dyn} , safe-set classifier f_{\mathcal{S}} , value function V_{\pi} , goal indicator f_{G} and constraint estimator f_{\mathcal{C}} using data from \mathcal{D} Current Return R=0 for i_{\text{episode}}=0 to N_{\text{\text{episodes}}} do Initialize environment \mathcal{M} with s_{0}\sim\mu for t=0 to T_{\text{episode}} do Select and execute a_{t} in \mathcal{M} with PETS MPC Observe state s_{t} , reward r_{t} , constraint c_{t} \mathcal{D}:=\mathcal{D}\cup\{(s_{t},a_{t},s_{t+1},r_{t},c_{t})\} R\mathrel{+}=r_{t} end for Update: f_{dyn} , f_{\mathcal{S}} , V_{\pi} , f_{G} , f_{\mathcal{C}} if i_{\text{episode}}\mathbin{\%}N_{\text{forget}}\equiv 0\cup R\leq R_{\text{min}} then Drop \mathcal{D}_{i_{\text{episode}}-N_{\text{forget}}}\subset\mathcal{D} R=0 end if end for The improvement provided by optimistic forgetting is illustrated in Figure 3 , which shows the results of the same experiments depicted in Figure 1 , but with the inclusion of the optimistic forgetting technique. Notably SPB can now learn from as few as 25 demonstrations and SVB is able to learn a goal-reaching policy. Figure 2 demonstrates how optimistic forgetting allows the safe-set to grow during online training and helps to overcome the binary failure mode present in the original setup. In these experiments we set the N_{\text{forget}} term to 25 episodes and normalized R_{\text{min}} to 0.5 . Performance could likely be improved by experimenting with these parameters further. Figure 3: Episodic return from the 3 navigation environments, trained using the procedure outlined in algorithm 1 , using optimistic forgetting. For SPB we can learn an optimal policy from as few as 25 demonstrations and SVB is able to learn a goal-reaching policy with 125 demonstrations. These empirical results emphasize that the quantity of demonstrations used for learning is a key factor for safe exploration online. In the next section we expand upon this by considering the impact of data quality on online exploration, specifically the significance of data diversity. 5 Safe Learning without a Teacher Designing controllers capable of generating useful goal-reaching and constraint-violating behaviors becomes increasingly difficult and time-consuming as task complexity grows. Unsupervised learning is an attractive alternative to hand designed controllers. It enables the automatic generation of diverse datasets without the need for extensive engineering effort, providing the practitioner with large amounts of data. Unsupervised RL is usually framed as a 2 stage process of: 1. Pretraining a policy based on an intrinsic reward function to generate a diverse group of behaviors. 2. Fine-tuning the policy to maximize a task specific extrinsic reward function. Fundamentally unsupervised RL is a method of finding useful behaviors without relying on a traditional reward function, whilst also ensuring a balance between diversity and utility. Safe-RL adds a layer of complexity, requiring behaviors to also be classified as either constraint-violating or goal-reaching in order to generate samples to train the LMPC procedure outlined in algorithm 1 . Competence based unsupervised RL algorithms express the policy using a prior encoded vector representation, denoted as \pi(s|z) , where the skill vector is sampled from a distribution z\sim\mathcal{Z} . This prior-encoded skill vector offers a convenient representation for classifying behavior types and subsequently sampling from the policy. To investigate the effectiveness of unsupervised learning and the subsequent effects of balancing optimality and diversity, we considered three experimental setups. Each one uses the SimplePointBot environment with state-based observations: • Controller : all demonstrations come from the deterministic controllers outlined in Appendix A . • Semi-Supervised : constraint-violating demonstrations come from the deterministic controller and goal-reaching demonstrations from SAC, trained using the p^{*} objective in equation 4 . • Unsupervised : all demonstrations come from the unsupervised learning procedure described in algorithm 2 . Soft-Actor Critic (SAC) ( Haarnoja et al., 2018a ) a state-based off-policy RL algorithm is used to train the semi-supervised and unsupervised policies used in the last 2 experiments. SAC’s objective is to learn a policy that not only maximizes the expected discounted sum of rewards, but also maximizes the expected entropy of the policy over the policy state distribution \rho_{\pi}(s) : J(\pi)=\sum^{T}_{t=0}\mathbb{E}_{(\mathbf{s}_{t},\mathbf{a}_{t})\sim\rho_{\pi}(\mathbf{s}_{t})}[r(\mathbf{s}_{t},\mathbf{a}_{t})+\alpha\mathcal{H}(\pi(\cdot|\mathbf{s}_{t}))] (2) The balance between maximizing the expected discounted sum of rewards and maximizing the expected entropy of the policy is controlled by the \alpha parameter. In this work optimization is completed using the procedure outlined in Haarnoja et al. (2018a) , and a state-representation with a prior \mathbf{s}_{t}=(\mathbf{o}_{t};\mathbf{z}_{t}) and intrinsic reward r_{\mathbf{z}_{t}}(\mathbf{s}_{t}) . As detailed in Appendix C.2 we also considered using DrQ-v2 ( Yarats et al., 2021 ) , similar to URLB, but found it challenging to manage the explicit exploration schedule implemented in DrQ-v2. State-Marginal Matching (SMM) ( Lee et al., 2019 ) , a competence based unsupervised RL algorithm is used to pretrain the prior skill-encoded policy and generate goal-reaching and constraint-violating samples. Based on DIAYN ( Eysenbach et al., 2018 ) , SMM includes a p^{*}(s) target distribution term in the intrinsic reward function, which significantly improves the efficiency of learning goal-reaching behaviors. Additionally, SMM’s discrete one-hot vector encoding facilitates easy classification and sampling of demonstrations from each prior, a challenge faced by algorithms like APS ( Liu & Abbeel, 2021a ) . SMM combines the entropy based mutual information criteria found in DIAYN (Appendix C.1 ): \log q_{\phi}(z|s)-\log p(z) with a state-marginal matching objective: \min\limits_{\pi\in\Pi}D_{KL}(\rho_{\pi}(s)||p^{*}(s)) These objectives can be combined as an intrinsic reward (Appendix C.2 ) and used in algorithm 2 : r_{z}(s)=\log p^{*}(s)-\log\rho_{\pi_{z}}(s)+\log p(z|s)-\log p(z) (3) In order to make the data-collection process more efficient, we use a target distribution, p^{*}(s) , based on the distance from the goal state, g . This is defined as: p^{*}(s)=\sqrt{||s_{t}-g_{t}||_{2}^{2}} (4) In experimentation, we found it necessary to incorporate a penalty for violating the constraint, along with a bonus for attaining the goal-state, in order to prevent the agent from getting trapped on the constraint block. Algorithm 2 Unsupervised Safe Dataset Collection Input: environment \mathcal{M} , intrinsic reward r_{intrinsic} , discount factor \gamma , pretrain-steps N_{pt} , exploratory-samples N_{samples} , dataset-size N_{dataset} , prior-distribution z\sim\mathcal{Z} , minimum return R_{min} Randomly Initialize: actor \pi_{\theta} , critic Q_{\phi} for i_{step}=0 to N_{pt} do Sample skill z\sim p(z) , initial state s_{0}\sim\mu(s) for t=0 to T_{episode} do a_{t}\sim\pi_{\theta}(a_{t}|s_{t},z) s_{t+1}\sim P(s_{t+1}|s_{t},a_{t}) Calculate r_{z}(s_{t}) using equation 3 \mathcal{D}_{replay}:=\mathcal{D}_{replay}\cup\{s_{t},a_{t},r_{t},s_{t+1},z_{t}\} Update \pi_{\theta} , Q_{\phi_{1,2}} , using mini-batches from \mathcal{D}_{replay} with SAC end for end for for i_{z}=0 to N_{samples} do Sample skill z\sim p(z) , initial state s_{0}\sim\mu(s) Collect trajectory \tau_{i_{z}}=\{(s_{t},a_{t},s_{t+1},r_{t},c_{t})\} with \pi_{\theta}(.|.,z) from \mathcal{M} if \sum_{t=0}^{T}r_{t}\geq R_{min} then Store prior z\cup Z_{gr} end if end for for i_{dataset}=0 to N_{dataset} do Sample skill z\sim Z_{gr} , initial state s_{0}\sim\mu(s) Collect trajectory \tau_{i_{z}}=\{(s_{t},a_{t},s_{t+1},r_{t},c_{t})\} with \pi_{\theta}(.|.,z) from \mathcal{M} Store trajectory \mathcal{D}_{gr}:=\mathcal{D}_{gr}\cup\tau end for i_{data}=0 while i_{data}60 then a_{t}=\mathbf{a}_{max}|s_{g}-s_{t}| end if action a_{t} Algorithm 4 SPBController - Constraint Violating 0: constraint center s_{c} , random position s_{d} 0: time t , state s_{t} if t<15 then a_{t}=\mathbf{a}_{max}|s_{d}-s_{t}| t\mathrel{+}=1 end if if t>15 then a_{t}=\mathbf{a}_{max}|s_{c}-s_{t}| t\mathrel{+}=1 end if action a_{t} A.2 Simple Velocity Bot (SVB) The Simple Velocity Bot (SVB) task uses the acceleration of the agent to control the state, and velocity is unobserved, therefore it is a partially observable environment. A.2.1 Environment Definition • State : [x,y] • Action : [a_{x},a_{y}] • Goal : [x_{g},y_{g}] A.2.2 Hand-Controllers For the SVB controller we use a PID controller with gains scheduled based upon the agents position. With controller gains: Table 1: Controller Gains K_{p} K_{i} K_{d} s_{0} \to s_{2} 5.0 0.05 0.0 s_{2} \to s_{g} 5.0 0.5 4.0 The controller acts upon the velocity error e_{t}=v_{t}-v_{dem} , these are the same for x and y velocity components. \ddot{x}_{t}=K_{p}e_{t}+K_{i}\int_{0}^{t}e_{\tau}d\tau+K_{d}\dot{e}_{t} (6) To navigate the agent around the environment to the goal state we then use a set of states \mathbf{G}=(\mathbf{s}_{0},\mathbf{s}_{1},\mathbf{s}_{2},\mathbf{s}_{g}) and define two control modes whilst en-route to the final goal state. When the goal state is the next ensuing point the agent then just moves to the goal state. We define 3 points as part of this trajectory, based on the goal index i_{g} . We then work out a track angle to go from a\to b and b\to c : 1. Calculate track points: \mathbf{s}_{a}=\mathbf{G}[i_{g}] , \mathbf{s}_{b}=\mathbf{G}[i_{g}+1] and \mathbf{s}_{c}=\mathbf{G}[i_{g}+2] 2. Calculate track angle in and out: \theta_{in}=\arccos(\frac{s_{a}}{|s_{a}|}\cdot\frac{\mathbf{s}_{b}}{|\mathbf{s}_{b}|}) , \theta_{out}=\arccos(\frac{s_{b}}{|\mathbf{s}_{b}|}\cdot\frac{\mathbf{s}_{c}}{|\mathbf{s}_{c}|}) 3. Calculate turn angle \phi=\theta_{out}-\theta_{in} Figure 11: Navigation problem setup Figure 11 labels the key points and desired track from a\to b\to c . We define 2 navigation modes, track-to-point and fly-by-point . In track-to-point the agent tracks a straight line to the target position, whilst in fly-by-point the agent aims to maintain a constant turn radius around 2 points bounded by planes based on the turn radius r . When the agent passes through a plane it changes mode. Figure 12: Track-to-Point key values and dimensions Figure 12 illustrates the key values and dimensions used by the track-to-point controller. If in track-to-point mode, used from \mathbf{s}_{a}\to\mathcal{H}_{ab} : 1. Define \mathbf{\hat{q}}=\frac{\mathbf{s}_{b}-\mathbf{s}_{a}}{|\mathbf{s}_{b}-\mathbf{s}_{a}|} as the unit vector perpendicular to the plane \mathcal{H}_{ab} . 2. Define the point where the vector \mathbf{s}_{b}-\mathbf{s}_{a} intersects the plane \mathcal{H}_{ab} as: \mathbf{z}=\mathbf{s}_{b}-\frac{r}{tan(\frac{\phi}{2})}\mathbf{\hat{q}} . 3. Calculate vector from agent to plane center: \mathbf{h}=\mathbf{z}-\mathbf{s}_{t} . 4. Calculate which side of the plane the agent lies on: h=\mathbf{h}\cdot\mathbf{\hat{q}} . 5. If h>0 , change to fly-by-point mode on next iteration. To then calculate the heading allowing us to track the line from \mathbf{s}_{a}\to\mathbf{z} : 1. Define angle from agent to \mathbf{s}_{b} as \beta=\arccos(\frac{\mathbf{s}_{t}}{|\mathbf{s}_{t}|}\cdot\frac{\mathbf{s}_{b}}{\mathbf{s}_{b}}) 2. Define distance to the objective as: d_{obj}=||\mathbf{s}_{t}-\mathbf{s}_{b}|| 3. Define track distance as: d_{track}=||\mathbf{s}_{b}-\mathbf{s}_{a}|| 4. Calculate off track error as: \phi=\beta-\theta_{in} 5. Calculate demanded heading as: \theta_{hdg}=\frac{d_{track}\phi}{2d_{obj}}+\theta_{in} Figure 13: Fly-by-Point key values and dimensions Figure 13 illustrates the key values and dimensions used by the fly-by-point controller. If in fly-by-point mode, used from \mathcal{H}_{ab}\to\mathcal{H}_{bc} : 1. Define \mathbf{\hat{q}}_{0}=\frac{\mathbf{s}_{b}-\mathbf{s}_{a}}{|\mathbf{s}_{b}-\mathbf{s}_{a}|} and \mathbf{\hat{q}}_{1}=\frac{\mathbf{s}_{c}-\mathbf{s}_{b}}{|\mathbf{s}_{c}-\mathbf{s}_{b}|} as unit vectors perpendicular to \mathcal{H}_{ab} and \mathcal{H}_{bc} respectively. 2. Define \mathbf{\hat{q}}_{\nabla}=\frac{\mathbf{\hat{q}}_{1}-\mathbf{\hat{q}}_{0}}{|\mathbf{\hat{q}}_{1}-\mathbf{\hat{q}}_{0}|} as the unit vector between \mathcal{H}_{ab}\to\mathcal{H}_{bc} . 3. Calculate center point of turn fillet as: \mathbf{s}_{center}=\mathbf{s}_{b}-\frac{r}{\sin(\frac{\psi}{2})}\mathbf{\hat{q}}_{\nabla} . 4. Define point where the vector \mathbf{s}_{c}-\mathbf{s}_{a} intersects the plane \mathcal{H}_{bc} as: \mathbf{z}=\mathbf{s}_{b}+\frac{r}{\tan(\frac{\psi}{2})}\mathbf{\hat{q}}_{1} . 5. Calculate vector from agent to plane center: \mathbf{h}=\mathbf{z}-\mathbf{s}_{t} . 6. Calculate which side of the plane the agent lies on: h=\mathbf{h}\cdot\mathbf{\hat{q}}_{1} . 7. If h>0 , change to track-to-point mode on next iteration. To then calculate the heading, allowing us to track along the arc from \mathcal{H}_{ab}\to\mathbf{z} with radius r : 1. Define d_{center}=||\mathbf{s}_{t}-\mathbf{s}_{center}|| as the distance of the agent from the center of the fillet. 2. Define circle angle as: \gamma=\arctan(\mathbf{s}_{t}-\mathbf{s}_{center}) . 3. Calculate circle error as: e_{t}=\frac{\mathbf{s}_{center}-r}{r} . 4. Declare turning orbit gain: k_{orbit}=4.0 . 5. Calculate heading as: \theta_{heading}=\gamma+\frac{\pi}{2}-\arctan(k_{orbit}e_{t}) . If in the final state from s_{2}\to s_{g} we simply drive the agent to the goal state and stop when close enough: 1. Define heading and distance as: d_{goal}=||\mathbf{s}_{t}-\mathbf{s}_{g}|| , \theta_{heading}=\arctan(\mathbf{s}_{t}-\mathbf{s}_{g}) . 2. If d_{goal}>20.0 then demanded speed is |u|_{dem}=0.075d_{goal}^{2} . 3. Else If d_{goal}<10.0 then demanded speed is |u|_{dem}=0.0 . 4. Else demanded speed is |u|_{dem}=3.0 . To generate an action to feed to the agent we calculate the relative demanded velocity components based on \theta_{heading} and use the PID gains from table 1 to get values for [\ddot{x},\ddot{y}] . To gather constraint-violating behavior samples we use the same approach as defined in algorithm 4 and use a PID controller to control the agent (using gains from the first row of table 1 ). Appendix B Safe RL Implementation details for each component of the safe RL algorithm are described in the following section. B.0.1 Variational AutoEncoder (VAE) For image based observations inputs are scaled to (64,64,3) input images before being fed into the \beta -VAE, using a Convolutional Neural Network (CNN) for f_{enc} and a transpose CNN for f_{dec} . Training is with mean-squared error (MSE) loss and Kullback-Leibler (KL) divergence, for s_{t}\in\mathcal{S} the loss is: J(\theta)=\|f_{dec}(z_{t})-s_{t}\|_{2}^{2}+\beta D_{KL}(f_{enc}(z_{t}|s_{t})\|\mathcal{N}(0,1)) (7) B.0.2 Probabilistic Dynamics Based on PETS ( Chua et al., 2018 ) , we trained a probabilistic ensemble of neural networks to learn the agent’s dynamics. Trained with a max log-likelihood objective, each network has 2 hidden layers and 128 hidden units. The loss for states s_{t},s_{t+1}\in\mathcal{S} with action a_{t}\in\mathcal{A} for dynamics model f_{dyn,\theta} is: J(\theta)=-\log(f_{dyn,\theta}(s_{t+1}|s_{t},a_{t})) (8) We use the TS-1 method for planning with f_{dyn} . B.0.3 Value Functions We trained an ensemble of value functions using fully connected neural networks (consisting of 3 hidden layers and 256 hidden units) recursively to predict long-term reward. During offline training the value function is trained to predict the cost-to-go on all trajectories in \mathcal{D} . The training loss with the \theta parameterized value function V is given as: J(\theta)=\biggl(V_{\theta}^{\pi}(s_{t})-\sum_{i=1}^{T-t}\gamma^{i}r_{t+i}\biggr)^{2} (9) In online training the target network V^{\pi^{{}^{\prime}}} is stored and the temporal difference (TD-1) error calculated: J(\theta)=(V_{\theta}^{\pi}(s_{t})-(r_{t}+\gamma V_{\theta^{{}^{\prime}}}^{\pi^{{}^{\prime}}}(s_{t+1}))) (10) The value function is updated only using data from suboptimal demonstrations, or online data. In equation 10 \theta^{{}^{\prime}} is the parameters of a lagged target network and \pi^{{}^{\prime}} is the policy at the time-step \theta^{{}^{\prime}} was set, \gamma=0.99 is the discount factor. B.0.4 Constraint and Goal Estimators The constraint indicator f_{\mathcal{C}}:\mathcal{Z}\to\{0,1\} is represented by a neural network with 3 hidden layers and 256 hidden units. Each layer is trained with a binary cross entropy loss using demonstrations from \mathcal{D}_{\text{constraint}} as unsafe examples and \mathcal{D}_{\text{constraint}}^{{}^{\prime}} as safe examples. The goal estimator f_{\mathcal{G}}:\mathcal{Z}\to\{0,1\} similarly is represented by a neural network consisting of 3 hidden layers and 256 hidden units. Goal-reaching demonstrations are used to train the classifier from \mathcal{D}_{gr} . Both estimators use a binary cross entropy loss function for training. B.0.5 Safe Set The safe set classifier f_{\mathbb{S}}(s_{t}) is represented by a neural network with 3 hidden layers and 256 hidden units, trained to predict: f_{\mathbb{S}}(s_{t})=\max(\mathds{1}_{\mathbb{S}}(s_{t}),\gamma_{\mathbb{S}}f_{\mathbb{S}}(s_{t+1})) Where \mathds{1}(s_{t}) is an indicator function, indicating whether s_{t} is part of a successful trajectory. Training data is sampled uniformly from the entire dataset, \mathcal{D} . B.0.6 Model Based Planning The Cross Entropy Method (CEM) is used to solve the following optimization problem in order to select actions a_{t:t+H-1} : \displaystyle\argmax_{a_{t:t+H-1}} \displaystyle\mathbb{E}_{s_{t:t+H}}\Biggl[\sum_{i=1}^{H-1}f_{\mathcal{G}}(s_{t+i})+V^{\pi}(s_{t+H})\Biggr] (11) \displaystyle\text{s.t.} \displaystyle s_{t+1}\sim f_{dyn}(s_{t+1}|s_{t},a_{t})\forall\in\{t,\ldots,t+H-1\} \displaystyle\hat{\mathbb{P}}\bigl(s_{t+H}\in\mathbb{S}_{\mathcal{S}}\bigr)\geq 1-\delta_{\mathbb{S}} \displaystyle\hat{\mathbb{P}}\bigl(s_{t+i}\in\mathbb{S}_{\mathcal{C}}\bigr)\leq\delta_{\mathcal{C}}\forall i\in\{0,\ldots,H-1\} Details of the application of the CEM method for model-based planning are provided in Chua et al. (2018) and Wilcox et al. (2021) . For image based observations equation 11 is re-parameterized based on the VAE latent variable \mathcal{Z} , such that f_{enc}(s_{t})\to z_{t} . Parameters used for the safe-RL procedure are provided in table 2 . Table 2: Safe RL (LS 3 ) parameters SafeRL Parameter Value SS Violation ( \delta_{\mathbb{S}} ) 0.8 Constraint Violation ( \delta_{\mathcal{C}} ) 0.2 VAE Beta ( \beta ) 10^{-6} Planning Horizon ( H ) 5 CEM ( n_{\text{particle}} ) 20 CEM ( n_{\text{candidate}} ) 1000 CEM ( n_{\text{elite}} ) 100 CEM iterations ( N_{\text{cem}} ) 5 CEM ( d ) 32 CEM ( p_{\text{random}} ) 1.0 Frame Stacking No Batch Size 0.01 Discount Factor ( \gamma ) 0.99 SS discount ( \gamma_{\mathbb{S}} ) 0.3 Appendix C Unsupervised RL To generate a group of constraint violating and goal-reaching skills we made use of competence based unsupervised RL techniques. To learn this skill embeding we used Soft Actor Critic (SAC) ( Haarnoja et al., 2018b ) , an off-policy actor-critic deep RL algorithm. SAC augments the standard RL expected discounted sum of rewards objective \sum_{t}\mathbb{E}_{(\mathbf{s}_{t},\mathbf{a}_{t})\sim\rho_{\pi}}[r(\mathbf{s}_{t},\mathbf{a}_{t})] with the expected entropy over the policy \rho_{\pi}(s_{t}) . This makes the policy objective: J(\pi)=\sum^{T}_{t=0}\mathbb{E}_{(\mathbf{s}_{t},\mathbf{a}_{t})\sim\rho_{\pi}}[r(\mathbf{s}_{t},\mathbf{a}_{t})+\alpha\mathcal{H}(\pi(\cdot|\mathbf{s}_{t}))] (12) Where \alpha represents the importance of the entropy term against the CMDP reward term. All the unsupervised learning algorithms used in this work include an entropy maximization term \mathcal{H}(\pi(\cdot|\mathbf{s}_{t})) . In order to find the optimal policy \pi^{*} defined by the objective function in Equation 12 SAC uses a parameterized: state value function V_{\psi}(\mathbf{s}_{t}) , soft Q-function Q_{\theta}(\mathbf{s}_{t},\mathbf{a}_{t}) and tractable policy \pi_{\phi}(\mathbf{a}_{t}|\mathbf{s}_{t}) . The soft value function is trained to minimize the residual square error: J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t}\sim\mathcal{D}}\Biggl[\frac{1}{2}\Bigl(V_{\psi}(\mathbf{s}_{t})-\mathbb{E}_{\mathbf{a}_{t}\sim\pi_{\phi}}[Q_{\theta}(\mathbf{s}_{t},\mathbf{a}_{t})-\log\pi_{\phi}(\mathbf{a}_{t}|\mathbf{s}_{t})]^{2}\Bigr)\Biggr] The soft Q-function is trained to minimize the soft Bellman error: J_{Q}(\theta)=\mathbb{E}_{(\mathbf{s}_{t},\mathbf{a}_{t})\sim\mathcal{D}}\biggl[\frac{1}{2}\bigl(Q_{\theta}(\mathbf{s}_{t},\mathbf{a}_{t})-\hat{Q}(\mathbf{s}_{t},\mathbf{a}_{t})\bigr)^{2}\biggr] where, \hat{Q}(\mathbf{s}_{t},\mathbf{a}_{t})=r(\mathbf{s}_{t},\mathbf{a}_{t})+\gamma\mathbb{E}_{\mathbf{s}_{t+1}\sim\mathcal{D}}\bigl[V_{\overline{\psi}}(\mathbf{s}_{t+1})\bigr] Bringing these together and using the reparameterization trick to represent actions with samples from a spherical Gaussian \mathbf{a}_{t}=f_{\phi}(\epsilon_{t};\mathbf{s}_{t}) the policy objective becomes: J_{\pi}(\phi)=\mathbb{E}_{\mathbf{s}_{t}\sim\mathcal{D},\epsilon_{t}\sim\mathcal{N}}[\log\pi_{\phi}(f_{\phi}(\epsilon_{t};\mathbf{s}_{t})|\mathbf{s}_{t})-Q_{\theta}(\mathbf{s}_{t},f_{\phi}(\epsilon_{t};\mathbf{s}_{t}))] SAC then uses two Q-functions to reduce positive bias during policy improvement and alternates between adding experience into the replay buffer, with the policy \pi_{\phi}(\mathbf{a}_{t}|\mathbf{s}_{t}) and updating the policy via gradient descent. The hyperparameters used for the SAC implementation in this work are outlined below in Table 3 . Compared to SAC, DrQ-V2 is more robust to reward scaling as exploration is not directly tied to the objective function but is instead controlled by applying Gaussian noise to the objective function. We found selecting an appropriate noise schedule difficult for the navigation tasks in this work because the agent needs many exploratory samples to understand how to reach the goal but relatively fine control to remain at the goal once it reached it. The exploratory noise values used in DrQ-v2 made it difficult for the agent to stay at the goal after it had reached it, but any lower noise values caused the agent to run into the block too often. Table 3: Soft Actor Critic (SAC) Parameters SAC Parameter Value Replay Buffer Capacity 10^{6} Batch Size 256 Discount ( \gamma ) 0.99 Learning Rate ( \lambda ) 10^{-3} Activation Function ReLU Optimizer Adam Hidden Layers 5 Hidden Dims 64 Polyack Average ( \tau ) 0.005 Entropy Regularization ( \alpha ) 0.1 Target Step Interval 1 Step Update Frequency 1 Figure 14: Comparison of SAC return and Q-loss using L_{2} goal norm from equation 4 on SPB and SVB. SPB is deceptively difficult as the agent needs to explore outside the box to reach the maximum reward which moves it further from the goal resulting in temporarily lower return. Runs are taken over 5 random seeds and error bounds at 95% confidence interval shown. Figure 14 compares the training performance on SPB and SVB, we found that even though we can learn an effective goal-reaching policy for SVB with SAC. We could not learn useful goal reaching behavior with SMM in a reasonable timeframe. We believe this is due to the larger action space available to SVB compared to SPB, which helps SVB to learn a single optimal policy with SAC faster, but causes too much diversity when using SMM, such that the agent never learns useful goal-reaching behavior. C.1 DIAYN Diversity Is All You Need (DIAYN) from Eysenbach et al. (2018) aims to construct a group of skills that are maximally diverse but also discriminable. Using Mutual Information (MI), I(\cdot;\cdot) with skills Z , actions A and states s the objective function contains terms to: • Minimize MI, between skills and actions in a given state: -I(A;Z|S) • Maximize the entropy of the policy: \mathcal{H[A|S]} • Maximize MI between skills and states, so skills control which states are visited: I(S;Z) Putting this together as an entropy based objective: \displaystyle\mathcal{F}(\theta) \displaystyle=I(S;Z)+\mathcal{H}[A|S]-I(A;Z|S) \displaystyle=\mathcal{H}[Z]-\mathcal{H}[Z|S]+\mathcal{H}[A|S,Z] The entropy over the policy ( \mathcal{H}[A|S,Z] ) is included as part of the SAC exploration reward. For the remaining terms we can rewrite the objective as an expectation, replacing p(z|\mathbf{s}_{t}) with a discriminator q_{\phi}(z|\mathbf{s}_{t}) , bounded using Jensen’s inequality as a variational lower bound \mathcal{G}(\phi,\theta) on the objective \mathcal{F}(\theta) . \displaystyle\mathcal{F}(\theta) \displaystyle=\mathcal{H}[A|S,Z]+\mathcal{H}[Z]-\mathcal{H}[Z|S] \displaystyle=\mathbb{E}_{z\sim p(z),s\sim\pi(z)}[\mathcal{H}(\pi(\cdot|s))]+\mathbb{E}_{z\sim p(z),s\sim\pi(z)}[\log p(z|s)]-\mathbb{E}_{z\sim p(z)}[\log p(z)] \displaystyle\geq\mathbb{E}_{z\sim p(z),s\sim\pi(z)}[\alpha\mathcal{H}(\pi(\cdot|s))+\log q_{\phi}(z|s)-\log p(z)]=\mathcal{G}(\theta,\phi) This can be bought into the SAC policy objective, in equation 2 , as an intrinsic reward: r_{z}(\mathbf{s}_{t},\mathbf{a}_{t})=\log q_{\phi}(\mathbf{z}_{t}|\mathbf{s}_{t})-\log p(\mathbf{z}_{t}) In this work we use a VAE to learn the skill discriminator q_{\phi}(z|s) and a fixed one-hot discrete skill vector representation for Z\sim p(z) . z is sampled from Z at the beginning of each episode and is then fixed until the episode terminates. Figure 15 shows the training plots from DIAYN and how it begins to overfit after too many samples are collected, the effect of this overfitting is shown in figure 16 . Note how none of the priors reach the goal-state in the good fit making DIAYN unsuitable for this goal-navigation problem as there are too many diverse states that are not goal-states or useful behaviors. Figure 15: DIAYN training over multiple skill vectors, yellow and black arrow denotes beginning of overfitting. Figure 16: Sampling from DIAYN policy where each colour represents a skill vector z\in Z , where Z is a discrete prior represented as a 64 dimension one-hot vector. C.2 SMM State Marginal Matching (SMM) by Lee et al. (2019) aims to find a policy \pi\in\Pi that matches the target state density p^{*}(s) to the policy state distribution \rho_{\pi}(s) . This objective can be written as: \displaystyle\min_{\pi\in\Pi}D_{\text{KL}}(\rho_{\pi}(\mathbf{s}_{t})||p^{*}(\mathbf{s}_{t})) \displaystyle= \displaystyle\max_{\pi\in\Pi}\mathbb{E}_{\rho_{\pi(\mathbf{s}_{t})}}[\log p^{*}(\mathbf{s}_{t})+\mathcal{H}(\pi(\cdot|\mathbf{s}_{t}))] The \rho_{\pi}(s) term can be decomposed into a set of policies conditioned on a latent variable z\in\mathcal{Z} , the state marginal of this mixture of policies with prior p(z) is then given as: \rho_{\pi}(s)=\int_{\mathcal{Z}}\rho_{\pi_{z}}(s)p(z)dz=\mathbb{E}_{z\sim p(z)}[\rho_{\pi_{z}}(s)] Using Bayes rule and the KL minimization objective the optimization problem becomes: \max_{\pi_{z},z\in\mathcal{Z}}\mathbb{E}_{p(z),\rho_{\pi_{z}}(s)}[r_{z}(s)] with r_{z}(s) : r_{z}(s)=\log p^{*}(s)-\log\rho_{\pi_{z}(s)}+\log p(z|s)-\log p(z) The \log p(z|s)-\log p(z) objectives follow from the earlier DIAYN objective formulation, where we aim to identify distinguishable behaviors whilst exploring the state-space. The \log p^{*}(s) is simply the target distribution and \log\rho_{\pi_{z}(s)} term aims to explore unvisited states. Table 4: State Marginal Matching (SMM) Parameters SMM Parameter Value Skill Distribution Discrete Skill Number 4 Learning Rate Discriminator ( \lambda_{\text{sr}} ) 10^{-3} Learning Rate VAE ( \lambda_{\text{vae}} ) 10^{-2} VAE Beta (\beta) 0.5 State Entropy Coefficient ( {c}_{\mathcal{H}(S|Z)} ) 0.1 Latent Entropy Coefficient ( c_{\mathcal{H}(Z)} ) 0.1 Latent Conditional Entropy Coefficient ( c_{\mathcal{H}(Z|S)} ) 0.1 Table 4 describes the parameters used for training SMM. We found it necessary to reduce the size of each of the entropy related terms to ensure progress towards the p^{*} objective. C.3 APS & CIC We considered the use of both CIC ( Laskin et al., 2022 ) and APS ( Liu & Abbeel, 2021a ) for the experiments in appendix A . Whilst both offer greater sample efficiency than DIAYN they do not allow for the p^{*} term used by SMM. We found this term necessary to achieve usable datasets within the bounds of the compute we have available. APS is similar to DIAYN but uses successor features to estimate the MI decomposition: • \mathcal{H}[Z] is estimated using the particle estimator described in APT ( Singh et al., 2003 ; Liu & Abbeel, 2021b ) • \mathcal{H}[Z|S] is estimated with successor features, described in VISR ( Hansen et al., 2019 ) Similarly, CIC uses the MI decomposition with Contrastive Predictive Control (CPC) ( van den Oord et al., 2018 ) to learn a prior encoded policy. In future work it would be interesting to investigate if the improved discriminator presented in CIC can be used with the same state marginal matching objective found in SMM to create a more effective discriminator in more complex state spaces.