Risk-Sensitive Exponential Actor Critic Alonso Granados, Jason Pacheco Abstract Model-free deep reinforcement learning (RL) algorithms have achieved tremendous success on a range of challenging tasks. However, safety concerns remain when these methods are deployed on real-world applications, necessitating risk-aware agents. A common utility for learning such risk-aware agents is the entropic risk measure, but current policy gradient methods optimizing this measure must perform high-variance and numerically unstable updates. As a result, existing risk-sensitive model-free approaches are limited to simple tasks and tabular settings. In this paper, we provide a comprehensive theoretical justification for policy gradient methods on the entropic risk measure, including on- and off-policy gradient theorems for the stochastic and deterministic policy settings. Motivated by theory, we propose risk-sensitive exponential actor-critic (rsEAC), an off-policy model-free approach that incorporates novel procedures to avoid the explicit representation of exponential value functions and their gradients, and optimizes its policy w.r.t. the entropic risk measure. We show that rsEAC produces more numerically stable updates compared to existing approaches and reliably learns risk-sensitive policies in challenging risky variants of continuous tasks in MuJoCo. 1 Introduction Model-free deep reinforcement learning (RL) algorithms have been successful at learning complex behaviors using only interactions with the environment ( 37 ; 23 ; 22 ) . However, interacting with real-world applications, such as autonomous driving ( 35 ) , robotics ( 39 ) and finance ( 2 ; 30 ) , can lead to catastrophic events, motivating a need for risk-aware agents in decision-making problems. Risk-sensitive RL aims to learn policies that maximize performance metrics that incorporate a penalty for the variability of the return — either due to intrinsic uncertainty in the transition dynamics or randomness in the reward signal. Risk-sensitive RL has been studied through various risk criteria: variance-related risk measures ( 49 ; 29 ) , reward-volatility risk ( 5 ; 53 ) , Gini-deviation ( 32 ) , distortion risk ( 12 ) , and conditional value at risk (CVaR) ( 38 ; 10 ) . In this paper, we consider risk-sensitive RL with the entropic risk measure. This metric incorporates risk into its objective via the exponential utility function ( 25 ) , resulting in a nonlinear Bellman equation. In contrast to the standard RL problem, sample-based estimates of the expectation operation are biased. Therefore, most of the work under this framework has been in the context of MDP control, where the transition dynamics and reward model are known ( 19 ) . The exponential Bellman equations provide a model-free approach that leads to improved regret bounds for risk-sensitive RL ( 16 ) . In the approximate setting, we can minimize the exponentiated TD error, which accelerates the learning process and produces robust policies to model perturbations ( 44 ) . However, the estimation of these functions is numerically unstable. Policy gradient algorithms improve their policy by updating it in the direction of the performance gradient. The fundamental result for these methods is the policy gradient theorem ( 48 ) , which allows for the estimation of the gradient using only sampled trajectories. In risk-sensitive RL, an estimate of the gradient can be derived using the likelihood ratio trick ( 39 ) . However, the estimator may suffer from high variance as its computation is w.r.t. the full trajectory, and it is scaled by the exponentiated return, which can be numerically unstable. These numerical instabilities are further exacerbated when introducing function approximators ( 33 ) . We now present our contributions to address these shortcomings. First, we derive risk-sensitive policy gradient theorems for the on-policy setting and approximations for the off-policy setting. These results circumvent the need to estimate a gradient for the full trajectory. Second, we study the instabilities that emerge from learning a critic network that approximates the exponentiated return, and propose a representation that avoids its explicit computation by allowing for computation in log-domain. Additionally, we consider a stabilizing mechanism for the critic gradient, based on mini-batch normalization and clipping. Finally, we present risk-sensitive exponential actor-critic (rsEAC), a practical off-policy algorithm that optimizes the entropic risk measure where the actor is optimized using our risk-sensitive off-policy gradient, and the critic incorporates our stabilizing mechanisms to estimate the entropic risk measure for the current policy. We evaluate our algorithm on complex continuous tasks where risk-aversion can be verified, and demonstrate that our method can learn risk-sensitive policies with high-return. All code is included in the supplemental materials. 2 Background Consider a finite horizon Markov decision process (MDP) which is composed of: a state space \mathcal{S} , an action space \mathcal{A} , an initial state distribution p_{1}(s_{1}) , a transition dynamics distribution p(s_{t+1}|s_{t},a_{t}) that satisfies the Markov property p(s_{t+1}|s_{t},a_{t},\dots,s_{1},a_{1})=p(s_{t+1}|s_{t},a_{t}) , and a reward function r:\mathcal{S}\times\mathcal{A}\to\mathbb{R} , which we abbreviate as r_{t}:=r(s_{t},a_{t}) . We assume that the agent does not have access to the transition dynamics and the reward function. The agent selects actions based on a policy denoted by \pi_{\theta}(a_{t}|s_{t}) , a conditional distribution over \mathcal{A} parameterized by \theta\in\mathbb{R}^{n} . The distribution over a trajectory \tau=(s_{1},a_{1},\dots,s_{T},a_{t},s_{T+1}) for a policy \pi_{\theta} is given by p_{\pi}(\tau)=p(s_{1})\prod_{t=1}^{T}p(s_{t+1}\mid s_{t},a_{t})\pi_{\theta}(a_{t}\mid s_{t}) . Value functions are defined to be the expected cumulative rewards, V_{\pi_{\theta}}(s)=\mathbb{E}_{p_{\pi}(\tau)}[\sum_{t=1}^{T}r_{t}|s_{1}=s] and Q_{\pi_{\theta}}(s,a)=\mathbb{E}_{p_{\pi}(\tau)}[\sum_{t=1}^{T}r_{t}|s_{1}=s,a_{1}=a] . We denote the distribution of state s^{\prime} after transitioning for t time steps from state s by p(s\to s^{\prime},t,\pi) . We also denote the state distribution by \rho_{\pi}(s^{\prime}):=\int_{S}\sum_{t=1}^{T}p_{1}(s)p(s\to s^{\prime},t,\pi)\,\mathrm{d}s . The standard objective in RL is to find a policy that maximizes expected return J(\pi_{\theta})=\mathbb{E}_{p_{\pi}(\tau)}[\sum_{t=1}^{T}r_{t}] . 2.1 Policy Gradient Algorithms Policy gradient algorithms update the parameters of their policy in the direction of the performance gradient \nabla_{\theta}J(\pi_{\theta}) . For stochastic policies \pi_{\theta} , the gradient can be derived from the policy gradient theorem ( 48 ) , \nabla_{\theta}J(\pi_{\theta})=\int_{S}\rho_{\pi}(s)\int_{A}\nabla_{\theta}\pi_{\theta}(a|s)Q_{\pi_{\theta}}(s,a)\,\mathrm{d}a\,\mathrm{d}s. (1) Although the state distribution \rho_{\pi}(s) depends on the policy parameters, the gradient surprisingly does not involve the derivative of the state distribution. The work by 47 extended this result to deterministic policies \mu_{\theta} , yielding an analogous gradient that can be estimated more efficiently than the stochastic policy gradient as it lacks the integral over actions: \nabla_{\theta}J(\mu_{\theta})=\!\!\int_{S}\!\!\rho_{\mu}(s)\nabla_{\theta}\mu_{\theta}(s)\nabla_{a}Q_{\mu_{\theta}}(s,a)|_{a=\mu_{\theta}(s)}\,\mathrm{d}s. (2) These two results can be extended to off-policy trajectories sampled by a different behavior policy b(a\mid s)\neq\pi_{\theta}(a\mid s) . The performance objective is modified to be the value function of the target policy V_{\pi_{\theta}} , weighted by the state distribution of the behavior policy, J_{b}(\pi_{\theta})=\int_{S}\rho_{b}(s)V_{\pi_{\theta}}(s)\,\mathrm{d}s . The gradient of this objective can be approximated by an off-policy variant of the policy gradient theorem ( 13 ) : \nabla_{\theta}J_{b}(\pi_{\theta})\approx\int_{\mathcal{S}}\int_{\mathcal{A}}\rho_{b}(s)\nabla_{\theta}\pi_{\theta}(a|s)Q_{\pi}(s,a)\,\mathrm{d}a\,\mathrm{d}s. (3) This approximation drops a term that depends on \nabla_{\theta}Q_{\pi_{\theta}}(s,a) , because it can be difficult to estimate in an off-policy fashion. However, this approximation still produces policy improvement when using a tabular representation for the policy ( 13 ) . 2.2 Risk-Sensitive Objective In risk-sensitive RL with the entropic risk measure, we aim to find a policy that maximizes: J^{\beta}(\pi_{\theta})=\frac{1}{\beta}\log\mathbb{E}_{p_{\pi}(\tau)}\left[e^{\beta\sum_{t=1}^{T}r_{t}}\right], (4) where \beta\in\mathbb{R} controls risk preference. This objective is closely related to mean-variance RL ( 34 ) — it admits the Taylor expansion \mathbb{E}_{p_{\pi}(\tau)}[\sum_{t}r_{t}]+\frac{\beta}{2}\text{Var}_{\pi}(\sum_{t}r_{t})+O(\beta^{2}) ( 36 ) . Thus, Eq. ( 4 ) reduces to the standard ( risk-neutral ) RL objective for \beta\to 0 . In addition, \beta>0 induces risk-seeking policies and \beta<0 induces risk-averse policies. We now define soft-value functions as the entropic risk measure of cumulative rewards, \displaystyle\!\!V_{\pi_{\theta}}^{\beta}(s)=\!\frac{1}{\beta}\!\log\mathop{\mathbb{E}}_{p_{\pi}(\tau)}\left[e^{\beta\sum_{t=1}^{T}r_{t}}|s_{1}\!=s\right]\!,\!\quad\!\!\!\! (5) \displaystyle Q_{\pi_{\theta}}^{\beta}(s,a)\!=\!\frac{1}{\beta}\log\mathop{\mathbb{E}}_{p_{\pi}(\tau)}\left[e^{\beta\sum_{t=1}^{T}r_{t}}\mid\!s_{1}\!=\!s,\!a_{1}\!=\!a\right]. (6) These value functions are recursively associated via Bellman-style backup equations: \displaystyle V_{\pi_{\theta}}^{\beta}(s_{t})\!=\!\frac{1}{\beta}\log\mathop{\mathbb{E}}_{\pi(a_{t}|s_{t})}\left[e^{\beta Q_{\pi_{\theta}}^{\beta}(s_{t},a_{t})}\right]\!,\!\!\!\quad (7) \displaystyle Q_{\pi_{\theta}}^{\beta}(s_{t},a_{t})\!=\!r_{t}\!+\!\frac{1}{\beta}\log\mathop{\mathbb{E}}_{p(s_{t+1}\mid s_{t},a_{t})}\left[e^{\beta V_{\pi_{\theta}}^{\beta}(s_{t+1})}\right]. (8) The main challenge of risk-sensitive RL lies in its nonlinear Bellman equations. In contrast to the standard RL problem, sample-based estimates of the value functions are biased due to the \log(\cdot) operator. Hence, this line of work is dominated by the MDP control framework, where the transition dynamics and the reward model are available to the agent ( 19 ) . 3 Gradients of the Entropic Risk Measure The entropic risk measure commonly invoke the likelihood ratio trick to obtain an estimate of the gradient of the performance objective ( 39 ) . However, these estimates suffer from high variance, as they are computed w.r.t. full trajectories, and are proportional to the exponentiated return, which can lead to numerical instabilities during learning. In this section, we address the first issue by extending the policy gradient framework to risk-sensitive policies under the entropic risk measure. First, we introduce policy gradient theorems under the on-policy setting for both stochastic (Thm. 1 ) and deterministic (Thm. 2 ) policies. Then, we extend these two theorems to the off-policy setting by approximating these gradients, and demonstrate policy improvements for these approximations when using a tabular representation of the policy (Thm. 3 ). All proofs can be found in Appendix A. 3.1 On-policy Risk-Sensitive Policy Gradients We begin by defining the exponential twisting of dynamics and policy ( 9 ) as: p^{*}(s^{\prime}|s,a)\!\propto p(s^{\prime}|s,a)e^{\beta V_{\pi_{\theta}}^{\beta}(s^{\prime})} , and \pi_{\theta}^{*}(a|s)\propto\pi_{\theta}(a|s)e^{\beta Q_{\pi_{\theta}}^{\beta}(s,a)}, respectively, and the state distribution for dynamics p^{*}(s^{\prime}|s,a) and policy \pi^{*}_{\theta}(a|s) as \rho_{\pi}^{*}(s) . We now provide our risk-sensitive policy gradient theorem for stochastic policies. Theorem 1 . (Risk-Sensitive Stochastic Policy Gradient Theorem). The gradient of J^{\beta}(\pi_{\theta}) w.r.t. \theta is given by: \displaystyle\frac{1}{\beta}\int_{S}\rho_{\pi}^{*}(s)\int_{A}\nabla_{\theta}\pi_{\theta}(a|s)e^{\beta(Q_{\pi_{\theta}}^{\beta}(s,a)-V_{\pi_{\theta}}^{\beta}(s))}\,\mathrm{d}a\,\mathrm{d}s. (9) This gradient has two key differences from its risk-neutral version in Eq. ( 1 ). First, it is weighted by the state distribution of exponential twisted dynamics and policy, which tend to be optimistic or pessimistic, depending on \beta . Second, the action-value function is substituted by e^{\beta(Q_{\pi_{\theta}}^{\beta}(s,a)-V_{\pi_{\theta}}^{\beta}(s))} , which can lead to numerical instabilities during learning due to the exponentiated value functions. For deterministic policies, we analogously define the state distribution \rho_{\mu}^{*}(s) . We now present our risk-sensitive policy gradient for deterministic policies \mu_{\theta} . Theorem 2 . (Risk-Sensitive Deterministic Policy Gradient Theorem). The gradient of J^{\beta}(\mu_{\theta}) w.r.t. \theta is given by: \displaystyle\int_{S}\rho_{\mu}^{*}(s)\nabla_{\theta}\mu_{\theta}(s)\nabla_{a}Q_{\mu_{\theta}}^{\beta}(s,a)|_{a=\mu_{\theta}(s)}\,\mathrm{d}s. (10) In particular, we notice that there is no need for an exponential twisted policy. The deterministic gradient is more convenient for two reasons: first, it avoids the integral over actions resulting in a more efficient approach, and second, it avoids the evaluation of an exponential term, which can be numerically unstable during learning. 3.2 Off-policy Risk-Sensitive Policy Gradients We now consider methods that optimize a risk-sensitive policy from trajectories generated by an arbitrary behavior policy b(a\mid s) . To achieve this, we consider the soft-value functions of a target deterministic policy weighted by the state distribution of the behavior policy as our objective, J_{b}^{\beta}(\mu_{\theta})=\int_{S}\rho_{b}(s)V_{\mu_{\theta}}^{\beta}(s)\,\mathrm{d}s, where \rho_{b} is the state distribution under the behavior policy. Similarly to ( 13 ) , we propose an approximation of the gradient \nabla_{\theta}J_{b}^{\beta}(\mu_{\theta}) : \int_{\mathcal{S}}\rho_{b}(s)\nabla_{\theta}\mu_{\theta}(s)\nabla_{a}Q_{\mu_{\theta}}^{\beta}(s,a)|_{a=\mu_{\theta}(s)}\,\mathrm{d}s\coloneqq g(\mu_{\theta}). (11) This approximation drops a term that depends on \nabla_{\theta}Q_{\mu_{\theta}}^{\beta}(s,a) . We show that this approximation results in policy improvement when using a tabular representation for the policy. Theorem 3 . (Deterministic Policy Improvement). Given a policy \mu with parameters \theta and a linear tabular representation, let \theta^{\prime}=\theta+\alpha g(\mu_{\theta}) . Then there exist an \epsilon such that for all \alpha<\epsilon : \displaystyle V_{\mu_{\theta}}^{\beta}(s)\leq V_{\mu_{\theta^{\prime}}}^{\beta}(s)\qquad\forall s\in S. (12) For completeness, we consider an equivalent result for an off-policy performance objective that uses a stochastic policy \pi_{\theta} as its target policy in Appendix A. 4 Risk-Sensitive Exponential Actor-Critic (a) Average Initial Value (b) L2 Gradient Norm (c) Average Performance Figure 1: Instabilities in value functions and gradients. We train value functions for DQN and Z_{\psi}(s_{t},a_{t}) for a range of \beta settings ( -1,-0.1,0.1,1 ). We plot the average log-value estimates ( left ), the L2 gradient norm in log-domain ( center ), and the average return ( right ) for 20 episodes over 200k environment steps. For \beta<0 , we observe that the value estimates and the gradient norm are almost zero, leading to unstable learning. For \beta>0 , the value estimates and gradients are orders of magnitude larger than DQN’s estimates. These estimates are propagated by the gradients, resulting in exploding gradients. Motivated by theoretical results established in Sec. 3 , we now derive a risk-sensitive off-policy actor-critic algorithm that avoids sampling the entire trajectory. We begin by illustrating the issues that arise when learning a critic network that approximates the exponential value function, using the squared exponential temporal-difference (TD). We then address these problems by introducing a critic representation that avoids explicit computation of the exponential value function, and design a stabilizing mechanism for the critic gradient, based on batch normalization and clipping. Finally, we combine our improvements to propose: risk-sensitive exponential actor-critic (rsEAC), a practical algorithm that considerably improves the stability of the optimization of the entropic risk measure. 4.1 Numerical Issues of Exponential TD Learning As discussed in Sec. 2.2 , sample-based estimates of Q_{\mu_{\theta}}^{\beta} are biased due to nonlinear Bellman equations. To obtain unbiased estimates, we must consider the exponential value functions: \displaystyle Z_{\mu_{\theta}}^{\beta}(s_{t})=e^{\beta V_{\mu_{\theta}}(s_{t})},\quad Z_{\mu_{\theta}}^{\beta}(s_{t},a_{t})=e^{\beta Q_{\mu_{\theta}}(s_{t},a_{t})}. (13) These functions are related by the exponential Bellman equations, which are obtained by applying an exponential transformation to both sides of Eq. ( 8 ): \displaystyle Z_{\mu_{\theta}}^{\beta}(s_{t})=\mathbb{E}_{\mu_{\theta}(a_{t}|s_{t})}[Z_{\mu_{\theta}}^{\beta}(s_{t},a_{t})],\quad (14) \displaystyle Z_{\mu_{\theta}}^{\beta}(s_{t},a_{t})=e^{\beta r_{t}}\mathbb{E}_{p(s_{t+1}\mid s_{t},a_{t})}\left[Z_{\mu_{\theta}}^{\beta}(s_{t+1})\right]. (15) From these equations, Z_{\mu_{\theta}}^{\beta} can be estimated using only samples, which can then be transformed to recover Q_{\mu_{\theta}}^{\beta} ( 16 ) . However, it is impractical to learn different exponential value functions for each state and action pair, especially in environments with large state/action spaces. Instead, these functions can be approximated with a parameterized critic network Z_{\psi}(s_{t},a_{t}) , which is trained using stochastic gradient descent to minimize the squared exponential TD error: \mathop{\mathbb{E}}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\left[\left(Z_{\psi}(s_{t},a_{t})-e^{\beta r_{t}}Z_{\psi^{\prime}}(s_{t+1})\right)^{2}\right], (16) where \mathcal{D} is an experience replay buffer that stores previously seen interactions with the environment. The value function Z_{\psi^{\prime}}(s_{t+1}) is implicitly parameterized with a target critic network as \max_{a_{t+1}}Z_{\psi^{\prime}}(s_{t+1},a_{t+1}) . Risk-sensitive actor critic (R-AC) ( 43 ; 44 ) considers a similar loss in the context of online learning for its critic. However, two major drawbacks persist in the learning the R-AC critic network: first, the value function estimates the exponentiated reward, \mathbb{E}[e^{\beta\sum_{t}r_{t}}] , which can lead to numerical overflow / underflow of the network weights. Second, directly minimizing Eq. ( 16 ) may cause exploding (or vanishing) gradients, resulting in divergent behavior. Fig. 1 , demonstrates these instabilities by learning value functions in the CartPole task ( 3 ) in Gymnasium ( 52 ) . (a) Average Initial Value (b) L2 Gradient Norm (c) Average Performance Figure 2: Stable value functions and gradients. We train value functions for DQN and Q_{\psi}(s_{t},a_{t}) using the normalized clipped gradient for a range of \beta settings ( -1,-0.1,0.1,1 ). We plot the average value estimates ( left ), the L2 gradient norm in log-domain ( center ), and the average return ( right ) for 20 episodes over 200k environment steps. 4.2 Stabilizing Updates We now consider a critic representation that avoids numerical instabilities by parameterizing the exponential value function as Z_{\psi}(s,a)=e^{Q_{\psi}(s,a)} where Q_{\psi}(s,a) is a neural network. In particular, \frac{1}{\beta}Q_{\psi}(s,a) serves as our approximation of the soft-value function Q_{\mu_{\theta}}^{\beta} . This reparameterization is more numerically stable as gradients \nabla Q_{\psi} are taken in the log-domain. We train this function by minimizing the squared exponential TD error J_{Q}(\psi) : \!\!\!\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\left[\left(e^{Q_{\psi}(s_{t},a_{t})}-e^{\beta r_{t}+Q_{\psi^{\prime}}(s_{t+1})}\right)^{2}\right]. (17) where \mathcal{D} is the experience replay buffer. We minimize Eq. ( 17 ) using stochastic gradient descent. The value function Q_{\psi^{\prime}}(s_{t+1}) is also implicitly parameterized with a target critic network as \max_{a_{t+1}}Q_{\psi^{\prime}}(s_{t+1},a_{t+1}) . The gradient \nabla J_{Q}(\psi) is then: \!\!\!e^{Q_{\psi}(s_{t},a_{t})}\!\left(e^{Q_{\psi}(s_{t},a_{t})}\!\!-e^{\beta r_{t}+Q_{\psi^{\prime}}(s_{t+1})}\right)\nabla Q_{\psi}(s_{t},a_{t}). (18) Note that the trailing gradient \nabla Q_{\psi} is taken in log-domain, and so is numerically stable. Ignoring this term the leading factors are of the form: e^{x}(e^{x}-e^{y}) , which can be numerically unstable. First, we rearrange factors so that the numerical instability occurs only in the leading term. To do this we factor out e^{x} or e^{y} , whichever is larger. We define the helper function, f(x,y)\triangleq\begin{cases}(1-e^{y-x}),&\text{if }x\geq y\\ (e^{x-y}-1),&\text{otherwise.}\end{cases} (19) Then we have that e^{x}(e^{x}-e^{y})=e^{x+\max(x,y)}f(x,y) . Observe that the trailing term f(x,y) is numerically stable as it is bounded: f(x,y)\in[-1,1] . Numerical instability occurs only in the leading term e^{x+\max(x,y)} , which must be stabilized by subtracting a constant z in log-domain, which is equivalent to normalizing e^{x}(e^{x}-e^{y})\propto e^{x+\max(x,y)-z}f(x,y) . We defer the details of our normalizing choice z and our clipping mechanism to Appendix B. Evaluation We now show that using the normalized clipped gradient addresses the numerical issues of Z_{\psi}(s_{t},a_{t}) and can stabilize the learning of the exponential-TD objective. Again we consider the CartPole environment, and train the value functions Q_{\psi}(s_{t},a_{t}) for the same set of \beta values as in Fig. 1 . In Fig. 2(a) , we observe that the value function Q_{\psi}(s_{t},a_{t}) is less prone to suffer from numerical overflow compared to Z_{\psi}(s_{t},a_{t}) . In fact, the magnitude learned by DQN ( 37 ) is close to \frac{1}{\beta}Q_{\psi}(s_{t},a_{t}) , which is not surprising given that CartPole is a deterministic task and our approach scales the rewards by \beta . Using the normalized clipped gradient, our approach also guarantees gradients with more manageable magnitudes compared to directly optimizing the exponential-TD objective (Fig. 2(b) ). Finally, our algorithm is capable of learning optimal policies for the all the tested \beta initializations, which was only possible for one setting when learning Z_{\psi}(s_{t},a_{t}) (See Fig. 2(c) ). 4.3 Actor-Critic Optimization We now present risk-sensitive exponential actor-critic (rsEAC), a practical off-policy algorithm that optimizes the entropic risk-measure. We develop an actor-critic algorithm that updates its policy in the direction of the off-policy deterministic policy gradient in Eq. ( 11 ), and substitutes the soft-value function Q_{\mu_{\theta}}^{\beta} with our approximation Q_{\psi} . We build on the twin delayed deep deterministic policy gradient algorithm (TD3) ( 18 ) for its mechanisms on over-estimation bias reduction and its empirical success in continuous control tasks. Similarly to TD3, we maintain a pair of critics with parameters \psi_{1} and \psi_{2} , a single actor with parameter \theta , and their respective target networks \psi_{1}^{\prime} , \psi_{2}^{\prime} and \theta^{\prime} . For each timestep, we update the pair of critic networks by replacing the target in the normalized clipped gradient derived in Section 4.2 with \!\!\!y_{t}\!=\begin{cases}\beta r_{t}+\min_{i=1,2}\gamma Q_{\psi_{i}^{\prime}}(s_{t+1},a_{t+1}),&\text{if }\beta>0\\ \beta r_{t}+\max_{i=1,2}\gamma Q_{\psi_{i}^{\prime}}(s_{t+1},a_{t+1}),&\text{if }\beta<0.\end{cases} (20) where a_{t+1}\!=\!\mu_{\theta^{\prime}}(s_{t+1})\!+\!\epsilon and \epsilon\sim\text{clip}(\mathcal{N}(0,\sigma^{2}),-c^{*},c^{*}) . The actor is a deterministic policy parameterized with a neural network \mu_{\theta}(s_{t}) and is optimized w.r.t. the risk-sensitive off-policy gradient in Eq. ( 11 ), for which we substitute the soft-value function with our estimate \frac{1}{\beta}Q_{\psi_{1}}(s_{t},a_{t}) : \mathbb{E}_{s_{t}\sim\mathcal{D}}\left[\frac{1}{\beta}\nabla_{\theta}\mu_{\theta}(s_{t})\nabla_{a_{t}}Q_{\psi_{1}}(s_{t},a_{t})\mid_{a_{t}=\mu_{\theta}(s_{t})}\right]. (21) where s_{t} is sampled from the buffer \mathcal{D} . Given that the policy \mu_{\theta}(s_{t}) is deterministic, we ensure adequate exploration by adding Gaussian noise \mathcal{N}(0,\sigma^{2}) . We update the target networks using an exponentially moving average of the weights. Pseudocode for rsEAC can be found in Appendix D. 5 Related Work The entropic risk measure was first investigated in ( 25 ) as an approach to incorporate risk sensitivity in MDPs. Following this work, risk-sensitive MDPs — where the agent knows the transition dynamics or has access to a simulator of the environment — have been widely studied ( 17 ; 24 ; 11 ; 14 ; 6 ; 26 ) . More recently, risk-sensitive RL has been connected to the RL-as-inference framework ( 42 ) , in which policy search is formulated as maximum likelihood estimation on an augmented MDP ( 51 ; 28 ; 31 ) . Another formulation of risk-sensitive RL with probabilistic inference is that of policy search as an expectation maximization (EM) style algorithm ( 40 ; 1 ; 9 ; 21 ) . Risk-sensitive model-free algorithms optimize the exponential criteria as an equivalent objective of the entropic risk measure ( 4 ; 7 ; 8 ) . These approaches employ exponential value functions — functions that are associated in a multiplicative way, rather than in an additive way as in standard RL ( 16 ) — that can be estimated using Monte Carlo methods. However, the exponential criteria suffer from numerical instabilities that arise from the estimation of the expected exponentiated return. These numerical instabilities are further exacerbated when introducing function approximators ( 33 ) . Risk-sensitive policy gradient algorithms ( 39 ; 41 ) may suffer from these same instabilities. Network saturation for the exponential function is a well-known issue in neural network design ( 20 ) . While log-likelihood objectives can prevent this problem by undoing the exponential operation, this approach is incompatible with exponential TD learning, which must use the squared loss. Another popular approach for training models with saturating nonlinearities is batch normalization which normalizes each layer of a deep neural network ( 27 ) . Motivated by this approach, we design a batch normalization scheme for the last layer to avoid the saturation of the gradient, and introduce gradient clipping on the exponent, which helps with the exploding/vanishing gradient problem ( 45 ) . Our method is, to our knowledge, the first actor-critic algorithm that incorporates mechanisms to stabilize the optimization of the exponential TD learning, and our policy gradient theorems establish a rigorous framework for risk-sensitive actor-critic learning with the entropic risk measure. 6 Experiments (a) Risk-averse ( \beta=-1 ) (b) Risk-Seeking ( \beta=1 ) (c) Risk-neutral ( \beta=0.1 ) Figure 3: Tabular Risk-Sensitive Policies. Optimal value functions Z^{*}(s_{t}) (in log-domain) and corresponding trajectory for different \beta values. Agents trained with negative \beta values learn risk-averse policies that avoid the cliff-region, while agents trained with positive \beta values tend to be risk-seeking and hug the cliff closely. We also recover risk-neutral policies when the magnitude of |\beta| is small. Figure 4: Stochastic Cliff GridWorld. Left : 2D grid environment with initial-state and goal-state given by blue circle and green star, respectively. The cliff region is given by gray states. Right : We plot the learned optimal value functions (in log-domain) for \beta=-10 and a sampled trajectory. The estimated values explode in magnitude, resulting in a policy incapable of reaching the destination. The goals of our experiments are to: verify that numerical instabilities are a direct result of optimizing the exponential Bellman equations, study the risk preference and robustness of rsEAC to different values of \beta , and demonstrate that our approach can produce competitive risk-sensitive policies in comparison to other risk-sensitive methods. We begin by considering a tabular environment where we can study the objective without introducing additional approximation error. We then demonstrate that rsEAC can learn stable policies while varying the risk parameter \beta . Finally, we compare rsEAC to risk-averse baselines in three challenging risky variations of MuJoCo tasks, where risk-aware agents should avoid a noisy region. 6.1 GridWorld Environment Our aim with this experiment is to illustrate the role that \beta plays in modulating risk-sensitivity and identify potential numerical instabilities that may appear when optimizing the exponential Bellman equations in the tabular setting. We examine the 2D GridWorld described in ( 15 ) . In this environment, the agent starts in an initial position and its objective is to travel to a given destination. When the agent selects a direction (up, left, down and right), it may possibly move to its chosen direction or to a random neighboring state. We introduce aleatoric risk by defining a cliff-region that the agent should avoid. Falling into the cliff incurs a large negative reward and terminates the episode. The GridWorld environment is composed of a 2D grid with 10 × 10 states (see Fig. 4 ). The initial state and destination are depicted by a blue circle and green star, respectively. The cliff-region is depicted by a grey rectangle. The probability of moving to a random neighboring state is 0.2. Each move has a -1 cost and the penalty for falling into the cliff is -10 . We learn the value functions Z^{*}(s_{t},a_{t}) using the exponential Bellman equations, with an \epsilon -greedy policy ( \epsilon=0.1 ) and a discount factor of \gamma=0.85 . We train this algorithm by running it for 50000 episodes. In Fig. 3 , we plot learned optimal value functions Z^{*}(s_{t}) (in log-domain) and a sampled trajectory using a learned optimal policy \pi^{*} for a range of \beta values. We observe that policies trained with \beta<0 prefer longer paths that avoid the cliff-region (Fig. 3(a) ), while policies trained with \beta>0 choose the shortest path which travels next to the cliff (Fig. 3(b) ). As expected, we recover risk-neutral policies when the magnitude of |\beta| is small (Fig. 3(c) ). The learned value functions also demonstrate exploding/vanishing gradient magnitudes when estimated using |\beta| values with larger magnitudes (Fig. 4 ). We expect these numerical instabilities to develop even more often when introducing function approximators. Figure 5: Risk-sensitive policies in Inverted Pendulum. We train rsEAC and R-AC for \beta=1 and \beta=-1 . Left : We plot the average return. Right : Percentage of steps on an episode in risky regions. The solid curves correspond to the mean and shaded regions to \pm one standard deviation over 5 random seeds. rsEAC learns, both, risk-seeking and risk-averse policies that achieve high return, while R-AC encounters numerical instabilities and learns poor policies. (a) Swimmer-v4 (b) HalfCheetah-v4 (c) Ant-v4 Figure 6: Risk-Averse MuJoCo. Top row : Average return on MuJoCo benchmarks with risky regions. Bottom row : Percentage of steps on an episode in risky regions. The solid curves correspond to the mean and shaded regions to \pm one standard deviation over 10 random seeds. 6.2 Inverted Pendulum Environment We study the role of \beta in modulating risk in rsEAC and that stable policies can be learned for different risk profiles. We consider a risky variant of Inverted Pendulum — a continuous version of the CartPole environment — from MuJoCo ( 50 ) . Following ( 32 ) , we induce aleatoric risk by adding stochastic noise ( \mathcal{N}(0,1) ) to the reward function when the agent’s X-position is greater than 0.01. For our evaluations, we consider R-AC ( 44 ) and our method using two different \beta values corresponding to risk-seeking ( \beta=1 ) and risk-averse ( \beta=-1 ) settings. In Fig. 5 , we plot the average return performance and the risky region visiting rate. We report the average and standard deviation for 5 independents runs. We verify that rsEAC can learn risk-seeking and risk-averse policies — our agent prefers the risky region when trained with \beta=1 , and avoids it when trained with \beta=-1 — while retaining high performance in terms of average return. In comparison, R-AC suffers from numerical instabilities that lead to low-return policies and a lack of risk-sensitivity. 6.3 Simulated Robotic Benchmarks We evaluate rsEAC on three modified MuJoCo ( 50 ) continuous control tasks: Swimmer, HalfCheetah, and Ant. Following ( 32 ) , we modify their reward functions to make the speed positive in both directions so the agent is free to move left or right. We also construct risky regions in these tasks based on the X-position. We add a stochastic reward sampled from \mathcal{N}(0,10^{2}) for Swimmer and HalfCheetah, when the agent’s position is greater than 0.5 and smaller than -3, respectively. For the Ant task, we add a stochastic reward sampled from \mathcal{N}(0,7^{2}) when its position is greater than 0.5. For all tasks, we append the X-position into the agent’s observation. Hence, a risk-averse agent would do its best effort to avoid these risky regions. We compare our algorithm against three risk-averse baselines: Mean Gini deviation (MG) ( 32 ) , a policy gradient algorithm that considers Gini deviation as its risk measure and outperforms other return variance methods; mean-variance policy iteration (MVPI) ( 53 ) , a flexible algorithm that optimizes reward-volatility risk measure and demonstrates great empirical success in risk-averse continuous domains; and risk-sensitive actor critic (R-AC) ( 44 ) , an online actor-critic algorithm that optimizes the entropic risk measure. To make the comparison fair between different methods, we implement every actor-critic algorithm on top of TD3 ( 18 ) , and use the same network architecture across all algorithms. For MG, we use PPO ( 46 ) to learn its policy gradient, as suggested by the author’s implementation. See Appendix C for additional details. We report the average return ( top-row ) and the risky region visiting rate ( bottom-row ) for the three risk-averse MuJoCo environments (See Fig. 6 ). The results show that rsEAC produces policies that are risk-averse (low rate in risky regions), and perform comparably to other baselines in terms of average return. In particular, it outperforms R-AC in every task, showing that our stabilizing mechanisms play an essential role in the learning of risk-averse and high-return policies. Our approach performs comparably to MVPI in terms of final performance across environments, while outperforming it in the higher-dimensional Ant task. 7 Conclusion In this paper, we study risk-sensitive reinforcement learning using the entropic risk-measure objective. We establish that existing policy gradient approaches learn high-variance and numerically unstable estimates that may lead to divergent behavior by the agent. To resolve these issues, we derive policy gradient theorems that circumvent the need to estimate a gradient for the full trajectory, and propose a critic parameterization for the exponential value function that avoids its explicit computation. We optimize this critic using stochastic gradient descent, for which we too propose a stabilizing mechanism based on mini-batch normalization and clipping. Taking these components together, we then propose risk-sensitive exponential actor-critic (rsEAC), a practical algorithm that greatly improves the stability of the optimization of the entropic risk measure. Our evaluations demonstrate that rsEAC can learn risk-sensitive and high-return policies in complex continuous tasks. Limitations Our approach keeps two main limitations. There is an inherent instability when working with the exponential function, and even our solutions may not be enough to stabilize updates in all settings. Second, a key component for our method is the risk parameter \beta which must be tuned to balance the amount of risk-sensitivy for different tasks. Appendix A Proofs of Theoretical Results A.1 Proof of Risk-Sensitive Stochastic Policy Gradient Theorem This proof considers similar regularity assumptions as in ( 47 ) : Assumption 1: \pi_{\theta}(a|s) , \nabla_{\theta}\pi_{\theta}(a|s) , p(s^{\prime}|s,a) , p_{1}(s) are continuous in s , a , s^{\prime} and \theta . Assumption 2: there exists a b such that \sup_{s}p_{1}(s)0 : \displaystyle e^{\beta V_{\mu_{\theta}}^{\beta}(s)} \displaystyle\leq e^{\beta Q_{\mu_{\theta}}^{\beta}(s,\mu_{\theta^{\prime}}(s))} \displaystyle=\mathbb{E}_{a\sim\mu_{\theta^{\prime}}(s)}\mathbb{E}_{r,s^{\prime}\sim p(\cdot|s,a)}[e^{\beta r}e^{\beta V_{\mu_{\theta}}^{\beta}(s^{\prime})}] \displaystyle\leq\mathbb{E}_{a\sim\mu_{\theta^{\prime}}(s)}\mathbb{E}_{r,s^{\prime}\sim p(\cdot|s,a)}[ \displaystyle e^{\beta r}\mathbb{E}_{a^{\prime}\sim\mu_{\theta^{\prime}}(s^{\prime})}[e^{\beta Q_{\mu_{\theta}}^{\beta}(s^{\prime},a^{\prime})}]] \displaystyle=\mathbb{E}_{a\sim\mu_{\theta^{\prime}}(s)}[e^{\beta Q_{\mu_{\theta^{\prime}}}^{\beta}(s,a)}] \displaystyle=e^{\beta V_{\mu_{\theta^{\prime}}}^{\beta}(s)}. Hence, we can conclude that V_{\mu_{\theta}}^{\beta}(s)\leq V_{\mu_{\theta^{\prime}}}^{\beta}(s) for all s\in\mathcal{S} . The proof for \beta<0 follows similar arguments. ∎ A.4 Proof of Policy Improvement for the Off-Policy Risk-Sensitive Stochastic Setting We now consider the off-policy performance objective for a stochastic policy, soft-value functions of a target stochastic policy weighted by the state distribution of the behavior policy, J_{b}^{\beta}(\pi_{\theta})=\frac{1}{\beta}\sum_{s\in\mathcal{S}}\rho_{b}(s)e^{\beta V_{\pi_{\theta}}^{\beta}(s)} where \rho_{b} is the state distribution under the behavior policy b(a|s) . Similarly to ( 13 ) , we propose an approximation of the gradient: \nabla_{\theta}J_{b}^{\beta}(\pi_{\theta})\!\approx\!\frac{1}{\beta}\sum_{s\in S}\rho_{b}(s)\sum_{a\in A}\nabla_{\theta}\pi_{\theta}(a|s)e^{\beta Q_{\pi_{\theta}}^{\beta}(s,a)}\!\!:=\!g(\pi_{\theta}) This approximation drops a term that depends on \nabla_{\theta}Q_{\pi_{\theta}}(s,a) . We show that this approximation results in policy improvement when using a tabular representation for the policy. Theorem 3.4 . (Stochastic Policy Improvement). Given a policy \pi with parameters \theta and a tabular representation, let \theta^{\prime}=\theta+\alpha g(\pi_{\theta}) . Then there exist an \epsilon>0 such that for all positive \alpha<\epsilon , \displaystyle V_{\pi_{\theta}}^{\beta}(s)\leq V_{\pi_{\theta^{\prime}}}^{\beta}(s) for all s\in S . Proof. Notice first that for any s , the gradient \sum_{a\in\mathcal{A}}\nabla_{\theta}\pi_{\theta}(a|s)e^{\beta Q_{\pi_{\theta}}^{\beta}(s,a)} is the direction that increases \sum_{a\in\mathcal{A}}\pi_{\theta}(a|s)e^{\beta Q_{\pi_{\theta}}^{\beta}(s,a)} for a fixed soft-value function Q_{\pi_{\theta}} . For an appropriate step size \alpha , we can guarantee that \displaystyle J_{\beta}(\pi_{\theta})=\sum_{s\in\mathcal{S}}\rho_{b}(s)\sum_{a\in\mathcal{A}}\pi_{\theta}(a|s)e^{\beta Q_{\pi_{\theta}}^{\beta}(s,a)} \displaystyle\leq\sum_{s\in\mathcal{S}}\rho_{b}(s)\sum_{a\in\mathcal{A}}\pi_{\theta^{\prime}}(a|s)e^{\beta Q_{\pi_{\theta}}^{\beta}(s,a)} Given that \pi_{\theta} has a tabular representation, we have that \displaystyle e^{\beta V_{\pi_{\theta}}^{\beta}(s)}=\sum_{a\in\mathcal{A}}\pi_{\theta}(a|s)e^{\beta Q_{\pi_{\theta}}^{\beta}(s,a)} \displaystyle\leq\sum_{a\in\mathcal{A}}\pi_{\theta^{\prime}}(a|s)e^{\beta Q_{\pi_{\theta}}^{\beta}(s,a)}=\mathbb{E}_{a\sim\pi_{\theta^{\prime}}}[e^{\beta Q_{\pi_{\theta}}^{\beta}(s,a)}] because the policy can be updated independently for each state with separate weights for each state. Hence, for any s\in\mathcal{S} and \beta>0 : \displaystyle e^{\beta V_{\pi_{\theta}}^{\beta}(s)} \displaystyle\leq\mathbb{E}_{a\sim\pi_{\theta^{\prime}}}[e^{\beta Q_{\pi_{\theta}}^{\beta}(s,a)}] \displaystyle=\mathbb{E}_{a\sim\pi_{\theta^{\prime}}}[\mathbb{E}_{r,s^{\prime}\sim p(\cdot|s,a)}[e^{\beta r}e^{\beta V_{\pi_{\theta}}^{\beta}(s^{\prime})}]] \displaystyle\leq\mathbb{E}_{a\sim\pi_{\theta^{\prime}}}[\mathbb{E}_{r,s^{\prime}\sim p(\cdot|s,a)}[e^{\beta r}\mathbb{E}_{a^{\prime}\sim\pi_{\theta^{\prime}}}[e^{\beta Q_{\pi_{\theta}}^{\beta}(s^{\prime},a^{\prime})}]]] \displaystyle=\mathbb{E}_{a\sim\pi_{\theta^{\prime}}}[e^{\beta Q_{\pi_{\theta^{\prime}}}^{\beta}(s,a)}] \displaystyle=e^{\beta V_{\pi_{\theta^{\prime}}}^{\beta}(s)}. Hence, we can conclude that V_{\pi_{\theta}}^{\beta}(s)\leq V_{\pi_{\theta^{\prime}}}^{\beta}(s) for all s\in\mathcal{S} . The proof for \beta<0 follows similar arguments. ∎ Appendix B Critic Gradient Stabilization Figure 7: Histograms for stabilizing operations on a mini-batch. We compute m_{i} values and z for a mini-batch of size 256. Histogram of: m_{i} values ( blue ), m_{i} values after subtracting z ( orange ), m_{i} values after subtraction by z and clipping ( green ). In this section, we present our derivation of a constant z that sufficiently normalizes the gradient over the current mini-batch, and our clipping mechanism. We achieve this by defining, \!\!\!\!m_{\beta}^{i}\triangleq Q_{\psi}(s_{t}^{i},a_{t}^{i})+\max(Q_{\psi}(s_{t}^{i},a_{t}^{i}),\beta r_{t}^{i}+Q_{\psi^{\prime}}(s_{t+1}^{i})) (22) where \{(s_{t}^{i},a_{t}^{i},s_{t+1}^{i},r_{t}^{i})\}_{i=1}^{N} are samples from the current mini-batch. Then, we set the constant z to \max_{i}m_{\beta}^{i} when \beta>0 , and to \min_{i}m_{\beta}^{i} when \beta<0 . This yields a gradient proportional to \nabla J^{i}_{Q}(\psi) : e^{m_{\beta}^{i}-z}f(Q_{\psi}(s_{t}^{i},a_{t}^{i}),\beta r_{t}^{i}+Q_{\psi^{\prime}}(s_{t+1}^{i}))\nabla Q_{\psi}(s_{t}^{i},a_{t}^{i}) (23) Finally, we avoid any numerical overflow (underflow) by clipping the exponent of the leading term in the interval [-c,c] as: e^{\text{clip}({m_{\beta}^{i}-z},-c,c)} to compute \nabla J_{\text{clip},Q}^{i} . In Fig. 7 , we illustrate the operations performed to the leading terms in a mini-batch during learning in the CartPole environment. Appendix C Implementation Details C.1 CartPole environment Table 1 lists the hyperparameters we used for the CartPole environment. Table 1: Hyperparameters for CartPole environment Schedule details Env steps before training 10000 steps Env steps per epoch 1000 steps Network details Discount factor 0.99 Soft target update 0.005 Experience buffer \mathcal{D}_{\text{env}} 10,000 Value Network 2 hidden layers size 128 Network optimizer AdamW Non-linear layers ReLU Learning rate 0.0003 Policy details \epsilon exploration value 0.1 Computational details per run CPU 1 Altus XE2214GT RAM 5GBs GPU 1 Tesla V100S 32GB Running time 2 hours C.2 Mujoco environments Hyperparameters Each task is run for 1 million environment steps with evaluations every 10000 time steps. Agents are tested for 20 episodes per evaluation. We perform 10 runs of each algorithm with different random seeds and report average and standard deviation. For MG and MVPI, we use the implementations in ( 32 ) and follow the same hyperparameters suggested by the authors. We implement R-AC ( 44 ) on top of the TD3, and select its risk parameter from \{-1,-0.1,-0.05,-0.01\} . For rsEAC, we select \beta from \{-0.05,-0.01,-0.0025\} . We use the same network architectures and learning rates for all algorithms. Table 2 lists the hyperparameters used for rsEAC and all actor-critic algorithms in risk-aware MuJoCo benchmarks. Table 2: Hyperparameters for risk-aware MuJoCo Schedule details Env steps before training 5000 steps Env steps per epoch 1000 steps Network details Discount factor 0.99 Soft target update 0.005 Experience buffer \mathcal{D}_{\text{env}} 1,000,000 Actor Architecture 2 hidden layers size 256 Critic Architecture 2 hidden layers size 256 Network optimizer Adam Non-linear layers ReLU Exponential gradient clip 5 Learning rate 0.0003 Policy noise 0.2 Noise clip 0.5 Iterations before policy update 2 Computational details per run CPU 1 Altus XE2214GT RAM 5GBs GPU 1 Tesla V100S 32GB Running time 8 hours Appendix D Pseudocode of rsEAC This section contains the pseudocode for our algorithm, rsEAC. Algorithm 1 rsEAC Initialize networks, parameters and buffers: for t=1 to T do Select action with exploration noise: a_{t}\sim\mu_{\theta}(s_{t})+\epsilon , \epsilon\sim\mathcal{N}(0,\sigma^{2}) Sample next state and reward from environment: s_{t+1},r_{t}\sim p(s_{t+1},r_{t}|s_{t},a_{t}) Store transition to experience buffer: \mathcal{D}\leftarrow\mathcal{D}\cup\{(s_{t},a_{t},s_{t+1},r_{t}) } Sample mini-batch of size N from \mathcal{D} : \{(s_{t}^{i},a_{t}^{i},s_{t+1}^{i},r_{t}^{i})\}_{i=1}^{N}\sim\mathcal{D} Sample next actions: a_{t+1}^{i}\sim\mu_{\theta}(s_{t+1}^{i})+\epsilon^{i} , \epsilon^{i}\sim\text{clip}(\mathcal{N}(0,\sigma^{2}),-c^{*},c^{*}) Compute target: y^{i}_{t}=\beta r_{t}^{i}+\gamma\min_{j=1,2}Q_{\psi_{j}^{\prime}}(s_{t+1}^{i},a_{t+1}^{i}) Compute our gradient \nabla\!J_{\text{clip},Q}^{i}(\psi) : =e^{\text{clip}(m_{\beta}^{i}-z,-c,c)}f(Q_{\psi}(s_{t}^{i},a_{t}^{i}),y_{t}^{i})\nabla Q_{\psi}(s_{t}^{i},a_{t}^{i}) Update critics: \psi\leftarrow\psi-N^{-1}\sum_{i}\nabla J_{\text{clip},Q}^{i}(\psi) if t\text{ mod }d then Compute deterministic policy gradient \nabla J(\theta) : =N^{-1}\sum_{i}\nabla_{\theta}\mu_{\theta}(s_{t}^{i})\nabla_{a_{t}}\frac{1}{\beta}Q_{\psi_{1}}(s_{t}^{i},a_{t}^{i})|_{a_{t}^{i}=\mu_{\theta}(s_{t}^{i})} Update actor: \theta\leftarrow\theta-\nabla J(\theta) Update target networks: \theta^{\prime}\leftarrow\tau\theta+(1-\tau)\theta^{\prime} \psi_{i}^{\prime}\leftarrow\tau\psi_{i}+(1-\tau)\psi_{i}^{\prime} end if end for