Risk-sensitive reinforcement learning using expectiles, shortfall risk and optimized certainty equivalent risk

Sumedh GupteIndian Institute of Technology Madras [email protected], Shrey Rakeshkumar Patel Indian Institute of Technology Madras [email protected] Soumen PachalIndian Institute of Technology Madras [email protected] Prashanth L. A.Indian Institute of Technology Madras prash · 2026

Read Risk-sensitive reinforcement learning using expectiles, shortfall risk and optimized certainty equivalent risk by Sumedh GupteIndian Institute of Technology Madras [email protected], Shrey Rakeshkumar Patel Indian Institute of Technology Madras [email protected] Soumen PachalIndian Institute of Technology Madras [email protected] Prashanth L. A.Indian Institute of Technology Madras prash (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09300.txt.

Introduction

Shrey Rakeshkumar Patel and four co-authors study reward learning under an expectile-based goal, a risk-touchy option beside plain hoped return. They first work out mean-squared-error and bunching limits for the classic expectile guesser from separate samples, the first non-endless study of expectiles, of solo interest. A Monte Carlo guesser for the expectile of a policy's Markov chain follows, with limits in hope and with high odds, plus a floor limit proving rank-bestness.

For steering, a policy-slope rule for expectiles yields a slope guesser with mean-squared error fading as one over the number of runs. Under standard policy-slope needs, smoothness of the expectile-touchy goal gives steady closing rates for the put-forward risk-touchy recipe. Number trials on popular tests back the use of expectiles for risk-aware learning.

Catalog record

CREATORSumedh GupteIndian Institute of Technology Madras [email protected], Shrey Rakeshkumar Patel Indian Institute of Technology Madras [email protected] Soumen PachalIndian Institute of Technology Madras [email protected] Prashanth L. A.Indian Institute of Technology Madras prash
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE124 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-09300

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Risk-sensitive reinforcement learning using expectiles, shortfall risk and optimized certainty equivalent risk Sumedh GupteIndian Institute of Technology Madras [email protected], Shrey Rakeshkumar Patel Indian Institute of Technology Madras [email protected] Soumen PachalIndian Institute of Technology Madras [email protected] Prashanth L. A.Indian Institute of Technology Madras [email protected] Sanjay P. BhatTCS Research, India [email protected] Abstract We propose risk-sensitive reinforcement learning algorithms catering to three families of risk measures, namely expectiles, utility-based shortfall risk and optimized certainty equivalent risk. For each risk measure, in the context of a finite horizon Markov decision process, we first derive a policy gradient theorem. Second, we propose estimators of the risk-sensitive policy gradient for each of the aforementioned risk measures, and establish O ⁡ ( 1 / m ) O\left(1/m\right) mean-squared error bounds for our estimators, where m m is the number of trajectories. Further, under standard assumptions for policy gradient-type algorithms, we establish smoothness of the risk-sensitive objective, in turn leading to stationary convergence rate bounds for the overall risk-sensitive policy gradient algorithm that we propose. Finally, we conduct numerical experiments to validate the theoretical findings on popular RL benchmarks. * * footnotetext: Equal contribution 1 Introduction In recent years, risk-sensitive reinforcement learning (RL) has received increased research attention, cf. borkar2001sensitivity ; kose2021risk ; prashanth2015cumulative ; markowitz2023risk ; borkar2010learning ; prashanth2016variance ; Mihatsch02RS ; tamar2012policy for some representative works. The shortcomings of expected value as an objective for decision making and the need for incorporating risk is well-known, especially in domains such as finance, robotics, and transportation. Several risk measures have been studied in an RL framework, e.g., variance, quantiles, Conditional Value-at-Risk (CVaR), and cumulative prospect theory. In this paper, we consider three risk measures, namely expectiles newey1987asymmetric ; bellini2017risk , utility-based shortfall risk (UBSR) follmer2002convex and optimized certainty equivalent (OCE) ben-tal_old-new_2007 risk. Each of these risk measures comes under the realm of convex risk measures, which in financial parlance ensures that diversification does not increase risk. UBSR and OCE constitute a rich class of risk measures with several interesting special cases, namely entropic risk, quadratic, CVaR, optimal net-present value (ONPV) Pflug2001Risk , mean-variance and quartic risks hamm_2013_stochastic_root-finding_oce , see Table 2 in Appendix A . For a textbook treatment of risk measures and their properties that capture the attitude towards risk, the reader is referred to follmer2016stochastic . A prominent solution approach for optimizing risk measures in an RL context involves the use of policy gradient algorithms, see prashanth2022risk . We now summarize our contributions for each class of risk measures that we study. Expectiles. First, we derive mean-squared error and high probability bounds for the classic estimator of expectiles from independent and identically distributed (i.i.d.) samples. Second, we derive both lower and upper bounds for estimating an expectile of the cumulative discounted cost in a Markov cost process. Third, we derive an analogue of the policy gradient theorem albeit with an expectile objective for a finite-horizon MDP. Using this theorem, we form an estimator of the expectile gradient using sample trajectories of the underlying MDP and establish mean-squared error bounds for our proposed estimator. UBSR and OCE. Unlike expectiles, estimation of UBSR and OCE from i.i.d. samples has been studied in the literature, see Hegde2024 ; hamm_2013_stochastic_root-finding_oce . In this work, we focus on the policy gradient approach for UBSR and OCE. Towards this goal, we first derive policy gradient theorem analogues for UBSR and OCE objectives in a finite horizon MDP. This result is then employed to arrive at UBSR and OCE gradient estimators using trajectories simulated from the underlying MDP. Risk-aware policy gradient. We analyze a general policy gradient algorithm for optimizing a risk measure under the following assumptions: (i) the objective is smooth; and (ii) the MSE of gradient estimators is bounded above by O(1/m) , where m is the number of trajectories. We establish convergence of this template algorithm to an approximate first-order stationary point in the non-asymptotic regime. Next, we show that expectiles, UBSR and OCE satisfy the aforementioned assumptions, and hence, the bounds for the template algorithm can be specialized to these risk measures. Experiments. We implemented REINFORCE Williams1992 and our risk-aware policy gradient algorithms on Reacher MuJoCo environment. From our experiments, we observe that expectile and UBSR with a quadratic loss function outperform REINFORCE by exhibiting better average rewards and lower variance. Related work. Expectiles have been studied for several decades, see taylor2008estimating ; newey1987asymmetric ; bellini2017risk ; delbaen2013remark ; Bellini_2015 ; ziegel_coherence_2016 ; adam2025nonparametric . However, to the best of our knowledge, bounds for expectile estimation in the i.i.d. and Markovian cases have not been derived in the literature and our work fills this research gap. Moreover, in the RL framework, there is limited exploration of expectiles barring a few exceptions, see clavier2024bootstrapping ; rowland2019statistics ; jullien2023distributional . In clavier2024bootstrapping , the authors formulate a expectile variant of the Bellman equation and derive algorithms for finding a policy that solves this equation. In rowland2019statistics ; jullien2023distributional , distributional RL is studied with expectiles. The aforementioned references do not consider a policy gradient approach for optimizing expectiles and the policy obtained by their algorithms do not necessarily optimize the expectile of the cumulative cost random variable. While UBSR and OCE estimation has been explored well in the literature, see dunkel_efficient_2007 ; dunkel2010stochastic ; JMLR-LAP-SPB ; hamm_2013_stochastic_root-finding_oce ; zhaolin2016ubsrest ; Hegde2024 , there is limited research that integrates UBSR and OCE within a RL framework with the exception of shen2014risk ; xu_regret_2023 . In shen2014risk , the authors study a nested variant of UBSR in a discounted MDP. As in the case of expectiles, the policy that optimizes nested UBSR is not necessarily optimal for UBSR applied directly to the cumulative cost random variable. We optimize the latter objective using policy gradients. In xu_regret_2023 , the authors consider iterative algorithms for the episodic risk-sensitive RL problems using the OCE risk measure, and obtain regret bounds in a model-based setting. In contrast, we consider the model free setting, and derive a policy gradient theorem for OCE, and obtain non-asymptotic bounds on the convergence of the policy parameters. 2 Preliminaries We define a finite-horizon Markov Decision Process (MDP) as the tuple \left<S,A,P,C,\gamma\right> , where S and A denote the state space and the action space, respectively. We use the symbol \Delta to denote a probability distribution. For instance, given a set A , \Delta(A) denotes a probability distribution over A . The sets S and A are compact, and horizon length is given by T . The MDP transition kernel is defined as P:S\times A\to\Delta(S) , where P(s^{\prime}|s,a) denotes the probability of going to state s^{\prime} after taking action a in state s and \Delta(S) denotes a probability distribution over S . The MDP cost function is defined as C:S\times A\times S\to\Delta(\mathrm{R}) respectively, and \gamma\in[0,1] denotes the discount factor. The initial state distribution is denoted by P_{0} . The trajectory space is denoted by \mathcal{T} , where every \tau\in\mathcal{T} is a set of states, actions and rewards, denoted as \tau=\left<s_{0},a_{0},r_{1},s_{1},a_{1},r_{2},s_{2},\ldots,s_{T-1},a_{T-1},r_{T},s_{T}\right> . The cumulative discounted cost c(\tau) corresponding to trajectory \tau is given by c(\tau)=\sum_{t=0}^{T-1}\gamma^{t}C(s_{t},a_{t},s_{t+1}) . We consider the set of stochastic policies denoted as \Pi , where each \pi:S\to\Delta(A) in \Pi is Markovian and stationary. We assume that the policy set \Pi is

Read the complete text