Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning

Mehrdad Moghimi, Hyejin Ku · 2025

Read Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning by Mehrdad Moghimi, Hyejin Ku (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03900.txt.

Introduction

Mehrdad Moghimi and Hyejin Ku build a risk-aware version of a standard learning algorithm, in a 2025 paper. Reinforcement learning agents usually chase average reward. In high-stakes settings, the bad tail of outcomes matters more than the average. Distributional methods let agents feel that tail, but bolting risk measures onto value estimates often yields poor policies.

Their framework optimizes spectral risk measures directly. This family of measures includes worst-case-focused scores like CVaR and lets designers tune how much each part of the outcome distribution counts. The method works for both online learning and learning from fixed logs. The authors prove convergence for finite problems. Across many test domains, their algorithms beat existing risk-sensitive methods in both settings. They argue proper optimization of the risk objective, not just risk-aware evaluation, is the missing piece.

Catalog record

CREATORMehrdad Moghimi, Hyejin Ku
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03900

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning Mehrdad Moghimi, Hyejin Ku Abstract The development of Distributional Reinforcement Learning (DRL) has introduced a natural way to incorporate risk sensitivity into value-based and actor-critic methods by employing risk measures other than expectation in the value function. While this approach is widely adopted in many online and offline RL algorithms due to its simplicity, the naive integration of risk measures often results in suboptimal policies. This limitation can be particularly harmful in scenarios where the need for effective risk-sensitive policies is critical and worst-case outcomes carry severe consequences. To address this challenge, we propose a novel framework for optimizing static Spectral Risk Measures (SRM), a flexible family of risk measures that generalizes objectives such as CVaR and Mean-CVaR, and enables the tailoring of risk preferences. Our method is applicable to both online and offline RL algorithms. We establish theoretical guarantees by proving convergence in the finite state-action setting. Moreover, through extensive empirical evaluations, we demonstrate that our algorithms consistently outperform existing risk-sensitive methods in both online and offline environments across diverse domains. 1 Introduction Risk management plays a crucial role in sequential decision-making tasks across various domains, including finance, healthcare, and robotics. To address these risks, many approaches change the standard objective of maximizing expected returns to alternative risk measures, such as the Conditional Value-at-Risk (CVaR) ( Bäuerle and Ott 2011 ) or coherent risk measures ( Tamar et al. 2017 ) . Another strategy is to impose constraints, such as variance-based limits ( Tamar et al. 2012 ) or dynamic risk measures ( Chow and Pavone 2013 ) , to control exposure to worst-case scenarios. Recently, Distributional Reinforcement Learning (DRL) has gained significant attention as a framework for risk-sensitive RL (RSRL) by estimating the entire distribution of returns rather than focusing solely on the expected value ( Morimura et al. 2010 ; Bellemare et al. 2017 ) . This approach offers a straightforward method for risk mitigation by employing a risk measure, such as a Mean-Variance or Distortion risk measure, instead of the expected value ( Dabney et al. 2018a ; Ma et al. 2020 ) . However, within the DRL framework, applying a fixed risk measure at each time step does not necessarily result in policies that optimize static or dynamic risk measures over the entire trajectory ( Lim and Malik 2022 ) . With such iterative risk measures, action selection at different states may deviate from the agent’s overall risk preferences, potentially leading to suboptimal policies. For example, optimizing \operatorname{CVaR}_{0.1} at a future state may not align with optimizing the worst 10% of returns from the initial state. This phenomenon, referred to as time inconsistency, is a well-documented challenge in risk-sensitive decision-making ( Shapiro et al. 2014 ) . Due to the simplicity of incorporating iterative risk measures with DRL, this approach has also been adopted in offline RL ( Urpí et al. 2021 ; Ma et al. 2021 ) . In offline RL, policies are learned from previously collected data ( Levine et al. 2020 ) and risk-sensitive offline RL with iterative risk measures not only face the same issues observed in the online setting but also encounter additional challenges unique to offline RL, such as the inability to explore new high-reward states outside the dataset and the distributional shift problem, where the policy is trained on one state-action distribution but evaluated on another. To address these issues, existing offline RL methods often employ techniques like policy constraints ( Fujimoto et al. 2019 ; Peng et al. 2019 ; Nair et al. 2021 ) and value function regularization ( Kumar et al. 2020 ) . Figure 1 : The risk-sensitivity changes implicitly with Static risk measures (Left) . However, Iterative risk measures have the same risk-sensitivity regardless of the observed path (Right) . Optimizing static risk measures offers a more interpretable solution by focusing on policies that maximize worst-case returns. Unlike dynamic or iterative risk measures, which assume that worst-case scenarios can occur at each step, static risk measures impose a limit on the worst-case scenarios that can occur over an entire episode ( Chow et al. 2015 ) . Consequently, with static risk measures, the risk level of the agent changes implicitly based on the observed path ( Pflug and Pichler 2016 ; Moghimi and Ku 2025 ) . While this behavior may seem unintuitive, it reflects a key feature of static measures: after observing a favorable sequence of events, the agent becomes more cautious in subsequent steps to respect the overall bound on worst-case scenarios. Although optimizing static CVaR has been the focus of many studies due to its simple formulation, one of its major drawbacks is that it exclusively focuses on the left tail of the return distribution, ignoring potential upsides. This limitation, often referred to as “Blindness to Success” ( Greenberg et al. 2022 ) , can result in suboptimal policies. To address this issue, as well as the previously mentioned challenges, we introduce a simple yet powerful actor-critic framework for optimizing static Spectral Risk Measures (SRM) that can be applied in both offline and online settings. By breaking down the steps required to optimize an SRM, we demonstrate how such a framework can be derived. Moreover, since SRM is a flexible risk measure that encompasses various measures, such as Mean-CVaR and the Exponential risk measure, this framework fully leverages the potential of DRL. In scenarios where avoiding worst-case outcomes is critical and collecting new data is costly, learning effective risk-sensitive policies from offline datasets is especially valuable, as it eliminates the need for environment interaction. Despite this importance, existing work on risk-sensitive offline reinforcement learning has largely focused on iterative risk measures. To the best of our knowledge, the optimization of static risk measures remains unexplored in the offline setting. A further advantage of learning risk-sensitive policies offline is the ability to tailor policies to different levels of risk aversion. In this regard, the flexibility of static spectral risk measures enables the learning of a diverse set of policies adapted to varying risk preferences. As we demonstrate in our experiments, our algorithms not only achieve strong performance under risk-sensitive criteria but also allow for the customization of policies to suit specific risk profiles. Our framework can be utilized with both stochastic and deterministic policies. Stochastic policies are well-suited for environments with uncertainty and imperfect information due to their inherent randomness ( Sutton and Barto 2018 ) . However, this randomness can degrade the performance of risk-averse policies, and the exploration advantages of stochastic policies are lost in the offline setting ( Urpí et al. 2021 ) . This motivates the use of deterministic policies in our framework. For this purpose, we utilize TD3 ( Fujimoto et al. 2018 ) and TD3BC ( Fujimoto and Gu 2021 ) as high-performing online and offline RL algorithms with deterministic policies. Accordingly, in addition to our stochastic policy-based methods, AC-SRM for online learning and OAC-SRM for offline learning, we introduce TD3-SRM and TD3BC-SRM as their deterministic counterparts for online and offline settings, respectively. Our empirical results demonstrate that optimizing static SRM not only addresses theoretical concerns but also produces policies that outperform

Read the complete text