Risk-Aware Reinforcement Learning Reward for Financial Trading
Uditansh Srivastava, Shivam Aryan, Shaurya Singh · 2025
Read Risk-Aware Reinforcement Learning Reward for Financial Trading by Uditansh Srivastava, Shivam Aryan, Shaurya Singh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04358.txt.
Introduction
Uditansh Srivastava, Shivam Aryan, and Shaurya Singh propose a risk aware reward for trading agents in a 2025 paper. Reinforcement learning agents learn to trade by chasing a reward score, but single measures like the Sharpe ratio capture only part of what investors want. Their reward blends four smooth terms: yearly return, downside risk, extra return over a benchmark, and a ratio that weighs return against market risk.
Weights on the four terms can be tuned by grid search to match a chosen risk and return profile. The authors derive exact gradients for each term to support gradient based training. They also analyze traits like monotonicity and boundedness. Readers get a modular blueprint for multi goal rewards in trading and beyond.
Catalog record
| CREATOR | Uditansh Srivastava, Shivam Aryan, Shaurya Singh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-04358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Risk-Aware Reinforcement Learning Reward for Financial Trading Uditansh Srivastava, Shivam Aryan, Shaurya Singh Abstract We propose a novel composite reward function for a reinforcement learning (RL) trading agent that explicitly balances return and risk by combining four differentiable components—annualized return, downside risk, differential return, and the Treynor ratio. Unlike traditional single-metric objectives (e.g., Sharpe or cumulative return), which can encourage reward hacking or over-optimization of one aspect of trading, our formulation is inherently modular and weighted w 1 … w 4 w_{1}\ldots w_{4} to allow practitioners to encode diverse investor preferences and adjust emphasis across multiple financial goals. We also demonstrate hyperparameter tuning of these weights via grid search to identify configurations that align with different risk–return profiles. We present the full mathematical form of the reward, derive closed-form gradients for each term to ensure compatibility with gradient-based learning, and analyze key theoretical properties including monotonicity, boundedness, and modularity. This abstract framework serves as a general blueprint for constructing robust, multi-objective reward functions in complex trading environments and can be readily extended with additional risk measures or adaptive weighting schemes. 1 Introduction In financial trading and portfolio optimization, designing effective objective functions that capture both return and risk is essential. Modern Portfolio Theory (MPT) emphasizes this trade-off [ 3 ] , and performance metrics such as the Sharpe ratio [ 7 ] and the Treynor ratio [ 11 ] extend this notion by measuring returns relative to total and systematic risk, respectively. In reinforcement learning (RL) applications to trading, prior work has shown that directly optimizing such financial performance metrics can be effective [ 5 ] . However, traditional RL reward formulations often rely on a single metric—such as cumulative return or Sharpe ratio—which can lead to reward hacking and poor generalization in the inherently multi-objective landscape of financial markets. To address this limitation, we propose a modular, multi-objective reward framework that explicitly balances return and risk using four domain-informed components. This formulation enables the reward to reflect a broader range of investor preferences and reduces the risk of over-optimization on any single aspect of trading performance. Our main contributions are as follows: • We introduce a composite reward function R that combines annualized return, downside risk (penalty), benchmark outperformance, and the Treynor ratio, and provide its full mathematical formulation. • We offer comprehensive financial interpretations of each component in R —including return, risk penalty, alpha, and systematic risk adjustment—supported by key references to financial theory [ 6 , 9 ] . • We derive and analyze the partial derivatives (gradients) of each component of R with respect to portfolio weights, confirming that the reward is piecewise differentiable and thus suitable for gradient-based optimization. • We propose a grid-search procedure for tuning the weight coefficients w_{1},w_{2},w_{3},w_{4} . We explore how variations in these weights influence the agent’s risk preferences and decision-making. • We investigate the theoretical properties of the reward, including monotonicity (the relationship between returns and risk), boundedness of each term, and modularity (the independence of components), ensuring a robust design. • We highlight the practical implications of our reward framework for RL-based stock trading and portfolio management, demonstrating how the agent’s objectives align with diverse financial goals. 2 Related Work Modern portfolio theory, initiated by Markowitz (1952) [ 3 ] , formalized the trade-off between expected return and variance in portfolio selection. Later, the Capital Asset Pricing Model (CAPM) and related metrics were developed to evaluate portfolio performance relative to market risk [ 6 ] . The Sharpe ratio uses total volatility (standard deviation) as a risk measure [ 7 ] , while the Treynor ratio uses systematic risk (beta) relative to a benchmark [ 11 ] . Both are foundational in risk-adjusted performance evaluation. The concept of downside risk (e.g., semi-variance or Sortino ratio) emphasizes losses below a threshold (such as zero or a minimum return) [ 9 ] . In reinforcement learning for finance, Moody and Saffell [ 5 ] pioneered the idea of optimizing trading policies by maximizing financial performance functions directly, such as profits, Sharpe ratio, or other utility functions. Subsequent work in deep RL for portfolio management has used similar reward structures. However, most existing RL studies optimize a single metric (e.g., Sharpe or cumulative return). Our approach generalizes this by linearly combining multiple performance criteria into one reward. This multi-component reward captures a richer set of financial objectives: maximizing returns, limiting downside risk, outperforming a benchmark (alpha), and achieving high risk-adjusted returns per unit of systematic risk. 3 Methodology We consider a portfolio of N assets. Let r_{i,t} be the return of asset i at time t , and let w_{i,t} be the corresponding portfolio weight chosen by the agent. The portfolio return at time t is R_{p,t}=\sum_{i=1}^{N}w_{i,t}\,r_{i,t}. (1) We focus on a trading horizon of T periods. Annualized Return R_{\mathrm{ann}}=\left(\prod_{t=1}^{T}(1+R_{p,t})\right)^{\frac{252}{T}}-1, (2) or approximately R_{\mathrm{ann}}\approx\frac{252}{T}\sum_{t=1}^{T}R_{p,t} . This term measures total performance, encouraging the agent to earn high returns. It generalizes the classic expected return term in Markowitz portfolio theory [ 3 ] . Downside Risk \sigma_{\mathrm{down}}=\sqrt{\frac{1}{T}\sum_{t=1}^{T}\max(0,-R_{p,t})^{2}}. (3) The downside deviation is a well-known risk metric used in the Sortino ratio [ 9 ] . By subtracting this in the reward, the agent is discouraged from incurring large losses. Differential Return D_{ret}=\frac{1}{\beta_{p}}\left(\mu_{p}-\mu_{b}\right)=\frac{1}{\beta_{p}T}\sum_{t=1}^{T}\bigl(R_{p,t}-R_{b,t}\bigr), (4) where \mu_{p} and \mu_{b} denote the average returns of the agent and benchmark, respectively. The term \beta_{p} is the portfolio beta, defined as \beta_{p}=\Cov(R_{p},R_{m})/\Var(R_{m}) , capturing the agent’s sensitivity to market fluctuations. This risk-adjusted measure, termed the differential return (DR) , represents the average outperformance of a portfolio relative to a benchmark, normalized by systematic (market) risk. It reflects a reward-per-unit-risk framework similar in spirit to the alpha from CAPM [ 6 ] , but explicitly accounts for the portfolio’s market exposure. Our formulation is a simplified differential return (SDR) derived from the foundational work by Simpson and formalized by Alam [ 1 ] . While Alam proposes more complex versions incorporating risk ratios and Sharpe-based scaling, our SDR focuses on systematic risk adjustment via beta only , avoiding explicit benchmark beta terms or volatility scaling. This maintains the key intention — rewarding portfolios that consistently outperform the benchmark on a risk-adjusted basis — while reducing sensitivity to unstable second-order statistics (like benchmark variance or cross-volatility) and non-convex functions that may hinder stable optimization in reinforcement learning. We later show the correctness of our formulation in our theoretical analysis (Section 4 ). The concept of differential return itself originates from Simpson’s work [ 8 ] , with subsequent refinements by Alam [ 1 ] , whose formulation informed our choice of SDR as a practical and principled performance metric. Treynor Ratio T_{\mathrm{ry}}\;=\;\frac{R_{\mathrm{ann}}-R_{f}}{\beta_{p}}\,, (5) where R_{\mathrm{ann}} is the portfolio’s annualized return, R_{f} is the risk-free rate, and \beta_{p} is the portfolio’s beta relative to the market return R_{m} . The Treynor ratio rewards returns achieved per unit of systematic risk: a well-diversified, low-beta portfolio that still earns strong returns will score highly on T_{\mathrm{ry}} . The use of the Treynor ratio as a foundational component in risk-adjusted performance measurement is supported by Alam [ 1 ] , who identifies it as one of the core metrics underlying differential return formulations. 3.1 Proposed Reward Function Putting these components together, our