Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets

Shaocong Ma, Heng Huang · 2025

Read Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets by Shaocong Ma, Heng Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19950.txt.

Introduction

Shaocong Ma and Heng Huang make reinforcement learning for trading robust to the market impact the learner itself causes. Agents trained on historical data assume their trades move nothing, but live trading shifts prices, and that mismatch can wreck performance. Standard robust methods guard against worst cases inside symmetric uncertainty sets, which miss impact's directional character.

The authors introduce elliptic uncertainty sets shaped to reflect directional price pressure, and they derive closed-form worst-case solutions that keep robust policy evaluation efficient. On single-asset and multi-asset trading tasks the method earned superior Sharpe ratios and stayed robust as trade volumes grew. The approach aims to give financial reinforcement learning a more faithful model of the markets it will actually face.

Catalog record

CREATORShaocong Ma, Heng Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19950

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets Shaocong Ma, Heng Huang Abstract In financial applications, reinforcement learning (RL) agents are commonly trained on historical data, where their actions do not influence prices. However, during deployment, these agents trade in live markets where their own transactions can shift asset prices, a phenomenon known as market impact. This mismatch between training and deployment environments can significantly degrade performance. Traditional robust RL approaches address this model misspecification by optimizing the worst-case performance over a set of uncertainties, but typically rely on symmetric structures that fail to capture the directional nature of market impact. To address this issue, we develop a novel class of elliptic uncertainty sets. We establish both implicit and explicit closed-form solutions for the worst-case uncertainty under these sets, enabling efficient and tractable robust policy evaluation. Experiments on single-asset and multi-asset trading tasks demonstrate that our method achieves superior Sharpe ratio and remains robust under increasing trade volumes, offering a more faithful and scalable approach to RL in financial markets. 1 Introduction Reinforcement learning (RL) has emerged as a promising decision-making framework for quantitative trading strategies [ 1 , 2 ] , including portfolio optimization [ 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 ] , automatic trading [ 3 , 14 , 15 , 16 , 17 , 18 , 19 ] , market making [ 20 , 21 , 22 , 23 , 24 ] , and option hedging [ 25 , 26 , 27 , 28 , 29 , 30 ] . RL’s appeal in finance lies in its ability to learn adaptive strategies directly from data, without strong market assumptions [ 1 ] . This makes it well-suited for capturing complex market dynamics and aligning with the sequential nature of financial decision-making. One of the primary challenges in training a robust and consistently profitable RL agent lies in handling the market impact [ 31 , 32 , 33 ] ; that is, the influence of the agent’s own trades on asset prices in the deployed environment. For example, when a trader buys or sells a large volume of an asset, it can temporarily drive the price up or down, respectively (as illustrated in Figure 1 ). Typically, RL agents are trained on historical market data where market impact is absent. However, during deployment, the environment shifts from a passive historical setting to the real market, where the agent’s actions actively affect prices. This discrepancy between training and deployment environments undermines the optimality and robustness of the learned policy, and leads us to the central question in this paper: Q : Can we train RL agents on historical data while robustly accounting for market impact during deployment? To address this central question, we adopt the framework of robust RL [ 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 ] , which is designed to handle model misspecification ; that is, the mismatch between the training and deployment environments. The robust RL framework explicitly acknowledges that the real market environment may differ from the simulated training environment, and it aims to learn policies that are resilient under a range of perturbations. However, existing robust RL approaches face a key limitations in financial applications; that is, traditional uncertainty sets are typically symmetric, which fail to capture the directional nature of market impact. Addressing this challenge motivates our threefold contributions : (1) We propose a novel class of uncertainty sets, elliptic uncertainty sets ( Definition 3.2 ), which generalize traditional \ell_{p} -norm balls to the ellipse-like structure. These sets better capture the empirically observed directional nature of market impact as illustrated in Figure 1 . (2) On the theoretical side, we derive closed-form solutions for solving the worst-case transition kernel under the proposed uncertainty sets ( Theorem 3.4 ). Furthermore, under certain conditions, we present the explicit solutions ( Theorem 3.5 ). This development significantly broadens the scope of tractable robust RL problems beyond symmetric (ball-shaped) uncertainty sets, enabling more faithful representation of market impact. (3) We empirically evaluate our approach on real-world financial data using trade-level market impact simulations in Section 4 . Experimental results demonstrate that our method consistently outperforms the standard single-asset intra-day trading strategy and existing RL baselines in terms of Sharpe ratio. Moreover, we validate the robustness of our method under increasing strategy volume, confirming its effectiveness in high-volume regimes. Figure 1 : Market impact illustration using AMZN stock on June 21, 2012, based on 5-level LOBSTER data [ 53 ] . The left panel shows the price response to executing a buy order of 100 shares at the time t=0\ \text{ms} , which consumes ask-side liquidity within 1000\ \text{ms} and induces an immediate upward shift in price. The right panel shows the analogous impact of a sell order. This plot indicates that the transition dynamics induced by trading are not symmetrically distributed around the nominal kernel. 1.1 Related Work Existing Approaches to Handle Market Impact The most common approach for handling market impact is to simulate the electronic market more accurately. By applying high-fidelity market simulators, often built upon limit order book (LOB) dynamics [ 31 , 32 , 54 , 55 , 56 ] , trade-level data [ 57 , 58 ] , or large-scale agent-based simulators [ 59 , 60 , 61 , 62 ] , it captures more detailed market microstructure and reduces the gap between the simulated and the real trading environments. Prominent approaches to incorporating market impact into backtesting include agent-based simulation frameworks [ 59 , 60 , 61 , 62 ] , data-driven LOB reconstruction models [ 63 , 64 , 56 ] , and hybrid systems that integrate historical data replay with synthetic order flow generation [ 65 , 66 ] . However, access to high-quality market data is often limited, and simulating market environments with agent-based systems remains prohibitively expensive. Therefore, a practical alternative is to train the agent directly on historical data without market impact while still encouraging it to account for potential worst-case scenarios. Robust RL in Finance Robust RL, with its intrinsic ability to handle model misspecification, provides a natural framework for incorporating market impact considerations during training. Jaimungal et al. [ 3 ] propose a robust reinforcement learning framework based on rank-dependent utility to address uncertainty in financial decision-making, demonstrating the effectiveness of robust RL in portfolio allocation, benchmark strategy optimization, and statistical arbitrage. Shi et al. [ 5 ] formulate portfolio optimization as a robust RL problem to enhance resilience. We et al. [ 30 ] extend robust risk-aware RL to manage the risks associated with path-dependent financial derivatives, showcasing its effectiveness in complex hedging scenarios. However, existing work primarily focuses on fully symmetric uncertainty sets, which fail to capture the directional characteristics of financial markets. Addressing this limitation is the central focus of our paper. Modeling the Uncertainty Set in Robust RL The uncertainty set captures the discrepancy between training and deployment environments. However,

Read the complete text