Reinforcement Learning in Queue-Reactive Models: Application to Optimal Execution
Tomas Espana Yadh Hafsi Fabrizio Lillo Edoardo Vittori · 2025
Read Reinforcement Learning in Queue-Reactive Models: Application to Optimal Execution by Tomas Espana Yadh Hafsi Fabrizio Lillo Edoardo Vittori (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15262.txt.
Introduction
Tomas Espana, Yadh Hafsi, and two co-authors train a reinforcement learning agent to execute large trades inside a simulated order book. Optimal execution means dribbling a big order into the market slowly enough to avoid moving the price against yourself. Historical data cannot answer what-if questions about different pacing, so a simulator must supply the feedback.
The team uses a queue-reactive model, which reproduces how order flow responds and how price impact fades, to generate realistic book dynamics. A double deep Q-network agent, watching time, inventory, price, and depth, learns the pacing policy there. Its performance is checked against established execution benchmarks, testing whether model-free learning beats parametric rules.
Catalog record
| CREATOR | Tomas Espana Yadh Hafsi Fabrizio Lillo Edoardo Vittori |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-15262
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reinforcement Learning in Queue-Reactive Models: Application to Optimal Execution Tomas Espana Yadh Hafsi Fabrizio Lillo Edoardo Vittori Abstract We investigate the use of Reinforcement Learning for the optimal execution of meta-orders, where the objective is to execute incrementally large orders while minimizing implementation shortfall and market impact over an extended period of time. Departing from traditional parametric approaches to price dynamics and impact modeling, we adopt a model-free, data-driven framework. Since policy optimization requires counterfactual feedback that historical data cannot provide, we employ the Queue-Reactive Model to generate realistic and tractable limit order book simulations that encompass transient price impact, and nonlinear and dynamic order flow responses. Methodologically, we train a Double Deep Q-Network agent on a state space comprising time, inventory, price, and depth variables, and evaluate its performance against established benchmarks. Numerical simulation results show that the agent learns a policy that is both strategic and tactical, adapting effectively to order book conditions and outperforming standard approaches across multiple training configurations. These findings provide strong evidence that model-free Reinforcement Learning can yield adaptive and robust solutions to the optimal execution problem. Keywords : Optimal Execution, Reinforcement Learning, Queue-Reactive Model, Limit Order Book, Market Microstructure, Price impact. 1 Introduction Executing large orders efficiently is a fundamental challenge in electronic financial markets. Large transactions, typically initiated by institutional investors such as banks, asset managers, hedge funds, or proprietary trading firms, often consume visible liquidity across multiple price levels of the limit order book (LOB), generating significant price impact. This impact is manifested through both immediate price shifts and subsequent order flow reactions and motivates the need for execution strategies that optimally balance market impact against timing risk. Consequently, the design of optimal execution strategies has become a central topic in market microstructure and algorithmic trading research. Several studies have deepened the theoretical and empirical understanding of market impact. Refs [ 9 ] and [ 8 ] documented the transient and concave nature of impact, highlighting its origin in the interplay between order flow, liquidity consumption, and market participant behavior. These empirical findings motivated the introduction of new constraints and mechanisms in the optimal execution problem. The first formal treatment of the problem is due to [ 6 ] , who derive closed-form solutions under specific assumptions using dynamic programming. This framework was extended by [ 2 ] , who introduced both permanent and temporary market impact components, as well as a risk-aversion parameter. Subsequent research has generalized the Almgren-Chriss model along several dimensions (see [ 3 , 31 , 23 , 4 , 40 , 15 , 25 , 11 , 12 , 21 , 22 , 13 , 18 , 17 ] ). Notably, [ 24 ] established that transient impact models must satisfy a no-dynamic-arbitrage condition, which tightly links the functional form of impact decay to the underlying price dynamics. This result rules out many ad-hoc impact kernels and provides structural constraints on any admissible transient impact model. [ 40 ] further developed this direction by introducing a continuous-time propagator model in which impact decays through limit-order-book resilience. Although these developments significantly enriched our understanding of execution costs and impact mechanisms, they still rely on strong parametric and structural assumptions that might be in contrast with the true data generating process and might be difficult to calibrate empirically. Model parameters such as impact coefficients, volatility processes, and resilience rates remain highly context-dependent and vary across time and assets, limiting the robustness of these approaches. In fact, traditional methods for solving stochastic control problems face limitations. Closed-form solutions are rare and require restrictive assumptions on the model dynamics and objective function. Numerical approaches typically involve solving the Hamilton-Jacobi-Bellman (HJB) equation, quasi-variational inequalities, or backward stochastic differential equations (BSDEs). These formulations often rely on viscosity solution theory to establish existence, uniqueness, and regularity of the value function. Yet, the associated numerical schemes, such as Monte Carlo methods for BSDEs and finite-difference methods for PDEs, suffer from the curse of dimensionality. As a result, these techniques are generally limited to problems with low-dimensional state spaces. Related Works. To address these limitations, reinforcement learning methods [ 46 ] have emerged as data-driven and model-free alternatives for sequential decision problems. Unlike classical approaches, reinforcement learning (RL) does not require strong parametric assumptions on market dynamics. In the context of optimal execution, RL provides a flexible framework to learn trading policies directly from simulated or historical market data. The first application of RL to optimal execution was introduced by [ 38 ] , who trained a tabular Q-learning agent to minimize implementation shortfall using historical trade data. Building on this, [ 28 ] proposed a hybrid approach that combined the Almgren–Chriss (AC) model with Q-learning, allowing the agent to adjust its execution trajectory dynamically based on the observed market state. To overcome the limitations of tabular methods in high-dimensional settings, [ 39 ] employed Double Deep Q-Networks (DDQN) (see [ 35 , 48 ] ), integrating deep neural networks with Q-learning to generalize across continuous state spaces and mitigate value overestimation bias. More recently, [ 34 ] applied DDQN within the AC framework under time-varying liquidity conditions, demonstrating improved performance relative to benchmark strategies, while [ 33 ] employed Proximal Policy Optimization (PPO) (see [ 45 ] ) to train agents that learn directly from limit order book data using sparse reward structures. Existing approaches rely on specific market impact assumptions and on historical data, which do not capture how market conditions evolve in response to trading actions. It therefore remains unclear what reinforcement learning algorithms can learn in more realistic, microstructure-based settings that incorporate endogenous market impact. In this work, we address this question using a limit order book (LOB) simulator that models both direct market impact, from liquidity consumption, and indirect impact, from participants’ reactions to trades. Many market simulators have been proposed in the literature, including agent-based models [ 10 , 1 , 14 , 27 ] and generative models [ 32 , 19 , 20 , 37 , 36 ] . Recent studies have explored such simulation-based frameworks [ 26 , 16 ] . While these approaches can capture complex market behaviors, they can be more difficult to implement and calibrate, and, more importantly, they often lack analytical tractability. Main Contributions. We employ the Queue Reactive Model (QRM) (see [ 29 ] ). The QRM provides a realistic yet tractable description of short-term limit-order-book dynamics. Instead of assuming that price changes follow a simple diffusion, it treats the best bid and ask as queues of standing orders that evolve through limit-order arrivals, cancellations, and market-order executions. The rates at which these events occur depend on the current state of the book, particularly the size imbalance between the bid and ask queues, which means that the model naturally links order-flow pressure to short-term price movements. When either queue is depleted, the price moves by one tick, and new queue sizes are redrawn from empirical distributions calibrated to market data. This simple mechanism reproduces key microstructure features such as liquidity resilience, mean-reverting price behavior, and transient price impact: a trade that consumes liquidity temporarily shifts the state of the book, altering subsequent order flow and gradually decaying as liquidity replenishes. Because QRM