JaxMARL-HFT: GPU-Accelerated Large-Scale Multi-Agent Reinforcement Learning for High-Frequency Trading
Valentin Mohl, Sascha Frey, Reuben Leyland, Kang Li, George Nigmatulin, Mihai Cucuringu, Stefan Zohren, Jakob Foerster, Anisoara Calinescu · 2025
Read JaxMARL-HFT: GPU-Accelerated Large-Scale Multi-Agent Reinforcement Learning for High-Frequency Trading by Valentin Mohl, Sascha Frey, Reuben Leyland, Kang Li, George Nigmatulin, Mihai Cucuringu, Stefan Zohren, Jakob Foerster, Anisoara Calinescu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02136.txt.
Introduction
Valentin Mohl and eight co-authors release an open-source simulator that makes multi-agent reinforcement learning practical for high-frequency trading research. Agent-based market models are hard to calibrate, and learning-based agents are more realistic but painfully slow to train. Their environment, JaxMARL-HFT, runs on graphics processors and works directly from market-by-order data, the full stream of individual orders.
Built on the JaxMARL framework and a limit order book implementation in JAX, it supports mixed agent types with different observations, actions, and rewards, and can also serve single-agent or fixed-policy studies. Training ran up to 240 times faster than reference implementations on the same hardware. In a demonstration with a year of data covering 400 million orders, execution and market-making agents learned to beat standard benchmarks.
Catalog record
| CREATOR | Valentin Mohl, Sascha Frey, Reuben Leyland, Kang Li, George Nigmatulin, Mihai Cucuringu, Stefan Zohren, Jakob Foerster, Anisoara Calinescu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-02136
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
JaxMARL-HFT: GPU-Accelerated Large-Scale Multi-Agent Reinforcement Learning for High-Frequency Trading Valentin Mohl, Sascha Frey, Reuben Leyland, Kang Li, George Nigmatulin, Mihai Cucuringu, Stefan Zohren, Jakob Foerster, Anisoara Calinescu Abstract . Agent-based modelling (ABM) approaches for high-frequency financial markets are difficult to calibrate and validate, partly due to the large parameter space created by defining fixed agent policies. Multi-agent reinforcement learning (MARL) enables more realistic agent behaviour and reduces the number of free parameters, but the heavy computational cost has so far limited research efforts. To address this, we introduce JaxMARL-HFT ( JAX -based M ulti- A gent R einforcement L earning for H igh- F requency T rading ), the first GPU-accelerated open-source multi-agent reinforcement learning environment for high-frequency trading (HFT) on market-by-order (MBO) data. Extending the JaxMARL framework and building on the JAX-LOB implementation, JaxMARL-HFT is designed to handle a heterogeneous set of agents, enabling diverse observation/action spaces and reward functions. It is designed flexibly, so it can also be used for single-agent RL, or extended to act as an ABM with fixed-policy agents. Leveraging JAX enables up to a 240x reduction in end-to-end training time, compared with state-of-the-art reference implementations on the same hardware. This significant speed-up makes it feasible to exploit the large, granular datasets available in high-frequency trading, and to perform the extensive hyperparameter sweeps required for robust and efficient MARL research in trading. We demonstrate the use of JaxMARL-HFT with independent Proximal Policy Optimization (IPPO) for a two-player environment, with an order execution and a market making agent, using one year of LOB data (400 million orders), and show that these agents learn to outperform standard benchmarks. The code for the JaxMARL-HFT framework is available on GitHub 1 1 1 https://github.com/vmohl/JaxMARL-HFT . 1. Introduction Modelling financial markets at high frequency is an inherently difficult problem, as prices emerge endogenously from the interactions of millions of market participants, often acting with bounded rationality and imperfect information. Agent-based modelling (ABM) addresses this by specifying individual agents and allowing the aggregate market behaviour to arise implicitly from their interactions at the micro-level. In most financial contexts, this amounts to simulating behaviour at the most granular level, i.e. orders submitted to the limit order book (LOB), which is the collection of unmatched orders submitted by market participants. However, as such models are mostly based on pre-defined agent policies, previous approaches have been criticised for having too simplistic agent strategies and emergent behaviour. Reinforcement Learning (RL) has been proposed as a potential solution ( Frey et al. 2023 ; Ardon et al. 2022 ; Spooner et al. 2018 ) , with most recent work focusing on single-agent RL. To build realistic and robust ABMs with intelligent agents – which learn from each other – we require multi-agent reinforcement learning (MARL) which aims to learn independent policies for a heterogeneous agent population. A natural first step is to successively train RL agents in the single-agent setting, progressively adding their learned policies to the pool of opponent strategies, a form of self-play with differing policy objectives ( Mascioli et al. 2024 ) . To allow for learning in an adversarial context, classical MARL has multiple agents independently collecting experiences and simultaneously updating their policies. To the best of our knowledge, there is no open-source framework for this in an HFT context. One reason for this is the computational restrictions and training instabilities common to MARL experiments, exacerbated by the very low signal-to-noise ratio in financial applications. To overcome these challenges, we require a highly efficient environment to enable large-scale MARL experiments. We therefore present JaxMARL-HFT, the first GPU-enabled MARL environment for HFT on the LOB. Leveraging the power of JAX ( Bradbury et al. 2018 ) and extending the parallelisation of JAX-LOB ( Frey et al. 2023 ) to the multi-agent setting, JaxMARL-HFT is able to achieve up to a 240x reduction in end-to-end training times compared to state-of-the-art implementations on the same hardware. This speed-up allows for extensive hyperparameter sweeps and the processing of tens of thousands of parallel environments, thus unlocking the potential of rich market-by-order (MBO) data sets, such as LOBSTER ( Huang and Polak 2011 ) . The environment is based on the JaxMARL framework ( Rutherford et al. 2024 ) , to allow plug-and-play interactions with state-of-the-art MARL algorithms. It opens up HFT at the most granular level as a complex testbed for evaluating the limits of MARL algorithms. In Section 2 we introduce the core concepts used in this paper, and in Section 3 we present an overview of the existing frameworks. Section 4 presents the design of our implementation, and an overview of the implemented agent archetypes (Section 4.1 ) and the changes made to the JaxMARL reference implementation of IPPO (Section 4.2 ). Results for the increased throughput compared to state-of-the-art implementations are discussed in Section 5.1 . Section 5.2 reports training results and evaluations against baselines of a simple two-player setup. In Section 6 , we conclude with a summary of our contributions, and future work. Our main contributions are summarised as follows: (1) Extension of the JAX-LOB ( Frey et al. 2023 ) environment to the multi-agent setting, allowing for heterogeneous agents . (2) Compatibility with, and extension of, JaxMARL ( Rutherford et al. 2024 ) algorithm implementations. (3) Major throughput increase and memory optimisation allowing for training with multiple years of LOBSTER ( Huang and Polak 2011 ) market-by-order data. (4) The code for the JaxMARL-HFT framework is available on GitHub 2 2 2 https://github.com/vmohl/JaxMARL-HFT . (5) A challenging environment with real-world applications to test the limits of MARL algorithm development. (6) Preliminary results showing that, under certain conditions, policies can be learned to outperform baselines, though they require carefully crafted action spaces and reward functions. 2. Background 2.1. High-Frequency Trading Limit order books (LOBs) ( Gould et al. 2013 ; Bouchaud et al. 2018 ) are the primary mechanism through which modern financial exchanges operate. Market participants can submit limit orders to express their intention to buy or sell a particular asset at a given price and quantity. As continuous double auction markets, LOBs collect buy and sell limit orders from market participants based on price and time priority, and match compatible orders. Populations of market participants sharing objectives are often classified by trading task. Despite significant intra-class heterogeneity, the formulations of market making, order execution and directional trading prove useful in understanding general population characteristics ( Gould et al. 2013 ; Bouchaud et al. 2018 ) . These categories provide mathematical frameworks from which control solutions have been designed. Market making refers to participants who provide liquidity to an exchange by quoting bid and ask prices, aiming to profit from the spread whilst minimising risks associated with accumulating an inventory of securities. Order execution is a task performed by participants looking to buy or sell a specified quantity of a security over a pre-determined period. The participants in this task aim to minimise the costs associated with this transaction. Directional trading refers to the general case where a participant aims to profit from short-term price movements by buying and selling a security without the explicit aim for liquidity provision as in the market making task. 2.2. Multi-Agent Reinforcement Learning (MARL)