LOB-Bench: Benchmarking Generative AI for Finance – an Application to Limit Order Book Data

Peer Nagy, Sascha Frey, Kang Li, Bidipta Sarkar, Svitlana Vyetrenko, Stefan Zohren, Anisoara Calinescu, Jakob Foerster · 2025

Read LOB-Bench: Benchmarking Generative AI for Finance – an Application to Limit Order Book Data by Peer Nagy, Sascha Frey, Kang Li, Bidipta Sarkar, Svitlana Vyetrenko, Stefan Zohren, Anisoara Calinescu, Jakob Foerster (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09172.txt.

Introduction

Nagy and co-authors release LOB-Bench, a benchmark for judging how realistic AI generated trading data looks. Limit order books record every bid and offer in a market, and models that fake them could aid research, but finance lacked agreed ways to score the fakes. Their 2025 framework compares made up and real message streams across many statistics.

The checks cover spreads, book volumes, order imbalance, and gaps between messages, plus scores from a trained discriminator. Market impact measures test whether events move prices in the data as they do in life. Several model families were tried, including state space generators, a GAN, and a classic parametric model. The autoregressive AI approach beat the traditional classes, the authors report.

Catalog record

CREATORPeer Nagy, Sascha Frey, Kang Li, Bidipta Sarkar, Svitlana Vyetrenko, Stefan Zohren, Anisoara Calinescu, Jakob Foerster
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-09172

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LOB-Bench: Benchmarking Generative AI for Finance – an Application to Limit Order Book Data Peer Nagy, Sascha Frey, Kang Li, Bidipta Sarkar, Svitlana Vyetrenko, Stefan Zohren, Anisoara Calinescu, Jakob Foerster Abstract While financial data presents one of the most challenging and interesting sequence modelling tasks due to high noise, heavy tails, and strategic interactions, progress in this area has been hindered by the lack of consensus on quantitative evaluation paradigms. To address this, we present LOB-Bench , a benchmark, implemented in python, designed to evaluate the quality and realism of generative message-by-order data for limit order books (LOB) in the LOBSTER format. Our framework measures distributional differences in conditional and unconditional statistics between generated and real LOB data, supporting flexible multivariate statistical evaluation. The benchmark also includes commonly used LOB statistics such as spread, order book volumes, order imbalance, and message inter-arrival times, along with scores from a trained discriminator network. Lastly, LOB-Bench contains “market impact metrics”, i.e. the cross-correlations and price response functions for specific events in the data. We benchmark generative autoregressive state-space models, a (C)GAN, as well as a parametric LOB model, and find that the autoregressive GenAI approach beats traditional model classes. Code and generated data are available at: https://lobbench.github.io/ . † † affiliationnotice: Equal contribution 1 Introduction Figure 1 : Schematic of LOB-Bench methodology for conditional distributional evaluation Practitioners have long been interested in high-quality synthetic financial data, which is especially difficult in the high-frequency domain due to high noise, heavy tails, and multi-agent interactions. Generative AI (GenAI) is currently revolutionizing different fields, ranging from natural language processing to image generation and real world applications. Perhaps surprisingly, the backbone of these methods is simply self-supervised pre-training on large datasets using a next-token prediction loss on autoregressive sequence models ( Nie et al., 2024 ; Dubey & et. al., 2024 ; Liu et al., 2024 ) . Recently, Nagy et al. (2023) applied this paradigm to limit order books (LOB), the mechanism through which stock markets keep track of buy and sell orders to determine any-time prices. Specifically, in contrast to prior works, which model only high level features ( Cont et al., 2010 ; Coletta et al., 2022 ; Byrd et al., 2020 ) , this approach learns a token-level distribution over messages in the LOBSTER dataset ( Huang & Polak, 2011 ) . An accurate , low level generative model of the financial system is extremely valuable from a societal and commercial point of view. For example, it could unlock better mechanism design, stability analysis, or learned-algorithms (e.g. order execution ( Frey et al., 2023 ) ) by providing counterfactuals. A key question then is how to determine the realism and trustworthiness of GenAI, and of other generative financial models. On the one hand, for high-level approaches and “old school” agent-based modelling ( Byrd et al., 2020 ; Chiarella & Iori, 2002 ; Paulin, 2019 ; Llacay & Peffer, 2018 ) the evaluation is usually based on a qualitative analysis of whether the model reproduces known high-level patterns – “stylized facts” from the literature, “impact” or the famous “square-root law” ( Tóth et al., 2016 ; Brokmann et al., 2015 ; Almgren et al., 2005b ) . However, most of these metrics are unquantifiable and may be disconnected from ground-truth data. Figure 2 : Schematic of the LOB. Not immediately executable orders are placed in First-In, First-Out (FIFO) queues at the specified limit price. Sell limit orders are placed on the Ask side (red) and Buy limit orders on the Bid side (blue). On the other hand, for GenAI the standard evaluation for pre-training is simply cross-entropy , i.e. how closely the model is able to predict the next token on held-out data. Unfortunately, this does not capture how the model performs under autoregressive sampling, when generating sequences of data one token at a time, where error accumulation can cause distribution shifts. In many applications of GenAI this is not a problem, since the pre-trained models are merely used as starting points for task specific finetuning (e.g. RLHF), rather than in their “bare” form. In contrast, we want to evaluate the pre-trained models in the sampling regime to unlock the mentioned use-cases. To address this, we propose a general framework for evaluating the similarity between the distribution induced by generative LOB models and the ground-truth data. At a high level, our unconditional evaluation consists of three steps. We first introduce a set of aggregator functions , \bm{\Phi} , which map from high-dimensional time series LOB data into a set of 1d subspaces. Secondly, we compute histograms to estimate distributions for the ground-truth and generated data in these subspaces and, finally, use a distance metric, e.g. L_{1} , to compare differences in these estimates. Some of the aggregator functions chosen are closely inspired by metrics used in literature ( Vyetrenko et al., 2021 ; Paulin, 2019 ; Chiarella & Iori, 2002 ; Cont, 2001 ) . They also directly relate to generative adversarial networks ( Goodfellow et al., 2014 ) , where the discriminator network is equivalent to a worst-case aggregator function for a given generator. For conditional distributional evaluation, we first apply an aggregator function and group these results into “buckets” based on the conditioning variable. We then score each of the resulting conditional distributions using the process described earlier. This approach enables, for example, assessing whether the distribution of bid-ask spreads, conditioned on the time of day, aligns with the corresponding conditional distribution in real data. To derive a single metric, we compute the average loss across the conditioning buckets, weighted by the probability of each bucket. Furthermore, we can also use this to evaluate model-drift by aggregating on the sampling step and comparing to the unconditional data, which is a good proxy for model-derailment in open-loop sampling. See Figure 1 for a process schematic. We test our evaluation framework on five different generative models: four modern GenAI models ( Coletta et al., 2022 ; Nagy et al., 2023 ; Peng et al., 2023 ; Peng et al., 2024 ) and a widely-used classic model as a baseline ( Cont et al., 2010 ) . All models are tested on data of Alphabet Inc (GOOG) and Intel Corporation (INTC) stock. We don’t present detailed results for the Coletta model trained on INTC, because the architecture was developed only for small-tick stocks and therefore fails on INTC data ( Coletta et al., 2022 ) . We find evidence of “model derailment,” since the distance scores increase for longer unrolls ( Figure 5 ). We also find that the LOBS5 model is best able to reproduce the standard price-impact curves that are well-known in the economics and finance literature ( Eisler et al., 2012 ) . Our contributions are summarized as follows: A novel LOB benchmark for distributional evaluation: the first LOB benchmark focused on full distributional quantification of model performance. This addresses limitations of prior work, which relied on qualitative comparisons of stylized facts, making rigorous model comparisons infeasible and hindering research progress. Interpretable scoring functions for targeted improvements: using intuitive scoring functions enables targeted model development and refinement. Difficult challenge of discriminator scores: discriminator-based scoring sets a high bar for future generative models,

Read the complete text