STEB: In Search of the Best Evaluation Approach for Synthetic Time Series
Michael Stenger, Robert Leppich, André Bauer, Samuel Kounev · 2025
Read STEB: In Search of the Best Evaluation Approach for Synthetic Time Series by Michael Stenger, Robert Leppich, André Bauer, Samuel Kounev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21160.txt.
Introduction
Michael Stenger, Samuel Kounev and their co-authors present STEB, a benchmark for judging measures of synthetic time series in 2025. Fake time series help with data growth and privacy, but comparing the many scoring measures for them has been an open problem.
STEB tests measures in an automated way on diverse data sets, with noise added and many data transforms applied. It tracks how reliable and steady each score is, plus run time and errors. In their tests, the authors ranked dozens of measures from past work. They found that the embedding chosen before scoring had a large effect on the final score.
Catalog record
| CREATOR | Michael Stenger, Robert Leppich, André Bauer, Samuel Kounev |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 100 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21160
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
STEB: In Search of the Best Evaluation Approach for Synthetic Time Series Michael Stenger, Robert Leppich, André Bauer, Samuel Kounev Abstract The growing need for synthetic time series, due to data augmentation or privacy regulations, has led to numerous generative models, frameworks, and evaluation measures alike. Objectively comparing these measures on a large scale remains an open challenge. We propose the S ynthetic T ime series E valuation B enchmark (STEB)—the first benchmark framework that enables comprehensive and interpretable automated comparisons of synthetic time series evaluation measures. Using 10 diverse datasets, randomness injection, and 13 configurable data transformations, STEB computes indicators for measure reliability and score consistency. It tracks running time, test errors, and features sequential and parallel modes of operation. In our experiments, we determine a ranking of 41 measures from literature and confirm that the choice of upstream time series embedding heavily impacts the final score. 1 Introduction Time series (TS) data is central to many domains and applications such as forecasting medical data [ 22 ] or human activity recognition [ 17 ] . Yet too often there is a lack of availability, quantity, and quality of data, for instance, due to privacy concerns. Data synthesis can be a solution, but it remains challenging in practice [ 14 , 8 ] . A key step towards high-quality synthetic data is a comprehensive, reliable evaluation strategy. The fundamental problem is the lack of ground truth data similar to unsupervised clustering. To handle this complex, indirect assessment, the common approach is to use different measures to quantify various quality aspects [ 60 ] . However, with dozens of measures having been proposed in recent years, the selection of the “ideal” set of measures is particularly challenging [ 52 ] . For instance, seminal works [ 60 , 59 , 49 ] and recently proposed frameworks [ 45 , 41 ] use different combinations with little overlap. Furthermore, the aspects of synthetic data quality considered typically differ [ 51 ] . As a consequence, comparability of generative performance is hindered, the state-of-the-art unknown. Previous works studied small groups of measures in a tailored analysis or developed frameworks to generate and evaluate synthetic data. However, to the best of our knowledge, there is no detailed, objective, and broad study on the effectiveness of these measures applied to TS. Critically, this raises serious concerns regarding the reliability of past evaluation results for TS generative models. Hence, we propose S ynthetic T ime series E valuation B enchmark (STEB), the first benchmark framework to conduct large-scale and multi-faceted analysis of synthetic TS evaluation measures. With STEB, we aim to narrow down the clutter of measures to a standardized set, drastically increasing the comparability of generative performance. Scope. We focus on uni-/multivariate, real-valued TS. An evaluation measure is a function m:\mathcal{P}(\mathcal{X})\times\mathcal{P}(\mathcal{X})\rightarrow\mathbb{R} where \mathcal{P} is the super-set operator and \mathcal{X} the data space, e.g., time series. For a real dataset D_{r}\subset\mathcal{X} and synthetic dataset D_{s}\subset\mathcal{X} , we call s=m(D_{r},D_{s}) its score. In this context, “real” data is the initial data to learn. Note that m assesses the generated data, not generators. Hence, we do not consider generator-dependent measures such as duality gap [ 50 ] . Similarly, we limit this study to quantitative measures with clear s and exclude, for instance, visualizations such as the popular t-SNE plot [ 60 ] . Contributions. With this work, we contribute to synthetic data research in three ways: 1. We design and implement STEB, a novel benchmark framework for comprehensive, interpretable, and automated analysis of quantitative synthetic TS measures. 2. We analyze 41 measures with respect to reliability, consistency, and running time. More specifically, we rank the measures in four aspects of quality and in running time. 3. We investigate the impact of the upstream TS embedding on the final score. This paper is structured as follows. Section 2 puts our work in the context of related work. Section 3 introduces STEB and Section 4 the experiments on TS synthesis measures. Section 5 presents and discusses the results. Section 6 concludes the paper. 2 Related Work With the increasing interest in synthetic data generation, different studies on evaluation measures were conducted. One noteworthy work examines three of the then most commonly used evaluation measures for image generation [ 53 ] . Their combined theoretical and empirical approach is tailored towards each measure. Key findings are that the behavior of different measures is often largely independent of each other and synthetic data utility is application specific. Lucic et al. [ 34 ] presented an empirical study on GAN models and evaluation measures for image synthesis. Their focus is on the Fréchet inception distance (FID), analyzing bias, variance, robustness to mode dropping, different image embeddings, and FID value range. In terms of experimental design, the closest work we know of is by Huang et al. [ 19 ] . They manipulate image data in different ways to study the behavior and efficiency of five evaluation measures w.r.t. overfitting, mode collapse/dropping, discriminability, and robustness. Recently, Ismail-Fawaz et al. [ 20 ] conducted a comparison of eight measures for the evaluation of human motion generation and proposed a ninth. They analyzed each measure qualitatively, followed by a quantitative comparison using a conditional generator on one real dataset to controllably create human motions. STEB differs from the above works in many ways: (i) While these work are all (very) focused in their analysis, we compare a wide range of measures of diverse designs and purposes; (ii) STEB is designed for TS, whereas previous works target images and human motion; (iii) STEB is model agnostic with regard to the data generator and evaluation measure, unlike Lucic et al. [ 34 ] and Huang et al. [ 19 ] , who focus on non-conditional GANs, or Theis et al. [ 53 ] , who employ analysis specific to the measures; (iv) We utilize different hand-crafted data manipulations, while Ismail-Fawaz et al. [ 20 ] use a neural model to create the test synthetic data; (v) Most of the related work does not capture recent developments. STEB incorporates established and recent measures and can be extended to support further analysis techniques and future measures, continuously tracking the state-of-the-art; (vi) STEB is more fine-grained in its analysis, as it differentiates four aspects of synthesis quality, and it is more comprehensive in terms of experimental parameters. There are three related synthetic data benchmarks. Synthcity is a framework for benchmarking tabular, image, and TS data generators [ 45 ] . It incorporates multiple generators, evaluation measures, and datasets in an automated test pipeline. Similarly, TSGBench [ 2 ] and Time Series Generative Modeling (TSGM) [ 41 ] are frameworks for time series synthesis. While TSGBench specializes on benchmarking generators, TSGM is presented as a more general solution including application cases. These benchmarks mainly differ in the choice of integrated measures, generators, and datasets. They evaluate the respective generative models and accept the implemented measures as given. STEB, however, assesses the measures themselves in order to determine which to best include in tools such as Synthcity, TSGBench, or TSGM. The diversity of evaluation suites highlights the need for a systematic analysis tool for measures. 3 STEB: Synthetic Time Series Evaluation