fev-bench : A Realistic Benchmark for Time Series Forecasting
Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen, Lorenzo Stella, Nick Erickson, Pablo Guerron, Michael Bohlke-Schneider, Yuyang Wang · 2025
Read fev-bench : A Realistic Benchmark for Time Series Forecasting by Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen, Lorenzo Stella, Nick Erickson, Pablo Guerron, Michael Bohlke-Schneider, Yuyang Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-26468.txt.
Introduction
Oleksandr Shchur, Abdul Fatir Ansari, and colleagues present a benchmark for time series forecasting in a 2025 study. Pretrained forecasting models need fair tests, yet older benchmarks cover few domains and ignore settings where side information, called covariates, is available. Their scoring also often lacks statistical care, so small gaps may be noise dressed as progress.
The new benchmark stresses realistic tasks across broader domains, including covariate cases. Its aggregation uses firmer statistical methods to judge whether differences are real. Consistent evaluation code fits existing pipelines without forcing rigid setups. Better benchmarks keep model claims honest as foundation models spread into forecasting.
Catalog record
| CREATOR | Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen, Lorenzo Stella, Nick Erickson, Pablo Guerron, Michael Bohlke-Schneider, Yuyang Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-26468
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
fev-bench : A Realistic Benchmark for Time Series Forecasting Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen, Lorenzo Stella, Nick Erickson, Pablo Guerron, Michael Bohlke-Schneider, Yuyang Wang Abstract Benchmark quality is critical for meaningful evaluation and sustained progress in time series forecasting, particularly given the recent rise of pretrained models. Existing benchmarks often have narrow domain coverage or overlook important real-world settings, such as tasks with covariates. Additionally, their aggregation procedures often lack statistical rigor, making it unclear whether observed performance differences reflect true improvements or random variation. Many benchmarks also fail to provide infrastructure for consistent evaluation or are too rigid to integrate into existing pipelines. To address these gaps, we propose fev-bench , a benchmark comprising 100 forecasting tasks across seven domains, including 46 tasks with covariates. Supporting the benchmark, we introduce fev , a lightweight Python library for benchmarking forecasting models that emphasizes reproducibility and seamless integration with existing workflows. Using fev , fev-bench employs principled aggregation methods with bootstrapped confidence intervals to report model performance along two complementary dimensions: win rates and skill scores. We report results on fev-bench for various pretrained, statistical and baseline models, and identify promising directions for future research. † † affiliationnotice: Equal contribution 1 Introduction Pretrained time series forecasting models are transforming forecasting practice. They often deliver more accurate forecasts than traditional methods ( Aksu et al. 2024 ) , while enabling zero-shot inference that simplifies production use and lowers barrier to entry for practitioners ( Cohen et al. 2025 ) . Advances in pretrained forecasting models are largely assessed empirically on benchmarks, making benchmark quality essential for continued progress. Oversights in forecasting benchmarks propagate directly to the model development. For example, most general forecasting benchmarks completely ignore covariates despite their prevalence in real-world applications ( Bojer & Meldgaard 2021 ; Arango et al. 2025 ) . As a result, the majority of pretrained models lack covariate support, limiting their effectiveness in domains like retail where promotional data and pricing information are essential for accurate demand forecasting ( Fildes et al. 2022 ) . Beyond task coverage, existing benchmarks often lack statistical rigor. Most studies report single-number summaries, leaving it unclear whether improvements reflect true advances or random variation. Small gains may vanish or even reverse with minor benchmark changes ( Roque et al. 2025 ) . This undermines the reliability of conclusions and can mislead researchers and practitioners about which models perform better. Finally, benchmark infrastructure presents additional barriers to progress and reproducibility. Many benchmarks provide only standalone datasets without evaluation code, leading to inconsistent implementations across studies that make results incomparable ( Hewamalage et al. 2023 ) . When infrastructure exists, it often consists of monolithic systems bundling models, datasets, and evaluation logic with extensive dependencies that become unmaintainable over time. These rigid systems prevent researchers from extending benchmarks to new domains or integrating evaluation into existing workflows, limiting their practical utility and lifespan. To address these challenges, we make the following three contributions: • New benchmark. We introduce fev-bench , a forecast evaluation benchmark containing 100 tasks spanning 7 real-world application domains. Our benchmark addresses a key gap in existing work by including 46 tasks with covariates alongside both univariate and multivariate forecasting scenarios, better reflecting real-world forecasting use cases. • Aggregation methods. In our benchmark, we employ principled aggregation strategies including bootstrap-based confidence intervals that quantify whether performance differences are statistically meaningful. This approach enables more reliable model comparisons and assesses the robustness of conclusions to variations in benchmark composition. • Evaluation package. We introduce fev 1 1 1 github.com/autogluon/fev , a lightweight Python package for forecasting evaluation that introduces minimal dependencies while remaining compatible with popular forecasting libraries. The package focuses on reproducibility and extensibility, enabling researchers to easily build and share new benchmarks and the corresponding results. Benchmark # datasets # tasks # tasks with covariates # multivariate tasks Forecast type Monash ( Godahewa et al. 2021 ) 42 42 0 0 point LTSF ( Zeng et al. 2023 ) 9 36 0 9 point TFB ( Qiu et al. 2024 ) 41 116 0 25 point BasicTS+ ( Shao et al. 2024 ) 20 40 0 20 point ProbTS ( Zhang et al. 2024 ) 18 18 0 14 point & quantile Chronos BM2 ( Ansari et al. 2024 ) 27 27 0 0 point & quantile GIFT-Eval ( Aksu et al. 2024 ) 55 97 0 43 point & quantile fev-bench (this work) 96 100 46 35 point & quantile Table 1: Overview of general time series forecasting benchmarks. The fev benchmark contains more unique datasets than existing benchmarks and includes 46 tasks with covariates, addressing a gap in current evaluation frameworks. 2 Preliminaries Problem definition. The multivariate time series forecasting problem can be formally stated as follows. We are given a collection \{{\bm{y}}_{n,1:T}\}_{n=1}^{N} of N multivariate time series. For n=1,\dots,N and t=1,\dots,T , let {\bm{y}}_{n,t}=(y_{n,d,t})_{d=1}^{D}\in\mathbb{R}^{D} denote the D -dimensional observation vector for series n at time t , where the special case D=1 corresponds to univariate forecasting. The goal of multivariate forecasting is to predict the future H values {\bm{y}}_{n,T+1:T+H} for each series n , where H is the forecast horizon. Each time series may be accompanied by covariates \mathbf{X}_{n,1:T+H} , which include (i) static covariates that do not vary over time (e.g., item or location identifiers), (ii) past-only dynamic covariates observed up to time T (e.g., past measurements of related variables), and (iii) known dynamic covariates available for all time steps 1,\dots,T+H (e.g., holiday indicators or planned interventions). In the most general form, the aim is to model the conditional distribution \displaystyle p\big({\bm{y}}_{n,T+1:T+H}\mid{\bm{y}}_{n,1:T},\mathbf{X}_{n,1:T+H}\big). (1) While this full distributional modeling provides the richest information, it is common in practice to produce point forecasts like the conditional mean or median of each y_{n,d,t} . Alternatively, it is often sufficient to estimate predictive quantiles of the conditional distributions p(y_{n,d,t}\mid{\bm{y}}_{n,1:T},\mathbf{X}_{n,1:T+H}) . We denote by {\mathcal{Q}}\subset(0,1) the set of quantile levels of interest and aim to produce forecasts \hat{y}^{(q)}_{n,d,t} such that \Pr(y_{n,d,t}\leq\hat{y}^{(q)}_{n,d,t})=q for all q\in{\mathcal{Q}} . Benchmarks. To evaluate forecasting methods systematically, the general problem must be instantiated as concrete tasks and combined with an evaluation protocol. We refer to such a collection of tasks together with their evaluation and aggregation procedure as a benchmark . Tasks. A task specifies a concrete forecasting problem. It consists of a dataset together with all parameters that define how forecasts are produced and evaluated, including the forecast horizon H , the evaluation cutoff dates, which columns serve as targets or covariates, and the evaluation metric. A single dataset can yield multiple distinct tasks by varying these parameters. The choice of evaluation metric is integral to task definition because different metrics can correspond to different optimal forecasts; combining conflicting metrics within a single task creates ambiguity about the intended goal ( Kolassa 2020 ) . Rolling evaluation. Each task is evaluated using a rolling-origin evaluation protocol with W windows ( Hyndman & Athanasopoulos 2018 ) . Let \tau_{1}<\tau_{2}<\dots<\tau_{W} denote the evaluation cutoffs. At each window w\in\{1,\dots,W\} , the model receives all observations up to \tau_{w} as input and is asked to produce H -step forecasts. Advancing the cutoff creates a sequence of forecast–target pairs, so that a single task produces W