Model selection with proper scoring rules on data sets of time series: prefer the mean scaled score.

Giorgio Corani, Stefano Damato, Dario Azzimonti, Lorenzo Zambon · 2026

Read Model selection with proper scoring rules on data sets of time series: prefer the mean scaled score. by Giorgio Corani, Stefano Damato, Dario Azzimonti, Lorenzo Zambon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24715.txt.

Introduction

Giorgio Corani and three colleagues ask how to pick the best forecasting model across many time series. Each model earns an average proper score on one series. Combining scores across a dataset is the hard part. Workers either scale and average them, or use ranks and win rates. These choices can point to different winners. The authors trace the conflict to skewness in the scores, worst when test sets are short.

Corani and colleagues show that rank, median, and win rate rules can pick a misspecified model. The plain mean scaled score does not suffer from this flaw. As test sets grow, all the rules converge on the same decision anyway. Experiments on intermittent demand data, including the M5 competition, stress the need for large test sets. The mean scaled score also stayed consistent across scaling factors. The advice, reflected in the title, is to prefer that mean.

Catalog record

CREATORGiorgio Corani, Stefano Damato, Dario Azzimonti, Lorenzo Zambon
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24715

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Model selection with proper scoring rules on data sets of time series: prefer the mean scaled score. Giorgio Corani, Stefano Damato, Dario Azzimonti, Lorenzo Zambon Abstract We study the problem of model selection among probabilistic forecasting models evaluated on datasets of multiple time series. The performance of a model on a single time series is quantified by the average value ( score ) of a proper scoring rule over a test set, but extending model selection to data sets of time series requires aggregating these scores. Common approaches either rely on scaling scores and averaging them (mean scaled score) or avoid scaling by using alternative statistics such as mean ranks or win rates. However, these approaches can yield conflicting conclusions. We show that such discrepancies arise from the skewness of the distribution of the scores, which is particularly pronounced when test sets are short. The skewness can cause non-mean criteria (e.g., mean rank, median, win rate) to select misspecified models. In contrast, the mean score is immune from this problem. We further show that, as the size of the test sets increases, all aggregation criteria converge to the same model selection decision, mitigating these discrepancies. Our experiments on intermittent demand time series, including data from the M5 competition, highlight the importance of sufficiently large test sets; the mean scaled score appears to be the more reliable approach, also because empirically we found its decision to remain consistent when different scaling factors are adopted. 1 Introduction Scoring rules ( 11 ; 10 ) quantify how good a probabilistic forecast is. A univariate scoring rule \mathcal{S} is a function \mathcal{S}(P,y) which assigns a numerical score to a probability distribution P , given the actual observation y\in\mathbb{R} . We consider negatively oriented scoring rules, i.e., better forecasts corresponds to lower scores. A scoring rule is proper if it is minimized in expectation by the true distribution, i.e., if \mathbb{E}_{y\sim Q}\left[\mathcal{S}(Q,y)\right]\leq\mathbb{E}_{y\sim Q}\left[\mathcal{S}(P,y)\right] for any valid forecast distributions P,Q . The most common proper scoring rules are the continuous ranked probability score (CRPS), the quantile score (QS), the interval score (IS), and the energy score (ES). In this paper, the test set refers to a set of n_{te} out-of-sample data points, related to different rolling origins and/or forecast horizons; the pointwise score is the scoring rule evaluated on a single data point, i.e. \mathcal{S}(P,y_{i}) for i\in\{1,\ldots,n_{te}\} , and the score is the average value of the scoring rule on the test set, i.e. \frac{1}{n_{te}}\sum_{i=1}^{n_{te}}\mathcal{S}(P,y_{i}) . Model selection on a single time series is straightforward: we select the model with the lowest score, assuming n_{te} to be large enough for the score to approximate the expected value. We can instead adopt different approaches for model selection on a data set containing n time series: the most common approaches are the mean scaled score and the mean rank . In the first case, we scale the n scores to make them comparable; then we average them and we select the model with the lowest mean scaled score . While the need for scaling is established ( 2 ) , the choice of the scaling factor is not obvious; most scaling factors ( 13 ; 19 ; 1 ; 22 ) are related to the scale of the time series or to the scale of the error of a benchmark model. Generally however no scaling factor is adequate for all the non-stationarities and non-normalities which might be present in the different time series of a data set ( 12 ) . The perceived risk is thus that the outcome of model selection might depend on the chosen scaling factor; however, to our knowledge, there is yet no study on this point. The mean rank approach is appealing as it avoids the choice of the scaling factor. It ranks the models according to their score on the test set; then it averages the ranks over time series and selects the model with lowest mean rank. The significance of the differences between the mean ranks is tested with a Friedman test ( 7 ) with Nemenyi post-hoc; this is also referred to as multiple comparison with the best (MCB) ( 15 ) in the forecasting literature. Yet, sometimes the mean scaled score and the mean rank lead to conflicting conclusions. For instance, 21 report an experiment on the \sim 30,000 bottom time series of the M5 competition ( 19 ) . On the high quantiles, the Poisson static distribution is competitive according to the mean rank, but its mean scaled score is higher (i.e., worst) than most competitors ( 21 , Tab. 1) . It remains unclear whether it is preferable to perform model selection based on the mean scaled score or the mean rank, and which criterion to trust when they lead to conflicting conclusions; in this paper we fill this gap. The paper is organized as follows. In Sect. 2 , we show that the distribution of the pointwise scores is typically strongly skewed. In Sect. 3 , we show that also the distribution of the scores, and of their differences, is generally skewed for common values of n_{te} . We also show that a skewed distribution of the scores can allow a misspecified model to have a better mean rank than the true model. The same problem is found for other non-mean statistics such as the median scaled score and the win rate. In Sect. 4 , we perform model selection on real data sets of intermittent time series, choosing between the Poisson and the negative binomial distribution. We discuss the cases in which the mean rank and the mean scaled score lead to different conclusions, why this happens and which criterion is arguably more reliable. We also study the sensitivity of the decisions of the mean scaled score to the scaling factor and eventually we provide our recommendations. 2 The skew of the distribution of the pointwise scores We now introduce the most common univariate scoring rules, defined to be negatively oriented. The quantile score with probability p , \text{QS}_{p}(y,\hat{y}_{p}) , is: \displaystyle\text{QS}_{p}(y,\hat{y}_{p})=\begin{cases}2(1-p)\big(\hat{y}_{p}-y\big)&\text{ if }y<\hat{y}_{p}\text{ (overestimation),}\\ 2(p)\big(y-\hat{y}_{p}\big)&\text{ if }y\geq\hat{y}_{p}\text{ (underestimation),}\\ \end{cases} (1) where \hat{y}_{p} is the quantile at level p of the forecast distribution and y is the observation. Let us discuss some properties of the pointwise scores of Eq. ( 1 ), assuming p>0.5 ; the case p<0.5 is symmetric. They aee skewed since they are lower-bounded by zero and without upper bound. Moreover, the skew increases with p as we explain in the following. The pointwise scores are drawn from a mixture with two components. The overestimation component yields many small pointwise scores, while the underestimation component yields few large pointwise scores. Given an underestimation and an overestimation error of the same size, underestimation results in a pointwise scores which is k=\frac{1-p}{p} times larger than overestimation. For p=\{0.8,0.9,0.99\} , we have k=\{4,9,99\} . We also notice that, for well calibrated forecasts, the underestimation errors are k times rarer than the overestimation ones. The interval score ( \text{IS}_{p} ) ( 11 ) for a prediction interval with coverage p is: \displaystyle\text{IS}_{p}(l,u,y) \displaystyle=(u-l)+\frac{2}{1-p}(l-y)\mathbbm{1}\{y<l\}+\frac{2}{1-p}(y-u)\mathbbm{1}\{y>u\} (2) \displaystyle=\frac{2}{1-p}\left[\text{QS}_{\frac{1-p}{2}}(y,l)+\text{QS}_{\frac{1+p}{2}}(y,u)\right] (3) where l and u are the lower and upper endpoints, and \mathbbm{1} is the indicator function, which is equal to

Read the complete text