Measures of predictive accuracy, miscalibration and discrimination

Łukasz Delong, Mario Wüthrich · 2026

Read Measures of predictive accuracy, miscalibration and discrimination by Łukasz Delong, Mario Wüthrich (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12679.txt.

Introduction

Delong and Wüthrich study point predictors under mean losses given by Bregman gaps. They derive a new Murphy split of mean loss without the outcome itself, keeping only its predictors. They then relate bias and spread to Lorenz-curve fit gauges in wide use. Besides the usual area between concentration and Lorenz curves, ABC, they introduce ABC squared, which eases flaws of the first in spotting mean fit. Both rely on model-based weights, so they fail to align with mean scoring.

Catalog record

CREATORŁukasz Delong, Mario Wüthrich
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE134323
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12679

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Measures of predictive accuracy, miscalibration and discrimination Łukasz Delong, Mario Wüthrich Abstract We study the evaluation of real-valued point predictors under the decision-theoretic framework of mean-consistent loss functions given by the Bregman divergences. We first derive a new version of Murphy’s decomposition of the expected loss which does not directly include the response itself but only its predictors. We then relate the miscalibration and the discrimination component of the Murphy’s decomposition to Lorenz-curve-based accuracy measures that are widely used in practice. Besides the usual area between the concentration and Lorenz curves, ABC, we introduce a mean-squared version ABC 2 that mitigates some of the weaknesses of the original ABC in identifying mean-calibration. More importantly, both ABC and ABC 2 are shown to rely on predictor-dependent weights, so they fail to align with the class of mean-consistent scoring functions. In the same spirit, we derive a similar result for the widely used Gini score. These results indicate that ABC, ABC 2 and Gini scores may lead to dishonest evaluation of point predictions when used for model selection; this gives support to use mean-consistent loss functions as well as the miscalibration and the discrimination measure from the Murphy’s decomposition of the expected loss for model evaluation. Finally, we study forecast dominance when Lorenz curves intersect. We show that Lorenz and Murphy’s curves have the same number of crossings and, in the one-crossing case, we establish weaker dominance criteria for subclasses of Bregman divergences through third-degree stochastic dominance. Keywords: Mean-consistent loss function, Bregman divergence, miscalibration, discrimination, Gini index, Murphy’s decomposition, Murphy’s curve, Lorenz curve, concentration curve, forecast dominance. 1 Introduction In his pioneering work, Gneiting 2011 initiated a discussion about the specific choice of the loss function and its role in evaluating point predictions. Assume there are K competing point predictions (x^{(k)}_{1},\ldots,x^{(k)}_{n})_{k=1}^{K} for forecasting the observations y_{1},\ldots,y_{n} . Select a loss function (y,x)\mapsto L(y,x) that measures the discrepancy between the prediction x and the outcome y of the response. To evaluate the point predictions, consider the following performance criterion \displaystyle\bar{S}_{k}=\frac{1}{n}\sum_{i=1}^{n}L\big(y_{i},x^{(k)}_{i}\big),\qquad\text{ for $k=1,\ldots,K$,} (1.1) and compare \bar{S}_{1},\ldots,\bar{S}_{K} to select the best prediction, i.e., the one with the smallest average loss ( 1.1 ). The key question is what loss function L should be selected and which specific choices have theoretical support. Gneiting 2011 develops a decision-theoretic approach to the evaluation of point predictions and concludes that one should always choose so-called consistent loss functions . The idea promoted by Gneiting 2011 is that one should first decide on a functional of the predictive distribution in which one is interested in – such as the mean or a quantile – and then select a loss function L that is consistent with this functional choice. It turns out that the class of loss functions consistent for a certain functional is identical to the class of loss functions under which the functional is the optimal point forecast. Hence, if a forecaster chooses the best prediction by minimizing the empirical score ( 1.1 ), they have to choose a loss function under which the functional of the predictive distribution under consideration is the optimal point forecast. Other choices of loss functions may lead to misguided conclusions. The notion of consistent loss functions has been later extended by Gneiting–Raftery 2007 to proper scoring rules . In this paper we are interested in the estimation of mean values of observations. Hence, by evaluating point predictions we always think of evaluating mean estimates. The class of loss functions consistent for mean values has been characterized by Savage 1971 , Gneiting 2011 and Ehm et al. 2016 , and it includes so-called Bregman divergences ; in fact, under mild conditions this is an if-and-only-if statement. The most popular examples include the mean squared loss, Kullback–Leibler divergences and loss functions related to deviance functions in the exponential dispersion family. The main criterion for evaluating point predictions is prediction accuracy characterized by closeness between predictions and observations measured with a consistent loss function. However, good predictions should also have other desirable properties. Murphy 1973 and Murphy–Winkler 1987 investigate decompositions of loss functions. Pohle 2020 proves that any consistent loss function can be decomposed into measures of miscalibration and discrimination . This decomposition, called Murphy’s decomposition, allows one to look deeper into predictive accuracy of predictors. Murphy’s decomposition can identify strengths and weaknesses of predictions and guide forecasters to improve their predictive models. The measures of miscalibration and discrimination from Murphy’s decomposition are in agreement with the decision-theoretic approach to the evaluation of point predictions from Gneiting 2011 since they are related to consistent loss functions, see Gneiting–Resin 2023 . The first contribution of this paper is to derive a new version of Murphy’s decomposition. In this new version, the measures of miscalibration and discrimination only depend on the predictor, and not on the responses. The main benefit is that this provides one with a more intuitive interpretation of theses measures. One can now directly measure, with a consistent loss function, how close the predictor is to its calibrated version (the miscalibration statistics) and how far the predictor is from a constant prediction (the discrimination statistics), and this does not involve the responses which, in particular, makes it robust towards different tail properties of the responses. The loss functions consistent for mean values can be decomposed to get the measures of miscalibration and discrimination of mean estimates. At the same time, there are other measures of miscalibration and discrimination which have gained popularity in practical applications. For actuarial applications, Denuit et al. 2019 , Denuit et al. 2020 , Denuit–Trufin 2021 introduced and promoted the area between the Lorenz and the concentration curve (the so-called ABC statistics) to measure the miscalibration of predictions. Recently, Denuit–Trufin 2025 investigate properties and failures of this measure. Yet, we believe that a deeper investigation of the ABC measure is still missing in the literature. In particular, Denuit et al. 2019 remark that there is a connection between the ABC and the miscalibration statistics linked with consistent loss functions. Moreover, the relation between the ABC measure and the miscalibration statistics from Murphy’s decomposition is not fully understood yet. The second contribution of this paper is to develop this relation by finding a new representation of the ABC measure. These new results presented in this paper allow us to assess if the ABC measure fits the framework of the decision-theoretic approach to the evaluation of point predictions from Gneiting 2011 . In addition to the ABC measure, we also introduce a new miscalibration statistics, the area between the Lorenz and the concentration curve measured in a mean squared sense, and study its properties. We also prove a new representation of this measure in the spirit of the new representation for the {\rm ABC} . As far as measures of discrimination are concerned, the Gini index, introduced by Gini 1912 , is the most popular statistics used in applications, also in actuarial applications. Denuit et al. 2019 , Denuit et al. 2020 and Denuit–Trufin 2021 use the classical Gini index based on the Lorenz curve to evaluate predictors. Brauer–Wüthrich 2025 investigate a machine learning version of the Gini index in an actuarial context where the treatment of case weights may pose some

Read the complete text