A generalisation of the signal-to-noise ratio using proper scoring rules
Jochen Bröcker, Eviatar Bach · 2025
Read A generalisation of the signal-to-noise ratio using proper scoring rules by Jochen Bröcker, Eviatar Bach (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26809.txt.
Introduction
Jochen Bröcker and Eviatar Bach generalise a classic measure of forecast quality. The signal-to-noise ratio, and its cousin the ratio of predictable components, say how much of a forecast's variation is real signal. The classical versions only fit forecasts that can be averaged, which limits their reach.
Bröcker and Bach rebuild the concept on proper scoring rules, scores that reward honest forecasts. Their version is the natural generalisation of the classical one, yet it applies to any forecast that can be scored. They illustrate it with ensemble forecasts scored by a standard ranked score, and with probability forecasts of a binary event scored by the logarithmic score. On synthetic data the statistics agree on which sets look anomalous, with different spread. On North Atlantic Oscillation hindcasts, long read as a signal-to-noise paradox, the verdict is less clear-cut.
Catalog record
| CREATOR | Jochen Bröcker, Eviatar Bach |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26809
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A generalisation of the signal-to-noise ratio using proper scoring rules Jochen Bröcker, Eviatar Bach Abstract A generalised concept of the signal-to-noise ratio (or equivalently the ratio of predictable components, or RPC \operatorname{RPC} ) is provided, based on proper scoring rules. This definition is the natural generalisation of the classical RPC \operatorname{RPC} , yet it allows one to define and analyse the signal-to-noise properties of any type of forecast that is amenable to scoring, thus drastically widening the applicability of these concepts. The methodology is illustrated for ensemble forecasts, scored using the continuous ranked probability score (CRPS), and for probability forecasts of a binary event, scored using the logarithmic score. Numerical examples are demonstrated using synthetic data with prescribed signal-to-noise ratios as well as seasonal ensemble hindcasts of the North Atlantic Oscillation (NAO) index. For the synthetic data, the RPC \operatorname{RPC} statistic as well as the scoring rule–based ones agree regarding which data sets exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties. For the NAO data, on the other hand, the results among the different statistics are more equivocal. \papertype Original Article \corraddress Jochen Bröcker and Eviatar Bach, Department of Mathematics and Statistics, University of Reading, Whiteknights, Reading, Berkshire, RG6 6AX \corremail [email protected], [email protected] Keywords — signal-to-noise ratio, signal-to-noise paradox, scoring rules, probabilistic forecasting 1 Introduction The signal-to-noise “paradox” was first described in Scaife and Smith 2018 in the context of ensemble forecasts on seasonal timescales. It refers to a situation in which the correlation between ensemble mean and actual verification is larger than the correlation between ensemble mean and individual ensemble members. That the ensemble mean is better able to predict the verification than the ensemble members might indeed seem paradoxical, although it has to be kept in mind that in this context the prediction skill is quantified in terms of correlation rather than any form of error. Following Bröcker et al. 2023 , we will henceforth refer to such a situation as an anomalous signal-to-noise ratio. The signal-to-noise ratio ( \operatorname{SNR} ) as a statistical diagnostic is not new. To our knowledge it was first introduced to the climate literature in Madden 1976 . The idea of estimating \operatorname{SNR} s from ensembles by partitioning the ensemble spread into “signal” and “noise” was considered by various authors and is reviewed in Rowell 1998 . Although estimated \operatorname{SNR} s can indeed be used to diagnose normal or anomalous signal-to-noise ratios, there are alternative (yet equivalent) diagnostics. The ratio of predictable components ( \operatorname{RPC} ) statistic, for instance, was introduced in Eade et al. 2014 . In spite of widespread use of the \operatorname{RPC} , a thorough analysis of the statistical properties of the \operatorname{RPC} or equivalent diagnostics is still outstanding. It is not clear, for instance, when the \operatorname{RPC} of a given data set (or more precisely an estimator of the \operatorname{RPC} ) is “significantly” larger than one; a proper testing methodology would have to be able to answer that question. The present paper aims to contribute to the development of this methodology. The precise motivation for this paper is threefold. Firstly, we want to widen the spectrum of possible statistics for diagnosing an anomalous signal-to-noise ratio. A greater flexibility in choosing test statistics is likely to be beneficial, rather than just relying on the classical \operatorname{RPC} . In particular, other test statistics may have better sampling properties or more accurately capture the probabilistic information contained in the forecasts. Secondly, the classical \operatorname{RPC} is only applicable to real-valued verifications. This is a serious limitation, since there are various other types of verifications such as spatial or categorical ones, which as of yet cannot be analysed in terms of the \operatorname{RPC} . Thirdly, by placing the \operatorname{RPC} in a more general information-theoretic framework, we hope to illuminate its theoretical origin and assumptions involved in its use. We propose to generalise the \operatorname{RPC} by considering instead the ratio of two skill scores , namely the skill score of the forecast against a verification drawn from the forecast, versus the skill score of the perfectly recalibrated forecast against the actual verifications. In common with previous authors, skill score in this paper refers to the score of a forecast, relative to the score of a reference forecast, typically a climatological forecast. To score forecasts we use proper scoring rules ( Bröcker and Smith 2007a ; Bröcker and Smith 2007a ) , as these provide a coherent framework for forecast evaluation, see also Dawid 2007 for the scoring rule–based notion of entropy, and Takaya et al. 2025 for its use in a climate context. In Section 2 we introduce the ratio of skill scores ( \operatorname{RSS} ), the proposed generalisation of the \operatorname{RPC} , after we briefly recalled the notion of proper scoring rules and their properties in Subsection 2.1 . The \operatorname{RPC} can be applied to any type of forecast that is amenable to scoring. In Subsection 2.2 we demonstrate that this definition is the natural generalisation of the classical \operatorname{RPC} . In Section 3 we briefly discuss some implications of an anomalous signal-to-noise ratio (these apply likewise to the general and the classical definition.) Numerical examples are discussed in Section 4 . We consider ensemble forecasts, scored using the continuous ranked probability score (CRPS), and probability forecasts of a binary event, scored using the logarithmic score. We compare the \operatorname{RSS} s with the classical \operatorname{RPC} in the context of synthetic data sets as well as data from an operational seasonal forecasting system. Section 5 concludes. 2 Mathematical framework 2.1 Proper scoring rules Let E be some space which, for the purpose of this paper, can be taken as either the Euclidean space \mathbb{R}^{d} or simply a finite set \{e_{1},\ldots,e_{K}\} . Further, let \mu,\nu be probability distributions over E which the reader might think of as densities (in the case of E=\mathbb{R}^{d} ) or probability vectors (of dimension K in the case of E=\{e_{1},\ldots,e_{K}\} ). A scoring rule over E is a function s(\mu,y) taking as input such a probability distribution \mu , and some y\in E which is a possible value for the observation or verification. The value s(\mu,y) measures how well \mu forecasts y . We adopt the convention of negatively oriented scores: a smaller score indicates a better forecast. For any scoring rule s and probability distributions \mu,\nu we define the associated expected score \mathcal{S}(\mu,\nu):=\int s(\mu,y)\nu(y)\;\mathrm{d}y, (1) and the entropy \mathcal{E}(\mu):=\mathcal{S}(\mu,\mu). (2) We furthermore define the divergence associated with s through the formula \mathcal{D}(\mu,\nu):=\mathcal{S}(\mu,\nu)-\mathcal{E}(\nu). (3) The scoring rule is proper if \mathcal{D}(\mu,\nu)\geq 0 , and strictly proper if \mathcal{D}(\mu,\nu)=0 implies \mu=\nu . Scoring rules have to be proper in order to be interpreted as measures of forecast performance ( Bröcker and Smith 2007a ; Gneiting and Raftery 2007 ) . We note that the scoring rule being strictly proper implies that \mathcal{D} is in fact a statistical divergence ( Gneiting and Raftery 2007 ; Bach et al. 2025 ) , so it can be regarded as a sort of distance or discrepancy between two probability distributions, although \mathcal{D} is not symmetric in its arguments. We henceforth fix a proper scoring rule s . A probability forecast is a probability distribution f on E which at the same time