AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability Score
Simon Lang, Mihai Alexe, Mariana C. A. Clare, Christopher Roberts, Rilwan Adewoyin, Zied Ben Bouallègue, Matthew Chantry, Jesper Dramsch, Peter D. Dueben, Sara Hahner, Pedro Maciel, Ana Prieto-Nemesio, Cathal O’Brien, Florian Pinault, Jan Polster, Baudouin Raoult, Steffen Tietsche, Martin Leutbecher · 2024
Read AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability Score by Simon Lang, Mihai Alexe, Mariana C. A. Clare, Christopher Roberts, Rilwan Adewoyin, Zied Ben Bouallègue, Matthew Chantry, Jesper Dramsch, Peter D. Dueben, Sara Hahner, Pedro Maciel, Ana Prieto-Nemesio, Cathal O’Brien, Florian Pinault, Jan Polster, Baudouin Raoult, Steffen Tietsche, Martin Leutbecher (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15832.txt.
Introduction
Simon Lang and co-authors present, in a 2024 paper, a machine learning weather model built to run as an ensemble. A single forecast hides its own doubt, so forecasters run many versions to map the odds of each outcome. Their model, AIFS-CRPS, grows out of a European AI forecasting system. It is trained with a score that rewards forecasts whose spread matches how wrong they tend to be.
A small fix to that score keeps it fair when the ensemble is not huge. Once trained, the model is random by design, so it can spin out as many forecast members as the computers allow. For medium-range forecasts it beats the big physics-based ensemble on most measures. At longer ranges it also leads before fine tuning, and stays close after, a strong sign for AI ensemble forecasting.
Catalog record
| CREATOR | Simon Lang, Mihai Alexe, Mariana C. A. Clare, Christopher Roberts, Rilwan Adewoyin, Zied Ben Bouallègue, Matthew Chantry, Jesper Dramsch, Peter D. Dueben, Sara Hahner, Pedro Maciel, Ana Prieto-Nemesio, Cathal O’Brien, Florian Pinault, Jan Polster, Baudouin Raoult, Steffen Tietsche, Martin Leutbecher |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15832
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability Score Simon Lang, Mihai Alexe, Mariana C. A. Clare, Christopher Roberts, Rilwan Adewoyin, Zied Ben Bouallègue, Matthew Chantry, Jesper Dramsch, Peter D. Dueben, Sara Hahner, Pedro Maciel, Ana Prieto-Nemesio, Cathal O’Brien, Florian Pinault, Jan Polster, Baudouin Raoult, Steffen Tietsche, Martin Leutbecher, European Centre for Medium-Range Weather Forecasts (ECMWF) Abstract Over the last three decades, ensemble forecasts have become an integral part of forecasting the weather. They provide users with more complete information than single forecasts as they permit to estimate the probability of weather events by representing the sources of uncertainties and accounting for the day-to-day variability of error growth in the atmosphere. This paper presents a novel approach to obtain a weather forecast model for ensemble forecasting with machine-learning. AIFS-CRPS is a variant of the Artificial Intelligence Forecasting System (AIFS) developed at ECMWF. Its loss function is based on a proper score, the Continuous Ranked Probability Score (CRPS). For the loss, the almost fair CRPS is introduced because it approximately removes the bias in the score due to finite ensemble size yet avoids a degeneracy of the fair CRPS. The trained model is stochastic and can generate as many exchangeable members as desired and computationally feasible in inference. For medium-range forecasts AIFS-CRPS outperforms the physics-based Integrated Forecasting System (IFS) ensemble for the majority of variables and lead times. For subseasonal forecasts, AIFS-CRPS outperforms the IFS ensemble before calibration and is competitive with the IFS ensemble when forecasts are evaluated as anomalies to remove the influence of model biases. A Preprint 1 Introduction Over the last few years, several machine-learned weather prediction models have emerged that show superior performance to that of traditional physics-based models in many forecast scores. The first generation of these models produced deterministic predictions and were trained to minimise a mean-squared-error (MSE) loss ( Pathak et al., 2022 ; Bi et al., 2023 ; Lam et al., 2023 ; Chen et al., 2023 ; Lang et al., 2024a ) . The MSE training objective incentivises the smoothing of forecast fields with lead-time and a reduction of forecast activity, to avoid the ’double-penalty’ incurred when forecasting misplaced structures (e.g. Hoffman et al. (1995) ; Ebert et al. (2013) ). Nevertheless, evaluations demonstrated that these models display surprisingly physical behaviour in classical forecast situations ( Hakim and Masanam, 2024 ) , and provide genuine, useful predictions, including of many extreme events ( Ben Bouallègue et al., 2024a ) . The European Centre for Medium-Range Weather Forecasts (ECMWF) is now producing semi-operational weather predictions using the Artificial Intelligence Forecasting System (AIFS, Lang et al. (2024a) ) four times per day. For the usefulness of a weather forecast it is important to account for forecast uncertainties. Ensemble forecasts are run to estimate the probability density of the atmospheric state at a future time ( Lewis, 2005 ; Leutbecher and Palmer, 2008 ) . In physics-based numerical weather prediction (NWP), this is achieved via running the forecast model from a range of perturbed initial conditions and by introducing stochastic perturbations into the forecast model itself. The aim is to generate a well calibrated ensemble. This means that on average, the ensemble standard deviation needs to match the root-mean square error of the ensemble mean (e.g., Fortin et al. (2014) ), and the predicted probability of an event should accurately reflect the observed probability of it occurring. For physics-based ensemble simulations with the Integrated Forecasting System (IFS) at ECMWF ( Molteni et al., 1996a ) , initial condition uncertainty is represented via an ensemble of data assimilations ( Buizza et al., 2008 ; Isaksen et al., 2010 ; Lang et al., 2019 ) and singular vector perturbations ( Leutbecher and Palmer, 2008 ) . Perturbations to the initial conditions of the ensemble members are then constructed from both (see Lang et al. (2021) for an up-to-date description of the initial perturbation methodology). Uncertainties associated with the forecast model are represented stochastically ( Leutbecher et al., 2017 ; Berner et al., 2017 ) . For the first generation of machine-learned weather prediction models, ensemble forecasts were mainly based on an ensemble of MSE trained forecast models ( Bihlo, 2021 ; Scher and Messori, 2021 ; Clare et al., 2021 ; Pathak et al., 2022 ; Bi et al., 2023 ; Weyn et al., 2024 ) . The resulting ensemble forecasts tend to have too little ensemble spread. This is in line with the models having difficulties to represent the inherent variability of the atmosphere due to their training regime ( Ben Bouallègue et al., 2024b ) . The second generation of machine-learned weather forecast models are based on probabilistic training. For example, Kochkov et al. (2024) developed a hybrid model that combined a differentiable solver for atmospheric dynamics with a machine-learned physics module; this approach achieved good ensemble scores. Most notably, denoising diffusion ( Sohl-Dickstein et al., 2015 ; Karras et al., 2022 ) has been used successfully to create machine-learned ensemble models that are competitive with physics-based NWP models across a range of probabilistic forecast scores ( Price et al., 2023 ; Lang et al., 2024b ; Alexe et al., 2024a ) . Next to providing useful information about forecast uncertainties, these models have more stable statistics than the deterministically trained models and do not smooth out small-scale structures with forecast lead time. This can make forecasts more stable, and in some cases allows for consistent simulations from months ( Alexe et al., 2024a ) to many years ( Kochkov et al., 2024 ) . At ECMWF, the first approach towards a machine-learned ensemble model is also based on a diffusion approach ( Lang et al., 2024b ; Alexe et al., 2024a ) that achieves competitive ensemble scores when compared to the physics-based 9 km IFS ensemble ( Lang et al., 2023 ) . In this work, we take a different approach: we introduce AIFS-CRPS, a machine-learned ensemble forecast model that is based on optimising a probabilistic proper score objective, similar to Pacchiardi et al. (2024) ; Shokar et al. (2024) and Kochkov et al. (2024) . AIFS-CRPS learns how to represent model uncertainty, through shaping Gaussian noise. For training, we use the almost fair continuous ranked probability score (afCRPS) , a modification to the fair continuous ranked probability score (fCRPS) ( Ferro et al., 2008 ; Ferro, 2013 ; Leutbecher, 2019 ) . AIFS-CRPS is a highly skilful machine-learned ensemble forecast model that is competitive or superior to the physics-based IFS ensemble across forecast lead times ranging from days to subseasonal predictions. 2 Probabilistic training The AIFS-CRPS architecture largely follows that of the deterministic AIFS v0.2.1 ( Lang et al., 2024a ) with the encoder-processor-decoder design. The encoder and decoder of AIFS-CRPS are transformer-based graph neural networks (GNNs), while the processor backbone is a sliding window transformer. In contrast to the deterministic AIFS, however, the training of the AIFS-CRPS model is inherently probabilistic. AIFS-CRPS uses 16 processor layers and an embedding dimension of 1024 with 8 attention heads. This results in 229 million parameters in total. AIFS and AIFS-CRPS operate on reduced Gaussian grids, such as the octahedral reduced Gaussian grid ( Wedi, 2014