Evaluating Extreme Precipitation Forecasts: A Threshold-Weighted, Spatial Verification Approach for Comparing an AI Weather Prediction Model Against a High-Resolution NWP Model
Nicholas Loveday, Tracy Hertneky · 2025
Read Evaluating Extreme Precipitation Forecasts: A Threshold-Weighted, Spatial Verification Approach for Comparing an AI Weather Prediction Model Against a High-Resolution NWP Model by Nicholas Loveday, Tracy Hertneky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25045.txt.
Introduction
Nicholas Loveday and Tracy Hertneky build a fair test to match an AI weather model with a classic fine grid model on hard rain. AI models now rival classic ones at less cost, but hard rain is where trust is won or lost. Space match counts here, and the two model grids do not line up.
Their frame joins space check tools with score rules that weight high rain lines more. Users can set how much each rain line counts for a call, to fit how a forecast desk would read it. A run of past hard rain cases shows the frame at work. It is built to test the events that hurt most, not mean days.
Catalog record
| CREATOR | Nicholas Loveday, Tracy Hertneky |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-25045
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Extreme Precipitation Forecasts: A Threshold-Weighted, Spatial Verification Approach for Comparing an AI Weather Prediction Model Against a High-Resolution NWP Model Nicholas Loveday, Tracy Hertneky Abstract Recent advances in AI-based weather prediction have led to the development of artificial intelligence weather prediction (AIWP) models with competitive forecast skill compared to traditional NWP models, but with substantially reduced computational cost. There is a strong need for appropriate methods to evaluate their ability to predict extreme weather events, particularly when spatial coherence is important, and grid resolutions differ between models. We introduce a verification framework that combines spatial verification methods and proper scoring rules. Specifically, the framework extends the High-Resolution Assessment (HiRA) approach with threshold-weighted scoring rules. It enables user-oriented evaluation consistent with how forecasts may be interpreted by operational meteorologists or used in simple post-processing systems. The method supports targeted evaluation of extreme events by allowing flexible weighting of the relative importance of different decision thresholds. We demonstrate this framework by evaluating 32 months of precipitation forecasts from an AIWP model and a high-resolution NWP model. Our results show that model rankings are sensitive to the choice of neighbourhood size. Increasing the neighbourhood size has a greater impact on scores evaluating extreme-event performance for the high-resolution NWP model than for the AIWP model. At equivalent neighbourhood sizes, the high-resolution NWP model only outperformed the AIWP model in predicting extreme precipitation events at short lead times. We also demonstrate how this approach can be extended to evaluate discrimination ability in predicting heavy precipitation. We find that the high-resolution NWP model had superior discrimination ability at short lead times, while the AIWP model had slightly better discrimination ability from a lead time of 24-hours onwards. 1 Introduction Over recent years, there has been rapid progress in the development of artificial intelligence weather prediction (AIWP) models, sometimes referred to as data-driven models ( Ben Bouallègue et al. 2024 ) . These models are typically based on neural networks trained on reanalysis datasets. This approach contrasts with traditional numerical weather prediction (NWP) models, which rely on equations that model physical processes. One major advantage of AIWP models is that they are significantly more computationally efficient than NWP models for making predictions once they have been trained. Most early global AIWP models are trained on ERA5 ( Hersbach et al. 2020 ) data, and therefore have a similar or coarser grid resolution of approximately 0.25^{\circ} . These include deterministic models ( Keisler 2022 ; Lam et al. 2023 ; Bi et al. 2023 ; Lang et al. 2024a , e.g.,) and ensemble models ( Price et al. 2024 ; Lang et al. 2024b , e.g.,) . In general, these models operate at much coarser resolutions than the high-resolution physical NWP models available to many international weather forecasting centers. However, higher-resolution, limited-area AIWP models have started to emerge more recently ( Nipen et al. 2024 ; Adamov et al. 2025 ; Abdi et al. 2025 , e.g.,) , and this trend is expected to continue. As modelling centers have access to high-resolution NWP models and with the accelerating development of higher-resolution AIWP models, there is a strong need for appropriate methodologies to compare the performance of models with differing spatial resolutions. Such comparisons are important to help meteorological agencies make informed decisions regarding future modelling strategies and to assist operational meteorologists and forecast system developers in understanding each model’s strengths and weaknesses. This raises several key questions we aim to address. First, most benchmarking efforts of global AIWP models to date have relied on point-to-point verification methods ( Rasp et al. 2024 , e.g.,) , with perhaps the exception of Radford et al. 2025a , who applied an object-based verification approach. Point-to-point methods may be appropriate for evaluating forecasts at individual locations (e.g., for end-users accessing weather forecasts via mobile apps). However, meteorologists and forecast systems often use forecasts spatially, where spatial coherence and physical realism are important. Point-to-point verification methods tend to favour smoother forecasts than are typically observed in gridded observations ( Subich et al. 2025 ) . When spatial structure is important, point-to-point metrics can suffer from the “double penalty” effect ( Ebert 2008 ) . This can occur when a forecast feature matches the observed shape and intensity but is slightly displaced in location, leading to it being penalised twice. It is penalised once for the false prediction at the forecast location, and once for the missed event at the observed location. This issue is particularly problematic for high-resolution models, which aim to represent fine-scale features. Lower-resolution models that fail to capture the feature at all may score better under point-to-point metrics, because they incur only a single penalty. Thus, while high-resolution NWP models may produce forecasts that appear more realistic to meteorologists, they may not perform better under standard point-to-point verification metrics ( Mass et al. 2002 ) . To address the limitations of point-based verification, a wide range of spatial verification methods have been developed over the past two and a half decades ( Gilleland et al. 2009 ; Dorninger et al. 2018 ) . These methods are typically grouped into five categories: neighbourhood methods, scale-separation methods, feature-based (or object-based) methods, field-deformation methods, and distance measures. In this paper, we focus on an approach within the neighbourhood methods category ( Ebert 2009 ; Schwartz and Sobash 2017 ) . Neighbourhood methods are verification approaches that evaluate gridded forecasts that are located within a local neighbourhood of the observations. Second, there is a pressing need to assess how well both AIWP and high-resolution NWP models predict extreme weather events. Much of the existing literature has focused on a limited number of case studies ( Charlton-Perez et al. 2024 ; Morisseau et al. 2025 , e.g.,) . While case studies are valuable, they may be prone to selection bias or may not generalise across more cases. Focusing solely on cases where extremes occur can lead to the “forecaster’s dilemma” ( Lerch et al. 2017 ) . The forecaster’s dilemma arises when forecasters or modellers must choose between issuing honest forecasts and adjusting their forecasts to optimise a particular scoring rule (we refer to the latter as “hedging”). Proper scores discourage hedging and avoid the “forecaster’s dilemma”. A scoring rule is proper if the expected score is optimised by producing a forecast that corresponds to the forecaster’s true belief ( Winkler and Murphy 1968 ; Gneiting and Raftery 2007 ) . There are some recent examples of broader evaluation of extremes across many events. For example, Lam et al. 2023 and Ben Bouallègue et al. 2024 evaluated the discrimination ability of AIWP models to differentiate between climatological extreme temperature events and non-events. Olivetti and Messori 2024 assessed performance in forecasting temperature and wind extremes primarily by conditioning on observed extremes, which is also susceptible to the forecaster’s dilemma. However, they also included an evaluation using a threshold-weighted squared error ( Taggart 2021 ) in their Appendix B, which avoids this problem and provides a more robust metric for evaluating extreme events. Zhang et al. 2025 evaluated several AIWP models and ECMWF’s HRES model against observations that are more extreme than the observations in the 1979-2017 training period of the AIWP models. In an attempt