Extreme Weather Bench: A framework and benchmark for evaluation of high-impact weather
Amy McGovern, Taylor Mandelbaum, Daniel Rothenberg, Nicholas Loveday, Corey Potvin, Montgomery Flora, Linus Magnusson, Eric Gilleland, John Allen · 2026
Read Extreme Weather Bench: A framework and benchmark for evaluation of high-impact weather by Amy McGovern, Taylor Mandelbaum, Daniel Rothenberg, Nicholas Loveday, Corey Potvin, Montgomery Flora, Linus Magnusson, Eric Gilleland, John Allen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01126.txt.
Introduction
Amy McGovern and eight co-authors introduce Extreme Weather Bench, an open community benchmark for evaluating forecasts of high-impact weather. Forecasting such events is hard for both AI and numerical weather models, and verification before deployment is critical. Current AI evaluation often uses global scores or a handful of hand-picked cases, which cannot support fair comparison across models or hazards.
The suite provides standard case studies across scales and parts of the weather spectrum, observational data, impact-based metrics, and open-source evaluation code. Users can test models on specific high-impact phenomena that matter to the public. The free system is designed to evolve with the worldwide forecast verification community, adding phenomena, cases, and metrics over time.
Catalog record
| CREATOR | Amy McGovern, Taylor Mandelbaum, Daniel Rothenberg, Nicholas Loveday, Corey Potvin, Montgomery Flora, Linus Magnusson, Eric Gilleland, John Allen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01126
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Extreme Weather Bench: A framework and benchmark for evaluation of high-impact weather Amy McGovern, Taylor Mandelbaum, Daniel Rothenberg, Nicholas Loveday, Corey Potvin, Montgomery Flora, Linus Magnusson, Eric Gilleland, John Allen Abstract Forecasting the wide variety of high-impact weather events experienced globally is a challenge for both Artificial Intelligence (AI) and Numerical Weather Prediction (NWP) models and it is critical that such models be properly verified before deployment. Although AI weather models are rapidly evolving (e.g., 37 , 54 , 1 , 4 ), much of their evaluation is currently done either with a global-scale evaluation (e.g., 56 , 57 ) or by hand-picking a small number of case studies or a region (e.g., 54 , 19 , 43 ). A widely-used open-source benchmark suite focusing on high-impact weather will help to drive the science forward for all scales of weather models, as it has for other AI fields (e.g. 16 ). Here we introduce Extreme Weather Bench (EWB), a new community-driven benchmark suite that facilitates model validation and verification on a variety of high-impact hazards that matter to people around the globe. EWB provides a standard set of case studies (spanning across multiple spatial and temporal scales and different parts of the weather spectrum), observational data, impact-based metrics, and open-source code for users to evaluate their models. Verifying that a model works against a standard set of case studies, especially events that are high-impact for the general public, is a key piece of improving the trustworthiness of AI models 46 . EWB will help to drive the science forward for all weather models, enabling true comparisons across models and evaluating models on specific high-impact phenomena through the use of case studies. EWB is a free open-source community-driven system and will continue to evolve to include additional phenomena, test cases and metrics in collaboration with the worldwide weather and forecast verification community. keywords AI weather prediction, Benchmarking, Extreme Weather, High-impact weather, Atmospheric Sciences, Machine Learning, AI 1 Introduction Weather forecasting impacts every aspect of our lives, from everyday choices about clothing and whether to pack an umbrella, to life-saving decisions and actions in advance of high-impact weather. With the rapid proliferation of AI-based weather forecasts, it is increasingly important that we ensure these predictions are both “trustworthy” 11 ; 68 ; 7 and “good” 49 . However, defining what makes a forecast “good” is challenging and varies between individuals as well as by the phenomena being predicted. Murphy 49 defined three categories in which we can measure the “goodness” of a forecast. First, a good forecast must be consistent , which means that the forecast should correspond to the forecaster’s best judgement of the forecast event. While Murphy focused on a human forecaster’s judgment, the concept of ensuring an AI or NWP forecast is consistent over time is also a critical component of a trustworthy forecast 11 . It is important that verification methods align with desirable behaviour of the forecasts. For probabilistic forecasts, evaluation using proper scoring rules 23 can encourage consistent forecasts. For single-valued point forecasts, where a forecast directive is expressed in the form of a statistical functional such as, “forecast the 90th percentile”, Gneiting 24 formally defined the concept of “consistent scoring functions”. For deterministic models, the notion of consistency is not formally defined, but can be encouraged by using verification methods that align with the core principles or goals of model development. Current evaluation systems of AI models often study only a handful of case studies, which does not address the consistency over time issue. Murphy’s second component of a good forecast was that it was of high quality , meaning the forecast matched the observations of the actual event as closely as possible. This aspect is studied by most current evaluation and verification systems. There are a wide variety of metrics to measure different aspects of forecast quality 50 ; 9 . In most current AI evaluation systems, metrics such as the Root Mean Squared Error (RMSE) of key surface and model-level fields such as temperature and humidity or the Anomaly Correlation Coefficient (ACC) of 500mb geopotential heights take a prominent role, but a handful of studies are beginning to compare the phenomena generated by AI models to the known physics 27 . Murphy’s final quality of a good forecast is that it should provide high value , which means it is of use to the end-users. This measure is often overlooked in favor of studying the quality of a forecast because many people assume that a high quality forecast automatically provides high-value. While they are correlated, it is not automatic that a forecast of high quality provides high value to the end-user, as it may not be predicting anything that impacts the user or it may not be focused on their specific needs. There is a growing body of research focused on working directly with forecasters to qualitatively understand what their needs are for forecasts 15 ; 59 and a smaller body of research focusing on economic value of forecasts 58 . As AI weather models propagate and are evaluated for possible operational use, a comprehensive and standardized ecosystem of model intercomparison, from global to local scales, is necessary. Forecast verification is an important topic that has been studied since at least 1884 20 ; 33 . As computer models and Numerical Weather Prediction (NWP) have provided additional value to forecasting, there have been a wide variety of different verification metrics and approaches developed. The World Meteorological Organization (WMO) has a joint working group, the WWRP/WGNE Joint Working Group on Forecast Verification Research (JWGFVR), focused on the issues of forecast verification worldwide 1 1 1 https://community.wmo.int/en/activity-areas/wwrp/wwrp-working-groups/wwrp-forecast-verification-research . The current state-of-the-art for global AI model evaluation primarily focuses on one or two scores, often Root Mean Squared Error (RMSE) or Anomaly Correlation Coefficient (ACC), and on prognostic variables. These metrics have several issues. First, they reward AI models for smooth fields, rather than fields that resemble physics-based weather. Second, they do not apply broadly across the high-impact weather spectrum. Third, they are often applied to variables at the 500mb height, rather than focused on the weather itself. ECMWF provides a standard evaluation framework for models 2 2 2 Quality of our forecasts — ECMWF also used for their operational forecasts but it does not provide an in-depth evaluation of models across a variety of scales. Additionally, it focuses on broad metrics and not on any specific test cases. WeatherBench 56 ; 57 is quickly becoming a standard for evaluation at the large-scale but it also lacks case studies, and evaluation across a variety of high-impact hazards. 53 and 70 focused on specific extremes using a WeatherBench style approach but did not look at specific events or use impact-based metrics. In addition, there are several efforts focused on organizing metrics by model type and use-case, including a Unified Forecasting Community workshop 3 3 3 2021 DTC UFS Evaluation Metrics Workshop - Final Metrics Lists , the World Meteorological Organization (WMO) WWRP/WGNE JWGFVR, the WMO WP-MIP: the Weather Prediction Model Intercomparison Project 4 4 4 WP-MIP: the Weather Prediction Model Intercomparison Project — ESMO Website , and the scores package 40 . This work presents Extreme Weather