SAFE: A Novel Approach to AI Weather Evaluation through Stratified Assessments of Forecasts over Earth

Nick Masi, Randall Balestriero · 2025

Read SAFE: A Novel Approach to AI Weather Evaluation through Stratified Assessments of Forecasts over Earth by Nick Masi, Randall Balestriero (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26099.txt.

Introduction

Nick Masi and Randall Balestriero argue that AI weather models should be judged place by place, not only by a global average. Standard scoring averages loss over all test points, which in weather means averaging over the whole Earth at once. That hides gaps tied to uneven human development and geography, since a model can excel in data rich regions and fail elsewhere.

Their package, SAFE, stratifies forecast scores by territory, subregion, income, and land cover, using linked data domains for each grid point. Applied to a zoo of leading AI weather models, it finds disparities across every attribute for all models. The authors use the tool to seed a fairness benchmark across lead times and climate variables.

Catalog record

CREATORNick Masi, Randall Balestriero
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26099

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SAFE: A Novel Approach to AI Weather Evaluation through Stratified Assessments of Forecasts over Earth Nick Masi, Randall Balestriero Abstract The dominant paradigm in machine learning is to assess model performance based on average loss across all samples in some test set. This amounts to averaging performance geospatially across the Earth in weather and climate settings, failing to account for the non-uniform distribution of human development and geography. We introduce Stratified Assessments of Forecasts over Earth ( safe ), a package for elucidating the stratified performance of a set of predictions made over Earth. safe integrates various data domains to stratify by different attributes associated with geospatial gridpoints: territory (usually country), global subregion, income, and landcover (land or water). This allows us to examine the performance of models for each individual stratum of the different attributes (e.g., the accuracy in every individual country). To demonstrate its importance, we utilize safe to benchmark a zoo of state-of-the-art AI-based weather prediction models, finding that they all exhibit disparities in forecasting skill across every attribute. We use this to seed a benchmark of model forecast fairness through stratification at different lead times for various climatic variables. By moving beyond globally-averaged metrics, we for the first time ask: where do models perform best or worst, and which models are most fair? To support further work in this direction, the safe package is open source and available at https://github.com/N-Masi/safe . 1 Introduction Artificial intelligence weather prediction (AIWP) models, alternatively machine learning weather prediction (MLWP) models or neural weather models (NWM), are becoming increasingly competitive with traditional numerical weather prediction (NWP) models. All of these approaches are typically used in making medium-range weather forecasts (interchangeably, “prediction”). The range of a forecast is determined by its lead time \tau . When a weather prediction model is fed the state of variables at time d , its task is to predict the state of those variables (or some subset of them) at time d+\tau . There is no consistent definition for medium-range, with the European Centre for Medium-Range Weather Forecasts (ECMWF) defining it as any prediction made with \tau (or n\times\tau if taking an autoregressive rollout of n steps) within 0–15 days ( European Centre for Medium-Range Weather Forecasts 2025b ) , while other sources more narrowly define it as 3–7 days ( Meteorological Society ; Weather Prediction Center ) . AIWP models are seeing increasing adoption in interfaces where they provide these medium-range forecasts, from Google’s Weather app ( Leffer 2024 ) to various experimental models at the National Oceanic and Atmospheric Administration (NOAA) ( Potvin et al. 2025 ; Sadeghi Tabas et al. 2025 ) . Root mean square error (RMSE) is the preeminent metric used in assessing the quality of AIWP models ( Radford et al. 2025 ; Rasp et al. 2020 ) . The general form of RMSE is shown in Equation 1 , where Y is the set of all ground truth variable values that a model is trying to predict, and \hat{y} is the model’s prediction for each corresponding y\in Y . Every y is the value of some variable (e.g., temperature) at some point in time d\in D , longitude i\in I , latitude j\in J , and, for certain atmospheric variables, vertical level v\in V . \sqrt{\frac{\Sigma_{y\in Y}(\hat{y}-y)^{2}}{|Y|}} (1) There are various approaches for how different models handle being able to make predictions at different lead times. The naive approach is to train a model with the ability to predict some fixed \tau^{\prime}\in T amount of time in the future, where T is a set of durations. This allows the model to forecast the weather with temporal resolution of \tau^{\prime} (i.e., multiples of \tau^{\prime} after the timestamp of the input variables) through autoregressive rollout. This is the approach taken by Keisler ( Keisler 2022 ) , the Spherical CNN ( Esteves et al. 2023 ) , FourCastNet ( Pathak et al. 2022 ) , and the spherical Fourier neural operator (SFNO) ( Bonev et al. 2023 ) , all with \tau^{\prime}=6 hours. Pangu-Weather ( Bi et al. 2022 ) trains four different models, each with a different, fixed lead time. This is used in tandem with a greedy algorithm that minimizes the number of autoregressive steps that need to be taken to make a prediction at any given lead time (which must be a multiple of their smallest lead time model). FuXi ( Chen et al. 2023 ) uses a cascaded set of three different models that cover different ranges of lead times. The square of RMSE, mean squared error (MSE), frequently referred to as the L2 loss, is often used as a training objective. This is the case for Spherical CNN ( Esteves et al. 2023 ) and GenNet ( Lopez-Gomez et al. 2023 ) . GraphCast ( Lam et al. 2023 ) and GenCast ( Price et al. 2023 ) use weighted MSE loss functions. Keisler takes a weighted sum of MSE values ( Keisler 2022 ) . NeuralGCM ( Kochkov et al. 2024 ) has a five-term loss function, each of which is a variation of MSE. FuXi ( Chen et al. 2023 ) uses the mean absolute error (MAE, the L1 counterpart of MSE). The underlying commonality across all of these functions is that they completely reduce across the spatial dimensions I and J . One issue with spatial averaging as the loss function is the resulting “double penalty" that arises when predictions for high resolution events are even slightly spatially displaced, incurring the penalization for both that faulty prediction and the lack of prediction at the true location ( Gilleland et al. 2009 ) . This encourages models to blur their predictions, dropping these highly localized events ( Lam et al. 2023 ) . However, neglecting to predict these outlier events can have dramatic real-world consequences. For example, improved accuracy of extreme heat predictions has been found to reduce mortality ( Shrader et al. 2023 ) . Another issue with spatial averaging for evaluation is that it becomes unknown precisely where models are and are not performing well. Accordingly, it is impossible to know whether they can be trusted at inference time in a given location. With safe , we aim to uncover spatial disparities in performance by separating the spatial dimensions into different strata and calculating performance within each. Takeaway 1 The state of the art of AI weather prediction relies on spatially-averaged objective functions and evaluation metrics. These de-emphasize high-frequency events despite the fatal consequences of losing this predictive power. They also mask disparities that exist in where models perform well. 2 Related work WeatherBench 2 (WB2) ( Rasp et al. 2024 ) is an existing benchmark that assesses the spatially-averaged error of models using weather data from ERA5, ECMWF’s most modern reanalysis dataset ( Hersbach et al. 2020 ) . It provides functionality to get per-region RMSE, but these regions are coarse-grained and exclusively rectangular, making them unusable for the real-world attributes we care about. Stable equitable error in probability space (SEEPS) ( Rodwell et al. 2010 ) is a metric that was introduced to assess the quality of precipitation forecasts in particular. In the original paper ( Rodwell et al. 2010

Read the complete text