MAUSAM: An Observations-focused assessment of Global AI Weather Prediction Models During the South Asian Monsoon

Aman Gupta, Aditi Sheshadri, Dhruv Suri · 2025

Read MAUSAM: An Observations-focused assessment of Global AI Weather Prediction Models During the South Asian Monsoon by Aman Gupta, Aditi Sheshadri, Dhruv Suri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01879.txt.

Introduction

Aman Gupta, Aditi Sheshadri, and Dhruv Suri grade seven AI weather systems on the South Asian monsoon using real observations. Most AI weather grading leans on reanalysis, a blended data product that can hide flaws. Their MAUSAM check uses ground stations, rain gauges, and satellite images instead. The monsoon brings heavy rain and storms where sparse data make forecasts hard.

The models show strong skill on broad traits, from surface heat and winds to rain and cloud cover. Finer traits still fail, with extreme rain underdone, cyclone paths split, and small scale energy off. Errors against observations run notably larger than against reanalysis, so past grades flatter the models. One system proves most consistent overall, with two rivals close behind.

Catalog record

CREATORAman Gupta, Aditi Sheshadri, Dhruv Suri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01879

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MAUSAM: An Observations-focused assessment of Global AI Weather Prediction Models During the South Asian Monsoon Aman Gupta, Aditi Sheshadri, Dhruv Suri Abstract Accurate weather forecasts are critical for societal planning and disaster preparedness. Yet these forecasts remain challenging to produce and evaluate, especially in regions with sparse observational coverage. Current evaluation of artificial intelligence (AI) weather prediction relies primarily on reanalyses, which can obscure important deficiencies due to prevailing biases in reanalyses themselves. Here we present MAUSAM (Measuring AI Uncertainty during South Asian Monsoon), an evaluation of seven leading AI-based forecasting systems - FourCastNet, FourCastNet-SFNO, Pangu-Weather, GraphCast, Aurora, AIFS, and GenCast - during the South Asian Monsoon, using ground-based weather stations, rain gauge networks, and geostationary satellite imagery. The AI models demonstrate impressive forecast skill during monsoon across a broad range of variables, ranging from large-scale surface temperature and winds to precipitation and cloud cover. The models also generate realistic zonal means and eddy statistics on subseasonal to seasonal timescales, highlighting the strength of purely data-driven weather prediction. However, the models still exhibit systematic errors in finer-scale features like the underprediction of extreme precipitation ("tail" events), divergent cyclone tracks, and the mesoscale kinetic energy spectra, highlighting avenues for future improvement. A comparison against a combination of in situ and remote sensing observations reveals forecast errors up to 15–45% larger than those relative to reanalysis and traditional forecasts, indicating that reanalysis-centric benchmarks can overstate forecast skill. Of the models assessed, ECMWF’s AIFS achieves the most consistent representation of atmospheric variables, with GraphCast and GenCast also showing strong skill. The analysis presents a framework for systematically evaluating global AI weather models on regional prediction, and the results highlight both the promise and current limitations of AI weather prediction in data-sparse regions. This underscores the importance of observation-based evaluation for future operational adoption. Introduction Recent progress in global weather prediction has been dominated by the emergence of artificial intelligence (AI)-driven weather prediction models. Advances in AI have inspired the development of fast and inexpensive global weather emulators that reduce a complex numerical task to a data-driven task that can be accomplished within minutes on a single GPU, i.e., up to a factor of 10,000-50,000 faster. Pioneering efforts in this rising field include [ 1 , 2 ] , DLWP [ 3 ] , FourCastNet [ 4 ] , Pangu-Weather [ 5 ] , and GraphCast [ 6 ] . More recent models including, but not limited to, GenCast [ 7 ] , NeuralGCM [ 8 ] , CorrDiff [ 9 ] , and AIFS [ 10 ] , have been developed to generate both deterministic and probabilistic (ensemble) forecasts. Concurrently, growing cognizance of the limited applicability and reanalysis-focused training of these models has also led to the creation of multiple foundation models, which aim to be used (in addition to weather forecasting) for multiple “downstream" applications, including air quality prediction, burn scar detection, hurricane track and intensity prediction, wave prediction, etc. Some notable foundational models include Prithvi HLS [ 11 ] , Prithvi WxC [ 12 ] , Aurora [ 13 ] , and AtmoRep [ 14 ] . Practically all AI weather prediction models (hereafter AIWP models) are, either fully or in part, trained on the global reanalysis dataset, ERA5 [ 15 ] , produced by the European Centre for Medium-Range Weather Forecasts (ECMWF), owing to its high spatio-temporal resolution and multi-decadal coverage. Some models, including GraphCast and PanguWeather, have both default versions, trained on ERA5, and operational versions, which are subsequently fine-tuned on forecasts from ECMWF’s 9 km High Resolution Forecast Model (HRES). As an upside, this allows AIWP models to bypass numerical integrations of the equations of fluid flow, and instead, directly learn weather evolution from global weather data. However, as a downside, heavy reliance on a single dataset puts an upper ceiling on the trained AI model’s skill and resolution. Despite incorporating over 10^{7} observation points daily through sophisticated data assimilation algorithms, ERA5 relies on time-marching the earth system initialization state using a forecasting model. As a result, ERA5 itself could be influenced by data assimilation errors, parameterization drifts, and numerical biases present in the underlying model, i.e., ECMWF’s Integrated Forecasting System (IFS) [ 16 , 17 , 18 , 19 ] . Multiple recent deterministic AIWP models have outperformed conventional numerical weather prediction (NWP) models on key error metrics, including global root mean square error (RMSE), mean absolute error (MAE), and Anomaly Correlation Coefficient (ACC) (for instance, [ 10 ] ). Likewise, probabilistic models like GenCast demonstrate a better continuous ranked probability score (CRPS) than ECMWF’s IFS ensemble over all lead times [ 7 ] . Beyond testing on RMSE-like metrics, multiple studies have tested AIWP models’ performance around key extreme weather events, revealing a systematic underestimation of wind speeds during cyclones [ 20 ] , and reduced predictability around Category-5 cyclones [ 21 , 22 ] and extreme precipitation [ 23 ] . At the same time, conflicting conclusions from independent studies have also highlighted both their high skill in representing the large-scale physics [ 24 , 25 ] , and poor skill in representing the small-scale dynamics and surface physics [ 26 , 27 ] , making them potentially suitable for at least some scale-selective dynamical analysis. The inter-model variability among AIWP models becomes apparent when examining forecasts during dynamically active periods like the Monsoon. Figure 1 illustrates substantial differences in day-ahead surface temperature predictions across seven state-of-the-art AIWP models during peak monsoon conditions over the Indian subcontinent on 15 July 2022. Forecast differences of several degrees Celsius are noted among the models, despite identical initialization using ERA5. The global differences are shown in Figure 11 . These disparities highlight the sensitivity of AI-based forecasting systems to architectural choices and training methodologies, while simultaneously revealing the inadequacy of reanalysis-based validation. Findings such as these inspire the need for continuing evaluation and benchmarking [ 28 , 29 ] of AIWP models over varied geographies and weather conditions. Most existing benchmarking initiatives test AIWP models against modeling output, leading to a circular paradigm where AIWP models are evaluated against the traditional model data they are trained on, obscuring their performance against actual ground-based and remote sensing observations that often drive operational decisions. This study focuses on evaluating the forecast of critical weather variables from seven advanced AIWP models against ground-based and remote-sensing observations (458 weather stations + rain gauge data + geostationary satellites). The models (as described in the Methods section) include FourCastNet, FourCastNet-SFNO, Pangu-Weather, GraphCast, Aurora, AIFS, and GenCast (both deterministic and ensemble versions). We compare both the global power spectra and regional forecasting errors over the Indian subcontinent during the South Asian Monsoon. This allows (a) connecting global model errors with regional model errors, and (b) testing model predictability in the tropics (where the dynamical evolution of the atmosphere is likely governed by an interplay of fast dynamics, moist convection, land-ocean coupling, and topographical effects) during one of the most dynamic periods of the year. The models are tested against observations of surface temperature, wind speeds, extreme precipitation, and cloud cover, to obtain an observationally validated sense of the models’ performance. Figure 1 : Systematic biases in AI weather models during peak monsoon conditions. Spatial distribution

Read the complete text