Mechanistic Interpretability Tool for AI Weather Models

✉Kirsten I. Tempest, Matthias Beylich, George C. Craig · 2026

Read Mechanistic Interpretability Tool for AI Weather Models by ✉Kirsten I. Tempest, Matthias Beylich, George C. Craig (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20467.txt.

Introduction

Tempest and two colleagues present an open-source tool bringing mechanistic interpretability to AI weather models. These models now rival traditional weather prediction, yet how they reach forecasts is poorly understood. The tool gathers latent representations from the model processor and supports early checks, including cosine similarity and PCA, so users can spot latent directions tied to weather features. Applying it to the graph neural network GraphCast, they show case studies for mid-latitude large-scale waves and specific humidity, finding latent channel mixes matching features a human can read.

Catalog record

CREATOR✉Kirsten I. Tempest, Matthias Beylich, George C. Craig
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27213
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20467

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mechanistic Interpretability Tool for AI Weather Models ✉Kirsten I. Tempest, Matthias Beylich, George C. Craig Abstract Artificial Intelligence (AI) weather models are improving rapidly, and their forecasts are already competitive with long-established traditional Numerical Weather Prediction (NWP). To build confidence in this new methodology, it is critical that we understand how these predictions are generated. This is a huge challenge as these AI weather models remain largely black boxes. In other areas of Machine Learning (ML), mechanistic interpretability has emerged as a framework for understanding ML predictions by analysing the building blocks responsible for them. Here we present an open-source, highly adaptable tool which incorporates concepts from mechanistic interpretability. The tool organises internal latent representations from the model processor and allows for initial analyses, including cosine similarity and Principal Component Analysis (PCA), enabling the user to identify directions in latent space potentially associated with meteorological features. Applying our tool to the graph neural network GraphCast, we present preliminary case studies for mid-latitude synoptic-scale waves and specific humidity. These demonstrate the tool’s ability to identify linear combinations of latent channels that appear to correspond to interpretable features. 1 Introduction Data-driven global weather forecast models developed in recent years are already competitive with traditional NWP, which has decades of research and development behind it. A major milestone was reached when AI models [ 7 , 2 ] surpassed the forecast skill of the world-leading European Centre for Medium-Range Weather Forecasts (ECMWF) High RESolution forecast (HRES) [ 10 ] . Recent developments include foundation models such as Aurora [ 3 ] , and models designed for probabilistic forecasting such as AIFS-CRPS [ 8 ] . The improvements in skill over time are tracked in the WeatherBench database [ 13 ] . The architectures of AI models and their resulting predictive capabilities are diverse. For example, GraphCast employs a graph neural network to produce a deterministic forecast, whereas AIFS-CRPS, a transformer-based model, generates ensemble members that sample a probability distribution. Despite these differences, these models share a common architectural framework. This consists of an encoding stage, a processor stage comprising multiple steps, including blocks that facilitate information exchange between locations, and a multilayer perceptron (MLP) operating on the latent space at each location, and finally a decoder stage which maps the learned representation back into gridded meteorological variables for humans to understand. This is quite different from conventional numerical integration, which evolves the physical variables through a sequence of time-steps to produce forecasts. Traditional NWP systems have large and complex codebases, which are built on our understanding of physical processes in the atmosphere, as well as in other Earth system components. AI weather models on the other hand are optimised to reproduce a training dataset, without regard to prior physical knowledge. By iteratively adjusting the weights (often millions, if not a billion as in Aurora [ 3 ] ) of the model during the training process, algorithms are created which allow the model to interpret input data and generate a meaningful output. This learning process allows the model to exploit any relevant relationships present in the weather data to be used in the production of the forecast, not just those programmed in. The trained model might make use of connections and concepts which are not yet understood or discovered by humans. This raises the question: how can we understand the internal workings of AI weather models? There have been various studies analysing AI weather models. In a recent survey [ 17 ] , these approaches have been grouped into post-hoc interpretability techniques, and those where the design of the model is inherently interpretable. The former, relevant for understanding a broad range of models without altering the model itself, includes perturbation-based methods, where the input features are systematically modified and the predictions examined. Game theory based and gradient-based attribution methods have also been attempted. Although these methods provide insight, they remain specific to scenarios and models; a comprehensive understanding of how predictions are made, and what connections are learned, is missing. This is vital to build trust for integration of such models into operational weather prediction, and might lead to new scientific discoveries. Mechanistic interpretability seeks to reverse engineer neural networks by understanding interpretable features and the circuits that connect them [ 11 ] . Features, which are meaningful patterns in the data, can be identified to correspond with directions in the latent space. By identifying features at different stages in the processor, it is possible to learn algorithms in the weights of the model which are used to get to the final output. For example, a vision model classifies an image of a dog’s head correctly by identifying eyes, snout, fur and tongue [ 12 ] . Mechanistic interpretability has been used as a framework in fields such as Large Language Models [ 18 , 6 ] , but has not yet been applied extensively to AI weather models. Initial work [ 9 ] has used sparse autoencoders to identify features in GraphCast, and has shown examples including surface heating in arid regions corresponding to directions in the latent space. This shows the potential of mechanistic interpretability techniques for understanding AI weather models. Here we present an exploratory visualisation tool that leverages mechanistic interpretability concepts to investigate the latent space of AI weather models. By providing a structured interface, the tool enables millions of data points to be efficiently organised and interpreted, facilitating rapid idea and hypothesis generation. The tool is open-source and has been designed for ease of customisation, allowing users to extend supported model configurations and incorporate additional analyses. Section 2 provides an overview of the tool design and methods, including the model and data used. Section 3 presents two case studies, one focusing on mid-latitude synoptic-scale waves and the other on specific humidity and how they correspond to directions in the latent space. The paper is then concluded in Section 4. 2 Design and Method Overview 2.1 Design We propose a visualisation tool that can be used by both meteorologists and computer scientists to identify complex connection pathways within AI weather models. This requires the structured organisation and analysis of millions of data points. In order to identify meaningful directions in the latent space, we begin by selecting a geographical region, one that contains a meteorologically relevant feature. Latent feature vectors within this region are then analysed. The tool comprises the following components: 1. Specify initial parameters. • Select the model configuration and forecast time ( t ). • Choose whether to apply the latent space translator. • Select T , the number of maximally activated latent channels to display and use in analysis. 2. Select location. • Choose the meteorological variable to be displayed on the two maps. The first map shows the input data at the forecast initialisation time, f(t_{\mathrm{init}} ), and the second shows the increment of the atmospheric state between initialisation and forecast time, f(t)-f(t_{\mathrm{init}} ). • Define a circular geographic region of interest. 3. Extract latent feature vectors and perform initial analysis. • Latent feature vectors are loaded for all processor steps and mesh nodes, and the mesh nodes within the selected area are identified. • The T most strongly activated latent channels are selected, and

Read the complete text