Attention on the Sphere

Boris Bonev, Max Rietmann, Andrea Paris, Alberto Carpentieri, Thorsten Kurth · 2025

Read Attention on the Sphere by Boris Bonev, Max Rietmann, Andrea Paris, Alberto Carpentieri, Thorsten Kurth (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11157.txt.

Introduction

Bonev and four co-authors design attention that works natively on the sphere for transformer models, in a paper from twenty twenty-five. Much world data, from weather to cosmology, lives on a sphere. Flat attention distorts distances and breaks spherical symmetry. Their attention folds in quadrature weights to stay faithful to sphere geometry.

The result nearly preserves rotational symmetry, giving a strong learning bias. A neighborhood variant limits attention to nearby points on the sphere. That cuts computing cost and adds a locality bias while keeping symmetry. Custom fast kernels make the method practical at scale. Tests on fluid simulation, segmentation, and depth estimation all favor spherical models over flat ones.

Catalog record

CREATORBoris Bonev, Max Rietmann, Andrea Paris, Alberto Carpentieri, Thorsten Kurth
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11157

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention on the Sphere Boris Bonev, Max Rietmann, Andrea Paris, Alberto Carpentieri, Thorsten Kurth Abstract We introduce a generalized attention mechanism for spherical domains, enabling Transformer architectures to natively process data defined on the two-dimensional sphere - a critical need in fields such as atmospheric physics, cosmology, and robotics, where preserving spherical symmetries and topology is essential for physical accuracy. By integrating numerical quadrature weights into the attention mechanism, we obtain a geometrically faithful spherical attention that is approximately rotationally equivariant, providing strong inductive biases and leading to better performance than Cartesian approaches. To further enhance both scalability and model performance, we propose neighborhood attention on the sphere, which confines interactions to geodesic neighborhoods. This approach reduces computational complexity and introduces the additional inductive bias for locality, while retaining the symmetry properties of our method. We provide optimized CUDA kernels and memory-efficient implementations to ensure practical applicability. The method is validated on three diverse tasks: simulating shallow water equations on the rotating sphere, spherical image segmentation, and spherical depth estimation. Across all tasks, our spherical Transformers consistently outperform their planar counterparts, highlighting the advantage of geometric priors for learning on spherical domains. \usephysicsmodule ab,doubleprod 1 Introduction The two-dimensional sphere embedded in three dimensions ( S^{2}) plays a crucial role in a variety of scientific and engineering domains such as geophysics, cosmology, chemistry, and computer graphics. All of these domains require the processing of functions defined on the surface of the sphere. With the advent and increased success of machine-learning in these domains, there is a growing demand to generalize state-of-the-art architectures to spherical data [ 12 , 16 , 17 , 30 , 6 ] . This is particularly important in scientific applications which require the solution of time-dependent partial differential equations which are often approximated with auto-regressive models. These models are particularly sensitive to spurious artifacts which arise from a non-geometrical treatment as these tend to build up and compromise stability [ 6 ] . While spherical convolutional neural networks (CNNs) and other geometric deep learning models have advanced the state of the art for learning on the sphere by respecting its symmetries and topology, these architectures are primarily designed for capturing local interactions and equivariant representations [ 12 , 30 , 26 ] . However, many scientific and engineering problems on the sphere, such as weather and climate modeling, 360°perception, and cortical surface analysis, require modeling complex, long-range dependencies that are not efficiently handled by convolutional approaches [ 7 , 3 , 8 ] . Transformers, with their global attention mechanism, have revolutionized learning in Euclidean domains by enabling flexible, data-driven modeling of both local and global relationships [ 13 , 34 , 4 , 21 ] . Extending Transformers to spherical domains promises to combine the geometric fidelity of spherical models with the expressive power of attention, enabling new capabilities for spherical data analysis. Recent works have begun to explore this direction, demonstrating that Transformer-based models tailored to spherical geometry can outperform traditional methods on tasks such as depth estimation or semantic segmentation by directly leveraging spherical geometry in their design [ 3 , 9 ] . However, due to their reliance on permutation equivariance on icosahedral grids, they fail to capture the full rotational equivariance associated with the sphere. Other works have successfully applied Euclidean Transformer architectures to weather forecasting on the sphere [ 4 ] , however, they show deterioration in longer auto-regressive rollouts [ 23 ] . This motivates the development of spherical attention mechanisms that operate natively on the sphere, unlocking the full potential of Transformer architectures for scientific and engineering applications involving spherical data. Our contribution We generalize the attention mechanism to the spherical domain. To respect rotational symmetry, we derive a continuous formulation on the sphere for both global attention and neighborhood attention mechanisms. A discrete formulation is then derived using quadrature rules to approximate kernel integrals, thus ensuring approximate equivariance w.r.t. three-dimensional rotations. Both spherical attention variants are implemented in PyTorch, including custom CUDA extensions of the spherical neighborhood attention. We demonstrate the effectiveness of our approach by generalizing standard Transformer architectures to the spherical domain using our formulation and scoring them against their Euclidean counterparts in three different spherical benchmark problems. Our implementation and code to reproduce experiments is available in the torch-harmonics library. 2 Related work Transformers Attention mechanism and Transformer architectures were initially developed for sequence modeling applications such as language models [ 33 ] and later applied to vision tasks in the Euclidean domain [ 13 ] . This has proven to be an effective method despite the lack of inductive biases such as locality [ 4 , 21 ] . The attention mechanism is equivariant with respect to permutations of the input tokens and therefore lacks any notion of the underlying topology of the domain. This makes position embeddings necessary, which introduce the notion of locality. A notable advantage of the attention mechanism is its ability to capture long-range interactions. However, this makes it prohibitively expensive for large resolutions due to its \mathcal{O}(N^{2}) asymptotic complexity, especially for multi-dimensional data where the number of pixels N grows exponentially with dimension limiting the resolution. Neighborhood Transformers Hassani et al. 2023 address this by simultaneously introducing the notion and inductive bias of locality into the architecture. This is achieved by using the neighborhood attention mechanism. In this approach, the attention layer is constrained to a neighborhood of grid points around the query point. This has great computational benefits as it lowers the asymptotic complexity of the attention mechanism to \mathcal{O}(kN) where k is the kernel size of the neighborhood. While this method is highly efficient [ 20 ] , it is generally only formulated for equidistant Cartesian grids. For spherical problems, this leads to distorted neighborhoods on spherical geometries. Graph based Transformers To address the above, Transformer architectures operating on icosahedral discretizations of the sphere have been formulated [ 9 , 8 , 3 ] . While this has proven to be an effective approach for dealing with spherical data, it is based on the regular attention mechanism, making it permutation equivariant on the graph. As such, the resulting architectures do not take into account the rotational symmetry of the sphere, but rather discrete permutations on the resulting graph. Equivariant architectures While several convolutional architectures address this and capture rotational symmetry [ 11 , 15 , 12 , 10 , 30 , 6 ] , it has not been addressed by Transformer architectures yet. Finally, Assaad et al. 2022 introduce a rotationally equivariant Transformer architecture in the context of vector neurons using the Frobenius norm. 3 Method (a) Euclidean (b) naive spherical (c) weighted spherical Figure 1 : Neighborhood attention on the sphere: (a) Euclidean neighborhoods use planar distances, distorting receptive fields near poles. (b) Naive spherical attention uses geodesic neighborhoods but equally weights points, oversampling points and breaking \mathrm{SO}(3) symmetry. (c) Our method introduces quadrature weights to account for non-uniform sampling density, preserving rotational equivariance regardless of grid alignment through proper discretization of the continuous formulation. 3.1 Attention mechanism and kernel regression We revisit the motivation behind the original attention mechanism. For two one-dimensional signals q,k:\mathbb{R}^{+}\rightarrow\mathbb{R} , we can compute the correlation function C[q,k](x,x^{\prime})=\mathrm{corr}(q(x),k(x^{\prime})) using the Pearson

Read the complete text