An Efficient Sparse Kernel Generator for O(3)-Equivariant Deep Networks

Vivek Bharadwaj, Austin Glover, Aydın Buluç, James Demmel · 2025

Read An Efficient Sparse Kernel Generator for O(3)-Equivariant Deep Networks by Vivek Bharadwaj, Austin Glover, Aydın Buluç, James Demmel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13986.txt.

Introduction

Vivek Bharadwaj, Austin Glover, Aydın Buluç, and James Demmel speed up a core math step in geometric deep learning in a 2025 paper. Networks that respect rotations lead their field on spatial tasks such as atom modeling, and their key step joins two feature sets through a sparse tensor named for Clebsch and Gordan. That step runs millions of times and had become the bottleneck.

Their graphics-chip code generator plans memory use at build time, keeps working data in fast registers, and fuses the step with the graph operation that follows it. The kernels beat both the best open package and a vendor package, with the largest gains on a chemistry model at high precision. Code for the step's gradients is included.

Catalog record

CREATORVivek Bharadwaj, Austin Glover, Aydın Buluç, James Demmel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-13986

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Efficient Sparse Kernel Generator for O(3)-Equivariant Deep Networks Vivek Bharadwaj, Austin Glover, Aydın Buluç, James Demmel Abstract Rotation equivariant graph neural networks, i.e. networks designed to guarantee certain geometric relations between their inputs and outputs, yield state of the art performance on spatial deep learning tasks. They exhibit high data efficiency during training and significantly reduced inference time for interatomic potential calculations compared to classical approaches. Key to these models is the Clebsch-Gordon (CG) tensor product, a kernel that contracts two dense feature vectors with a highly-structured sparse tensor to produce a dense output vector. The operation, which may be repeated millions of times for typical equivariant models, is a costly and inefficient bottleneck. We introduce a GPU sparse kernel generator for the CG tensor product that provides significant speedups over the best existing open and closed-source implementations. Our implementation achieves high performance by carefully managing the limited GPU shared memory through static analysis at model compile-time, minimizing reads and writes to global memory. We break the tensor product into a series of smaller kernels with operands that fit entirely into registers, enabling us to emit long arithmetic instruction streams that maximize instruction-level parallelism. By fusing the CG tensor product with a subsequent graph convolution , we reduce both intermediate storage and global memory traffic over naïve approaches that duplicate input data. We also provide optimized kernels for the gradient of the CG tensor product and a novel identity for the higher partial derivatives required to predict interatomic forces. Our kernels offer up to 1.3x speedup for the over NVIDIA’s closed-source cuEquivariance package, as well as > 10 >10 x speedup over the widely-used e3nn package. In FP64 precision, we offer up to 6.2x inference-time speedup for the MACE chemistry foundation model over the original unoptimized version. 1 Introduction Equivariant deep neural network models have become immensely popular in computational chemistry over the past seven years [ 32 , 34 , 20 ] . Consider a function \bm{f}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m} . Informally, \bm{f} is invariant if a class of transformations applied to its argument results in no change to the function output. A function is equivariant if a transformation applied to any input argument of \bm{f} can be replaced by a compatible transformation on the output of \bm{f} . For example: a function predicting molecular energy based on atomic positions should not change its result if the atom coordinates are rotated, translated, or reflected (invariance). Likewise, a function predicting 3D forces on point masses should rotate its predictions if the input coordinate system rotates (equivariance). The latter property is termed rotation equivariance , and it is the focus of our work. Rotation equivariant neural architectures appear in the AlphaFold [ 14 ] version 2 model for protein structure prediction, the DiffDock [ 7 ] generative model for molecular docking, and the Gordon Bell finalist Allegro [ 26 ] for supercomputer-scale molecular dynamics simulation, among a host of other examples [ 29 , 1 , 4 , 3 , 18 ] . Figure 1: The CG tensor product, which contracts a block-sparse tensor \mathcal{P} with two dense vectors to output a new vector. It is usually followed by multiplication with a structured weight matrix \bm{W} , and by convention we use “CG tensor product” to refer to both operations in sequence. Each blue block is itself sparse (see Figure 2 ), and several blocks may share identical structure. A core kernel in many (though not all) rotation equivariant neural networks is the Clebsch-Gordon (CG) tensor product, which combines two feature vectors in an equivariant model to produce a new vector [ 32 ] . This multilinear operation, illustrated in Figure 1 , contracts a highly-structured block sparse tensor with a pair of dense input vectors, typically followed by multiplication with a structured weight matrix. It is frequently used to combine node and edge embeddings in equivariant graph neural networks, which are used for molecular energy prediction in computational chemistry [ 4 , 2 , 26 ] (see Figure 4 ). With its low arithmetic intensity and irregular computation pattern, the CG tensor product is difficult to implement efficiently in frameworks like PyTorch or JAX. Because the CG tensor product and its derivatives must be evaluated millions of times on large atomic datasets, they remain significant bottlenecks to scaling equivariant neural networks. We introduce an open source kernel generator for the Clebsch-Gordon tensor product on both NVIDIA and AMD GPUs. Compared to the popular e3nn package [ 10 ] that is widely used in equivariant deep learning models, we offer up to one order of magnitude improvement for both forward and backward passes. Our kernels also exhibit up to 1.0-1.3x speedup over NVIDIA’s closed-source cuEquivariance v0.4.0 package [ 11 ] on configurations used in graph neural networks, although our second derivative kernel is slower by 30% on certain inputs. Our key innovations include: Exploiting ILP and Sparsity: Each nonzero block of \mathcal{P} in Figure 1 is a structured sparse tensor (see Figure 2 for illustrations of the nonzero pattern). Popular existing codes [ 10 ] fill these blocks with explicit zeros and use optimized dense linear algebra primitives to execute the tensor product, performing unnecessary work in the process. By contrast, we use Just-in-Time (JIT) compilation to generate kernels that only perform work for nonzero tensor entries, achieving significantly higher throughput as block sparsity increases. While previous works [ 19 , 17 ] use similar fine-grained approaches to optimize kernels for each block in isolation, we achieve high throughput by JIT-compiling a single kernel for the entire sparse tensor \mathcal{P} . By compiling kernels optimized for an entire sequence of nonzero blocks, we exhibit a degree of instruction level parallelism and data reuse that prior approaches do not provide. Static Analysis and Warp Parallelism: The structure of the sparse tensor in Figure 1 is known completely at model compile-time and contains repeated blocks with identical nonzero patterns. We perform a static analysis on the block structure immediately after the equivariant model architecture is defined to generate a computation schedule that minimizes global memory traffic. We break the calculation into a series of subkernels (see Figure 3 ), each implemented by aggressively caching \bm{x} , \bm{y} , and \bm{z} in the GPU register file. We adopt a kernel design where each GPU warp operates on distinct pieces of coalesced data and requires no block-level synchronization (e.g. __syncthreads() ). To accomplish this, each warp manages a unique portion of the shared memory pool and uses warp-level matrix multiplication primitives to multiply operands against nonzero blocks of the structured weight matrix. Fused Graph Convolution: We demonstrate benefits far beyond reduced kernel launch overhead by fusing the CG tensor product with a graph convolution kernel (a very common pattern [ 32 , 4 , 2 ] ). We embed the CG tensor product and its backward pass into two algorithms for sparse-dense matrix multiplication: a simple, flexible implementation using atomic operations and a faster deterministic version using a fixup buffer. As a consequence, our work is the first to reap significant model memory savings, a reduction in global memory writes, and data reuse at the L2 cache level. Section 2 provides a brief introduction to equivariant neural

Read the complete text