cuTeSpMM: Accelerating Sparse-Dense Matrix Multiplication using GPU Tensor Cores

Lizhi Xiang, Omid Asudeh, Gerald Sabin, Aravind Sukumaran-Rajam, P. Sadayappan · 2025

Read cuTeSpMM: Accelerating Sparse-Dense Matrix Multiplication using GPU Tensor Cores by Lizhi Xiang, Omid Asudeh, Gerald Sabin, Aravind Sukumaran-Rajam, P. Sadayappan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06443.txt.

Introduction

Lizhi Xiang, Omid Asudeh, Gerald Sabin, Aravind Sukumaran-Rajam, and Sadayappan wrote this paper in two thousand twenty-five, the last author named with an initial. It speeds up sparse-dense matrix multiplication on the tensor cores of modern GPUs.

Tensor cores chew small dense blocks at vast rates, but filling a sparse matrix with zeros to suit them has mostly lost to plain cores. The team's kernel, cuTeSpMM, answers yes for a broad span of real sparse matrices. A new gauge, TCU-Synergy, reads a matrix's nonzero shape to foretell the gain. For high-synergy matrices the kernel beats the best plain-core code; for low ones it stays close.

Catalog record

CREATORLizhi Xiang, Omid Asudeh, Gerald Sabin, Aravind Sukumaran-Rajam, P. Sadayappan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06443

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

cuTeSpMM: Accelerating Sparse-Dense Matrix Multiplication using GPU Tensor Cores Lizhi Xiang, Omid Asudeh, Gerald Sabin, Aravind Sukumaran-Rajam, P. Sadayappan Abstract . Many recent GPUs feature matrix multiplication engines (aka Tensor Core Units or TCUs) that perform small fixed-size matrix-matrix products at very high throughput. They have been used very effectively to speed up dense matrix-matrix multiplication libraries like Nvidia’s cuBLAS, enabling significantly higher performance over use of the traditional scalar GPU cores. There also been recent interest in using these dense TCUs for the important sparse-dense matrix-matrix multiplication (SpMM) kernel via explicit zero-filling. However, an examination of the attainable performance of TC-GNN, the state-of-the-art TCU-enhanced SpMM implementation, indicates that for a substantial majority of the sparse matrices in the SuiteSparse collection, the achieved performance falls significantly short of the state-of-the-art SpMM kernels that only utilize scalar cores. In this paper, we therefore address the question: Can dense TCUs be effectively used to accelerate SpMM for a range of sparse matrices arising from multiple application domains, such as those found in the SuiteSparse matrix collection? We answer this question in the affirmative by developing a very efficient TCU-based GPU kernel - cuTeSpMM (cuda Tensor core SpMM) that achieves substantially higher performance over TC-GNN. We also develop a notion of the TCU-Synergy of a sparse-matrix, based on its non-zero structure and a modeled Operational Intensity. For sparse matrices with high TCU-synergy, cuTeSpMM outperforms state-of-the-art scalar-core SpMM implementations, while achieving only slightly lower performance on matrices with low TCU-Synergy. Conference: ; 2025; Salt Lake City, USA 1. Introduction Emilia\_923 NotreDame\_www Figure 1 . Nonzero patterns of two sparse matrices from the SuiteSparse matrix collection ( 1 , 2 ) Matrix engines are now being incorporated in several recent CPUs (e.g., IBM Power 10 ( 3 ) , Intel Sapphire Rapids ( 4 ) ) and GPUs (e.g., Nvidia Ampere ( 5 ) and Hopper ( 6 ) , AMD MI250 and MI300 ( 7 ) ). These matrix engines enable very fast multiplication of small fixed-size matrices through new instruction set extensions and direct hardware implementation for fixed-size matrix multiplication. These matrix engines, also called Tensor Cores, have been used to significantly enhance the performance of vendor libraries like cuBLAS ( 8 ) that implement dense linear algebra operations. Can these high throughput, fixed size, dense tensor core hardware implementations be used for sparse-dense matrix multiplication (SpMM)? While sparse matrices used in scientific computing and data analytics tend to be extremely sparse (with far less than 1% of elements being non-zero), the distribution of non-zeros is generally not uniformly random across the 2D index space. The non-zero distribution pattern tends to exhibit clustering in sub-regions of the 2D index space, which leaves open the possibility of developing zero filling strategies to leverage these dense TC Units. Fig. 1 shows the non-zero pattern for two sparse matrices from the SuiteSparse matrix collection ( 1 , 2 ) . The matrix Emilia\_923 arose from a structural application and has 923136 rows/columns and 41005206 nonzero elements (i.e., it is extremely sparse with a density of less than 0.005\% ). However the non-zero elements tend to be clustered around the diagonal. If the 2D 923136\times 923136 index space is partitioned into a regular grid of 16\times 4 “bricks” (this is one of the supported dense matrix sizes in Nvidia’s WMMA API for using its tensor cores for matrix-matrix multiplication), the average number of non-zero elements in non-empty bricks is nearly 20% dense. NotreDame\_www is a web graph with 325760 rows/columns and 929849 nonzero elements, with an extremely low density of less than 0.0009\% . Its non-zeros are scattered over a much wider range of the 2D index space, but even for this much sparser and more scattered matrix, the average density of non-zero elements in non-empty 16\times 4 bricks is nearly 10%. Therefore, there is the potential to develop algorithms to leverage dense tensor core hardware to accelerate sparse matrix computations. The examples above illustrate that although sparse matrices arising in applications from scientific computing and data analytics may be extremely sparse, the non-zero patterns can be concentrated in subregions of the 2D index space. As explained later, further compaction into “bricks” corresponding to the matrix operand sizes of TCU instructions can further increase the density of non-zeros in non-empty bricks. Thus, there has been interest in developing SpMM implementations that utilize TCUs ( 9 , 10 , 11 ) . TC-GNN ( 10 ) is a recently developed TCU-based implementation of Graph Neural Networks (GNNs), where the dominant computational operation is SpMM. Wang et al. ( 10 ) demonstrated their TCU-based SpMM implementation (which we refer to simply as TC-GNN in this paper instead of TC-GNN-SpMM) to be superior to other TCU-based SpMM implementations. Since SpMM comparisons with other scalar-core (SC) based GPU implementation of SpMM were not performed by Wang et al. ( 10 ) , we carried out experimentation on around 1000 matrices (all matrices with more than 10,000 rows) from the SuiteSparse collection ( 2 , 1 ) . We compare the performance of TC-GNN with the fastest scalar-core implementation (best among cuSparse ( 12 ) , GeSpMM ( 13 ) , Sputnik ( 14 ) ) on this wide range of sparse matrices. Fig. 2 contains two scatter-plots depicting performance in TFLOPs achieved by TC-GNN vs. the best scalar-core SpMM (labeled best-SC). The top chart shows performance on an Nvidia RTX 4090 GPU, while the bottom chart shows performance on an NVidia Ampere A100 GPU. The width of the dense matrices was 128. The peak scalar-core 32-bit floating-point performance (FP32) on the A100 (4090) is 19.2 TF (82.6 TF), while the TCU peak performance (TF32) is 156 TF (A100) and 82.6 TF (4090). On the 4090 GPU, the performance of TC-GNN is worse than best-SC except for a handful of matrices. On the surface, this is not entirely surprising as the peak TCU throughput is equal to the peak SC throughput on the RTX 4090. Therefore, the handful of TCU “wins" must arise from other architectural and algorithm design differences. In contrast, the A100 has an 8x higher peak TCU throughput as compared to the A100 peak scalar-core throughput. Despite the significantly higher peak TCU performance of the A100, the performance of TC-GNN is relatively worse when compared to best-SC; TC-GNN was not faster on any of the 1000+ matrices! For these matrices, TC-GNN is not able to exploit the 8x higher peak throughput of the TCU. Given these experimental results on the state-of-the-art TC-GNN SpMM, we seek to answer the following two questions in this paper: (1) Is it possible to design and implement a TCU-based GPU SpMM that is actually faster than the best scalar-core SpMM on a non-trivial fraction of SuiteSparse matrices? (2) Can some inherent characteristics of a sparse matrix be identified that indicate whether it has good potential for a faster TCU-based SpMM implementation than the best-SC SpMM? Figure 2 . TC-GNN performance versus the best SC performance (cuSparse ( 12 ) , GeSpMM ( 13 ) , Sputnik ( 14 ) ). Left: Ampere a100, right:RTX-4090 In this paper, we successfully address both these questions. We develop a new TCU-based implementation of SpMM - cuTeSpMM (cuda

Read the complete text