Can Tensor Cores Benefit Memory-Bound Kernels? (No!)
Lingqi Zhang 1 , Jiajun Huang 2 , Sheng Di 3 , Satoshi Matsuoka 1 , Mohamed Wahib 1 · 2025
Read Can Tensor Cores Benefit Memory-Bound Kernels? (No!) by Lingqi Zhang 1 , Jiajun Huang 2 , Sheng Di 3 , Satoshi Matsuoka 1 , Mohamed Wahib 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16851.txt.
Introduction
Lingqi Zhang and four co-authors ask whether tensor cores help memory bound GPU work in a 2025 paper. Tensor cores are the fast matrix units in modern graphics chips, stars of AI training. Some past reports claimed they also beat plain cores on work whose true limit is memory speed, not math, a claim this team set out to test hard.
Their math caps the gain at about a third in double precision for three chip lines. Three classic memory bound tasks then bore the cap out in live runs. Tuned tensor core code did not beat tuned plain core code in any sound way. The title gives their blunt answer: no. Memory bound work should spend its effort on the memory path.
Catalog record
| CREATOR | Lingqi Zhang 1 , Jiajun Huang 2 , Sheng Di 3 , Satoshi Matsuoka 1 , Mohamed Wahib 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-16851
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Tensor Cores Benefit Memory-Bound Kernels? (No!) Lingqi Zhang 1 , Jiajun Huang 2 , Sheng Di 3 , Satoshi Matsuoka 1 , Mohamed Wahib 1 Abstract . Tensor cores are specialized processing units within GPUs that have demonstrated significant efficiency gains in compute-bound applications such as Deep Learning Training by accelerating dense matrix operations. Given their success, researchers have attempted to extend tensor core capabilities beyond dense matrix computations to other computational patterns, including memory-bound kernels. Recent studies have reported that tensor cores can outperform traditional CUDA cores even on memory-bound kernels, where the primary performance bottleneck is not computation. In this research, we challenge these findings through both theoretical and empirical analysis. Our theoretical analysis reveals that tensor cores can achieve a maximum speedup of only 1.33 1.33 × over CUDA cores for memory-bound kernels in double precision (for V100, A100, and H100 GPUs). We validate this theoretical limit through empirical analysis of three representative memory-bound kernels-STREAM Scale, SpMV, and stencil. We demonstrate that optimizing memory-bound kernels using tensor cores does not yield sound performance improvements over CUDA cores. DOI: XXXXXXX.XXXXXXX Conference: General Purpose Processing on Graphics Processing Units; Mar 1, 2025; Las Vegas, NV ISBN: 978-1-4503-XXXX-X/18/06 1. Introduction Since Nvidia introduced tensor cores in their Volta architecture in 2017 ( Choquette et al., 2018 ) , researchers have extensively explored their applications across various domains, including dense linear algebra ( Bhaskaracharya et al., 2020 ) , sparse linear algebra ( Fan et al., 2024 ; Okanovic et al., 2024 ) , and spectral methods ( Durrani et al., 2021 ; Pisha and Ligowski, 2021 ) . The low-precision computation capabilities of tensor cores, which offer substantially higher performance, have spawned innovative approaches such as mixed-precision algorithms ( Haidar et al., 2018 ; Lu et al., 2024 ) and precision recovery ( Wu et al., 2024 ; Ootomo and Yokota, 2022 ; Ootomo et al., 2024 ) . However, despite this broad adoption, fundamental analysis of tensor core performance remains limited, with only a few studies focusing on microbenchmarking ( Sun et al., 2023 ; Luo et al., 2024 ; Abdelkhalik et al., 2022 ) . While tensor cores represent a powerful tool for program acceleration, their effective utilization requires a thorough understanding of their performance characteristics. This research addresses this knowledge gap by focusing specifically on memory-bound kernels, which constitute a significant portion of HPC workloads ( Austin et al., 2024 ) . Our study seeks to answer: • What is the theoretical performance ceiling for tensor cores when applied to memory-bound kernels? • Do current tensor core implementation strategies provide real performance benefits for memory-bound kernels? To address these questions, we make the following contributions: • A comprehensive theoretical analysis of tensor core performance for memory-bound kernels. • An empirical evaluation comparing tensor core implementations against their CUDA core counterparts across representative memory-bound kernels. The rest of the paper is organized as follows. Section 2 provides essential background concepts. Section 3 introduces our studied memory-bound workloads. Section 4 analyzes two extreme scenarios to establish tensor core performance bounds. Section 5 empirically substantiates our claims using representative memory-bound kernels. Finally, we present key takeaways and conclusions. 2. Background 2.1. Tensor Core Figure 1. Nvidia GPU memory hierarchy. Tensor Core is a specialized systolic array matrix engine ( Domke et al., 2021 ) integrated within the Stream Multiprocessor (SM) of modern Nvidia GPUs. It operates alongside traditional CUDA cores. The execution pathway for both units follows the GPU’s memory hierarchy: data is first loaded from global memory into the register file, from which either CUDA cores or Tensor Cores can access it for computation. This memory access abstraction aligns with previous studies ( Luo et al., 2024 ; Sun et al., 2023 ) . Figure 1 illustrates this memory hierarchy and the relationship between different memory levels in Nvidia GPUs. 2.2. Machine Balance We define machine balance ( \mathbb{B} ) ( McCalpin et al., 1995 ) as the ratio between peak computational performance ( P ) and memory bandwidth ( B ): (1) \footnotesize\mathbb{B}=\tfrac{P}{B} 2.3. Roofline Model The roofline model ( Williams et al., 2009 ; Ofenbeck et al., 2014 ) provides an upper-bound performance prediction framework based on operational intensity ( \mathbb{I} ), which is defined as the ratio of computational work ( \mathbb{W} ) to memory traffic ( \mathbb{Q} ): (2) \footnotesize\mathbb{I}=\tfrac{\mathbb{W}}{\mathbb{Q}} The model calculates attainable performance ( \mathbb{P} ) as: (3) \footnotesize\mathbb{P}=\min(P,B\times\mathbb{I}) This visualization framework helps identify system bottlenecks and performance limits. 2.4. Tensor Core in the Roofline Model As discussed in Section 2.1 , tensor cores can be represented as an additional performance ceiling above the CUDA core baseline in the roofline model, because tensor cores and CUDA cores share the same memory hierarchy and cannot operate simultaneously due to the Dark Silicon Effect. This roofline abstraction aligns with an existing study ( Yang et al., 2020 ) . 2.5. Relationship Between Machine Balance and Roofline Model The roofline model provides a framework for understanding performance bottlenecks through the interplay of two critical metrics: operational intensity and machine balance. While operational intensity ( \mathbb{I} ) characterizes a kernel’s computational density, machine balance ( \mathbb{B} ) represents the hardware’s ratio of computational capability to memory bandwidth. The relationship between these metrics determines whether a kernel’s performance is mainly limited by computation or memory access: (4) \footnotesize A\ kernel\ is=\begin{cases}compute-bound,&\text{if }\mathbb{I}>\mathbb{B}\\ memory-bound,&\text{if }\mathbb{I}<\mathbb{B}\end{cases} As illustrated in Figure 2 , machine balance manifests as the inflection point in the roofline curve for both GH200 and A100-80GB GPUs. This point marks the transition from memory-bound to compute-bound behavior. Figure 2. An example of the roofline model for both GH200 and A100-80GB GPU. 3. Workloads: Memory-Bound Kernels This section examines three representative memory-bound kernels: SCALE (Section 3.1 ), Sparse Matrix-Vector Multiplication (SpMV) (Section 3.2 ), and Stencil (Section 3.3 ). We analyze these kernels through the lens of operational intensity ( \mathbb{I} ), focusing on double-precision operations (data size \mathbb{D}=8 bytes). While our analysis centers on double precision, the methodology can be extended to lower-precision scenarios. 3.1. SCALE SCALE, one of the STREAM benchmark ( McCalpin, 2007 ) , is defined as (5) \footnotesize a_{i}=qb_{i},\hskip 8.50012pt\forall i\in{1,\ldots,n},\hskip 8.50012pta,b\in\mathbb{R}^{n},\hskip 8.50012ptq\in\mathbb{R} Each element operation requires one load, one store, and one computation, yielding: \mathbb{W}(\text{SCALE})=1 , \mathbb{Q}(\text{SCALE})=2\times\mathbb{D} , and consequently \mathbb{I}(\text{SCALE})=\tfrac{1}{16} . STREAM benchmark is commonly used to measure sustainable memory bandwidth due to its low computational intensity. 3.2. Sparse Matrix–Vector Multiplication (SpMV) SpMV, crucial for iterative solvers, has format-dependent operational intensity. In this section, we begin by analyzing dense matrix-vector multiplication (GEMV) as a baseline. GEMV: For matrix A\in\mathbb{R}^{m\times n} and vectors x\in\mathbb{R}^{n} , y\in\mathbb{R}^{m} , GEMV is defined as: (6) \footnotesize y=Ax With computing \mathbb{W}(\text{GEMV})=m\times n\times 2 operations and memory traffic \mathbb{Q}(\text{GEMV})=(m\times n+m+n)\times\mathbb{D} yields: (7) \footnotesize\mathbb{I}(\text{GEMV})=\frac{m\times n\times 2}{(m\times n+m+n)\times\mathbb{D}}\approx\frac{2}{\mathbb{D}}=\frac{1}{4} SpMV: For sparse matrices with nnz non-zeros, SpMV is defined as: (8) \displaystyle y \displaystyle=Ax,\ A\in\mathbb{R}^{m\times n},\hskip 8.50012pt\text{nnz}(A) \displaystyle\ll mn,\hskip 8.50012ptx\in\mathbb{R}^{n},\hskip 8.50012pty\in\mathbb{R}^{m} With computation \mathbb{W}(\text{SpMV})=2\times nnz and memory traffic including coordinate information \alpha\mathbb{I} or packed values \beta\mathbb{Z} : (9) \footnotesize\mathbb{I}(\text{SpMV})=\frac{nnz\times 2}{(nnz+m+n)\times\mathbb{D}+\alpha\mathbb{I}+\beta\mathbb{Z}} Given nnz\ll m\times n , we have \mathbb{I}(\text{SpMV})<\mathbb{I}(\text{GEMV}) . Compressed Sparse Row (CSR) format: CSR format, the most common sparse representation, requires storing column indices and row pointers. With memory