Accelerating High-Order Finite Element Simulations at Extreme Scale with FP64 Tensor Cores

Jiqun Tu 1 , Ian Karlin 2 , John Camier 3 , Veselin Dobrev 3 , Tzanio Kolev 3 , Stefan Henneking 4 , Omar Ghattas 4 · 2026

Read Accelerating High-Order Finite Element Simulations at Extreme Scale with FP64 Tensor Cores by Jiqun Tu 1 , Ian Karlin 2 , John Camier 3 , Veselin Dobrev 3 , Tzanio Kolev 3 , Stefan Henneking 4 , Omar Ghattas 4 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09038.txt.

Introduction

Jiqun Tu and colleagues accelerate high-order finite element simulations with FP64 tensor cores. High resolutions demand high-order methods and supercomputing, and GPU ports still leave efficiency and speed gains on the table. Integrating tensor cores with kernel fusion, they gain up to twice the performance and up to 83 percent energy efficiency on Grace Hopper and Grace Blackwell chips. Scaling holds near-perfect for weak scaling and 90 percent for strong scaling across nearly 10,000 GPUs.

Catalog record

CREATORJiqun Tu 1 , Ian Karlin 2 , John Camier 3 , Veselin Dobrev 3 , Tzanio Kolev 3 , Stefan Henneking 4 , Omar Ghattas 4
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51070
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.09038

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Accelerating High-Order Finite Element Simulations at Extreme Scale with FP64 Tensor Cores Thanks: This research was supported by DARPA COMPASS HR0011-25-3-0242, DOD MURI grant FA9550-24-1-0327 and DOE ASCR grant DE-SC0023171. Work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 (LLNL-PROC-2014338). This research used resources from the Swiss National Supercomputing Centre (CSCS) and the National Energy Research Scientific Computing Center (NERSC; ALCC-ERCAP0030671, ScienceAtScale DDR-ERCAP0034808 and NESAP DDR-ERCAP0038013). Jiqun Tu 1 , Ian Karlin 2 , John Camier 3 , Veselin Dobrev 3 , Tzanio Kolev 3 , Stefan Henneking 4 , Omar Ghattas 4 Abstract Finite element simulations play a critical role in a wide range of applications, from automotive design to tsunami modeling and computational electromagnetics. Performing these simulations efficiently at the high resolutions needed for practical applications and scientific insights necessitates the use of high-order methods and large-scale supercomputing. While much progress has been made in porting finite element codes to GPU systems in recent years, additional improvements in the efficiency and computational speed of GPU-accelerated high-order finite element simulations are in constant demand. In this paper, we demonstrate that the FP64 tensor cores on NVIDIA GPUs can be used to further accelerate such simulations, achieving significant speedups in key kernels of MFEM, a scalable open-source finite element library widely used in HPC applications. By integrating FP64 tensor cores with kernel fusion optimizations, we were able to achieve up to 2 × \times performance gains and up to 83% energy efficiency gains on NVIDIA’s Grace Hopper GH200 and Grace Blackwell GB200 architectures. To the best of our knowledge, this is the first time that FP64 tensor cores have been directly programmed to accelerate large-scale finite element scientific computing applications. We demonstrate the performance of the optimized kernels at exascale by showing near-perfect weak scaling efficiency and 90% strong scaling efficiency across nearly 10,000 GPUs on the Alps system. The new algorithms and MFEM enhancements directly benefit complex production codes, including the 2025 Gordon Bell Prize-winning application for real-time tsunami forecasting. I Introduction Tensor cores were introduced in 2017 in the NVIDIA Volta V100 GPU architecture in order to drive higher performance in applications that use dense matrix multiplication. Initially, the tensor cores supported only mixed-precision (FP16, FP32) matrix-multiply-accumulate (MMA) computations where single-precision results are achieved through a combination of reduced-precision (FP16) multiplication and single-precision (FP32) accumulation [ 1 , Fig. 9] . As specialized hardware for MMA instructions, tensor cores delivered dramatically higher throughput for these workloads than standard FP16 and FP32 CUDA cores. For single-precision performance, tensor cores provided up to 8 \times higher FLOP/s performance than CUDA cores on V100 [ 2 ] , and the gains have since increased to over 15 \times on current Blackwell-class GPUs [ 3 ] with even larger speedups for lower-precision (FP4, FP8) computations. While many codes can benefit directly from the performance gains of lower-precision or mixed-precision tensor core computations, there are also many scientific applications that require the full accuracy of double precision computations. Some examples include singular perturbation problems, solutions with geometric singularities or boundary layers, and multiscale physics problems. In this paper, we apply tensor cores to one such application: an inverse problem for finite element-discretized wave propagation that aims to infer a high-dimensional parameter field from sparse, noisy data. This type of inverse problem is ill-posed [ 4 ] , making the solution sensitive to small errors; stable inversion necessitates double precision calculations. Starting in 2020 with the Ampere A100 GPU, NVIDIA introduced FP64 tensor cores supporting double precision MMA (DMMA) instructions. For double precision calculations, these tensor cores deliver a 2 \times increase in peak floating-point rates [ 5 ] . In recent years, other vendors have introduced tensor cores, or analogous concepts, in their compute architectures. In this paper, we focus specifically on the programming of tensor cores on two of NVIDIA’s state-of-the-art chips: the Grace Hopper GH200 Superchip [ 6 ] and the Grace Blackwell GB200 Superchip [ 3 ] . For applications that use FP64 dense linear algebra—specifically, General Matrix-Matrix Multiplication (GEMM)—for large matrices, including density functional theory, tensor-core usage has led to large performance gains [ 7 ] . This has been exploited both in scientific applications (e.g., BerkeleyGW [ 8 ] ) and in benchmarking codes (e.g., HPL [ 9 ] ) which typically utilize NVIDIA’s FP64 tensor cores by calling the GEMM APIs in CUBLAS and other vendor libraries to achieve significant speedups for large GEMMs. However, many applications, in diverse fields such as molecular dynamics, computational engineering, and weather modeling, have not been able to leverage tensor cores as successfully [ 10 ] because they either do not perform matrix-matrix multiplies or their operations do not map well to tensor units. More recently, tensor core computations have been extended to more applications through direct programming, including stencil codes [ 11 ] and finite element methods [ 12 ] . We build on this prior work and apply tensor cores to a full finite element production application—a digital twin for tsunami early warning [ 13 ] , where the coupled acoustic–gravity wave propagation is solved as a first-order system of partial differential equations (PDEs) discretized with high-order finite elements. Specifically, we decompose the tensor contractions into small GEMMs of order \mathcal{O}(10) and program the FP64 tensor cores directly in CUDA kernels. We show how directly programming tensor cores on state-of-the-art chips like NVIDIA’s Grace Hopper GH200 and Grace Blackwell GB200 can lead to significant performance and energy efficiency gains in a complex application code. Using the tensor-core-optimized kernels, we then demonstrate excellent full-system scalability of the application code on one of the world’s largest supercomputers—the Alps system at the Swiss National Supercomputing Centre (CSCS). The main contributions of this paper are: • Detailed design and analysis of the programming and optimization of FP64 tensor cores for irregularly shaped matrix multiplies. • To the best of our knowledge, the first example of using directly programmed FP64 tensor cores in a complex, PDE-based HPC application code, in order to improve throughput of high-order finite element kernels (by up to 59%). • Performance comparison of GH200 and GB200 FP64 tensor cores including energy efficiency analysis (27% improvement), which has not been reported for small matrix tensor core operations before. • Demonstration of full-system scalability of the application code for each of the implemented kernel versions on a leading-edge supercomputer. The rest of the paper is organized as follows. Section II provides a brief overview of the MFEM library, the finite element algorithms and application used, and the NVIDIA tensor cores. We then discuss how we optimized the key finite element kernels in Section III . Section IV shows the performance and energy efficiency improvements from our optimizations. In Section V and Section VI , respectively, we describe related work and summarize our conclusions. II Background II-A MFEM library In order to make our techniques more accessible, we focus our work on finite element kernels in the MFEM library [ 14 ] , an open-source high-performance finite element library that is widely used in the scientific computing community. MFEM is designed for solving PDEs via a variety of discretizations on general unstructured grids,

Read the complete text