Performance-Portable Optimization and Analysis of Multiple Right-Hand Sides in a Lattice QCD Solver

Shiting Long 1 , Gustavo Ramirez-Hidalgo 2 , Stepan Nassyr 2 , Jose Jimenez-Merchan 3 , Andreas Frommer 3 , Dirk Pleiter 1,4 · 2026

Read Performance-Portable Optimization and Analysis of Multiple Right-Hand Sides in a Lattice QCD Solver by Shiting Long 1 , Gustavo Ramirez-Hidalgo 2 , Stepan Nassyr 2 , Jose Jimenez-Merchan 3 , Andreas Frommer 3 , Dirk Pleiter 1,4 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05816.txt.

Introduction

Shiting Long and 5 co-authors speed up a lattice QCD solver in 2026 by solving for several right-hand sides at once. Lattice QCD simulates quark and gluon physics on a space-time grid, and its iterative solvers dominate computing budgets. Memory bandwidth, not raw arithmetic, is often the binding constraint.

The team extended the DD-alphaAMG solver to handle multiple right-hand sides in both operator evaluation and the GMRES solver, with and without a standard preconditioning trick. A flexible interface supports different data layouts, including one tuned for vector instructions on modern processors. Tests on x86 and Arm clusters showed similar speedups, demonstrating performance portability. Their analysis also documents how architecture and compiler behavior complicate such optimizations.

Catalog record

CREATORShiting Long 1 , Gustavo Ramirez-Hidalgo 2 , Stepan Nassyr 2 , Jose Jimenez-Merchan 3 , Andreas Frommer 3 , Dirk Pleiter 1,4
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-05816

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Performance-Portable Optimization and Analysis of Multiple Right-Hand Sides in a Lattice QCD Solver Shiting Long 1 , Gustavo Ramirez-Hidalgo 2 , Stepan Nassyr 2 , Jose Jimenez-Merchan 3 , Andreas Frommer 3 , Dirk Pleiter 1,4 Abstract Managing the high computational cost of iterative solvers for sparse linear systems is a known challenge in scientific computing. Moreover, scientific applications often face memory bandwidth constraints, making it critical to optimize data locality and enhance the efficiency of data transport. We extend the lattice QCD solver DD- α \alpha AMG to incorporate multiple right-hand sides (rhs) for both the Wilson-Dirac operator evaluation and the GMRES solver, with and without odd-even preconditioning. To optimize auto-vectorization, we introduce a flexible interface that supports various data layouts and implement a new data layout for better SIMD utilization. We evaluate our optimizations on both x86 and Arm clusters, demonstrating performance portability with similar speedups. A key contribution of this work is the performance analysis of our optimizations, which reveals the complexity introduced by architectural constraints and compiler behavior. Additionally, we explore different implementations leveraging a new matrix instruction set for Arm called SME and provide an early assessment of its potential benefits. I Introduction Although the computational power of HPC systems has grown rapidly over the years, scientific computations often remain memory-bound due to large datasets and complex memory access patterns. This is because memory technologies struggle to scale at the same rate as advances in computational capabilities, creating a bottleneck in HPC architectures. Lattice quantum chromodynamics (LQCD) is a non-perturbative method that employs a discretized 4-dimensional space-time lattice to investigate the strong interaction, a fundamental force governing the interactions between quarks and gluons. LQCD applications are memory-bound, as the computations are relatively lightweight compared to the volume of data that must be moved between memory and processors, making memory bandwidth and latency the limiting factors. Depending on the LQCD application and the specific workload, a very significant part (often >90\% ) of the execution time is spent in iterative solvers. In recent years, there has been a trend of introducing multiple right-hand sides (rhs) in LQCD solvers for performance improvement because 1) matrix-vector operations for a single rhs are replaced by batched operations, increasing the arithmetic intensity, and 2) block Krylov subspace solvers can reduce iterations to convergence by enabling information sharing among multiple rhs. However, the exploration of this rhs blocking lacks a detailed and systematic assessment of its performance. This is especially relevant when such blocking techniques are integrated with algorithmic advances, as the individual impact of each on performance remains unclear. As HPC systems evolve, traditional LQCD codes often struggle to adapt to modern hardware capabilities. With the growing diversity of modern CPU architectures, ensuring application portability is essential to accommodate foreseeable hardware changes and maintain performance efficiency. In this paper, we focus on optimizing the C++ version [ 2 ] of the open-source LQCD solver DD- \alpha AMG [ 16 ] , but we emphasize that it serves as an example with insights that can be broadly applied to other scientific computing applications. This work makes the following contributions: Firstly, we implement rhs blocking in the Wilson-Dirac operator evaluation kernel and integrate it into a batched GMRES solver within DD- \alpha AMG. The previous version of the code is constrained by its reliance on 128-bit SSE intrinsics for efficient execution, which not only limits its ability to utilize the meanwhile available 512-bit SIMD instructions on HPC systems but also restricts its portability to Arm-based architectures. We, therefore, perform a code refactorization that takes auto-vectorization into account and introduce a new data layout tailored for loop vectorization, while preserving a compatible data layout for seamless integration with the legacy code. Secondly, we perform a comprehensive performance and architectural analysis of the improved portable solver on three very different HPC architectures: JUWELS (x86), Ookami (Arm), and HAICGU (Arm) clusters. Our results show that rhs blocking in kernels achieves a speedup of 10% to 24% compared to the original unblocked versions. However, its performance is limited by the effective memory bandwidth, which is sensitive to the choice of blocking size. We also find that although auto-vectorization with optimized data layouts improves SIMD utilization, performance gains are often limited by architectural constraints. Moreover, we observe that a compiler with auto-vectorization may produce markedly different code for varying blocking sizes, complicating performance prediction. Solver-level algorithmic changes may further reduce kernel-level speedups. These findings indicate that rhs blocking is beneficial but constrained, warranting careful application. Finally, to address the new hardware capabilities of modern CPU architectures, we present a study of Arm’s Scalable Matrix Extension (SME) applied to LQCD. We implement an LQCD routine in the Wilson-Dirac operator evaluation with SME and analyze its potential performance using an emulator. The paper is structured as follows: Section II reviews previous work, Section III outlines the methodology and implementations, Section IV defines the evaluation configurations, Sections V and VI present and discuss the results, and Section VII provides a summary and outlook. II Related Work The solution of linear systems Ax_{i}=y_{i} for i=1,\ldots,b rhs can be achieved by expanding a common Krylov subspace generated collectively by all rhs vectors [ 17 ] . At iteration k , the method has access to the block Krylov subspace K_{k}(A,Y)=\mathrm{span}\{Y,AY,A^{2}Y,\ldots,A^{k-1}Y\}, where Y=[y_{1}\mid\ldots\mid y_{b}] is the matrix of all rhs. This combined subspace contains as subsets the individual Krylov subspaces corresponding to each y_{i} , enabling simultaneous residual reduction across all systems. Early parallel implementations of the methods concerning multiple rhs include block versions of GMRES [ 31 , 11 ] , and a block version of BiCGSTAB [ 15 ] . To address the stability issues of the latter, Block BiCGR was introduced in [ 29 ] . As an alternative to these algorithms, the EigCG method was developed in [ 28 ] as a modified Conjugate Gradient (CG) approach that builds an enlarged search space by accumulating a window of past residuals during the CG iterations. The block solvers have been adopted in physics applications to address specific computational challenges. Forcrand et al. [ 14 ] utilized these methods to reduce variance in force term calculations, while Boyle et al. [ 9 ] applied them to domain wall fermion systems. They were also successfully implemented in the context of LQCD for the Wilson-Dirac operator in [ 27 ] . Recent work of block solvers has focused on enhancing their efficiency for modern HPC systems. An efficient implementation of block Krylov methods on GPUs was presented in [ 13 ] . Birk et al. [ 8 ] incorporate simultaneous inversion of multiple rhs and shifted systems by employing block Krylov solvers in DD- \alpha AMG. LQCD has always been an area of research that was strongly dependent on the efficient use of HPC resources. This resulted in various performance analysis and modeling efforts, Bilardi et al. [ 6 ] and Bauer et al. [ 4 ] being early examples. The former is of particular interest for this work as it introduces a method for systematic analysis of the data transfers that are typically limiting the performance. The need for performance has also been a driver for exploring new processor architectures. The A64FX

Read the complete text