Multi-Partner Project: Multi-GPU Performance Portability Analysis for CFD Simulations at Scale

Panagiotis-Eleftherios Eleftherakis 1 , George Anagnostopoulos 1 , Anastassis Kapetanakis 1 , Mohammad Umair 2 , Jean-Yves Vet 3 , Konstantinos Iliakis 1 , Jonathan Vincent 2 , Jing Gong 2 , Akshay Patil 4 , Clara García-Sánchez 4 , Gerardo Zampino 2 , Ricardo Vinuesa 5 , Sotirios Xydis 1 · 2026

Read Multi-Partner Project: Multi-GPU Performance Portability Analysis for CFD Simulations at Scale by Panagiotis-Eleftherios Eleftherakis 1 , George Anagnostopoulos 1 , Anastassis Kapetanakis 1 , Mohammad Umair 2 , Jean-Yves Vet 3 , Konstantinos Iliakis 1 , Jonathan Vincent 2 , Jing Gong 2 , Akshay Patil 4 , Clara García-Sánchez 4 , Gerardo Zampino 2 , Ricardo Vinuesa 5 , Sotirios Xydis 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14159.txt.

Introduction

Panagiotis-Eleftherios Eleftherakis and a large team test CFD codes across many GPU types at scale in 2026. Heterogeneous supercomputing with GPUs is now central to high-performance computing. Computational fluid dynamics is a de-facto HPC workload with heavy demands. Performance portability means keeping near-optimal speed across distinct accelerators.

Their Multi-Partner Project analyzes multi-GPU portability for CFD simulations. Codes that slow down on new hardware waste costly machine time. The study spans several partners and their distinct systems. Results can guide how CFD teams write code for the next machines. The tests run at scale, not on toy grids. That scale is what makes the port costs real for HPC teams.

Catalog record

CREATORPanagiotis-Eleftherios Eleftherakis 1 , George Anagnostopoulos 1 , Anastassis Kapetanakis 1 , Mohammad Umair 2 , Jean-Yves Vet 3 , Konstantinos Iliakis 1 , Jonathan Vincent 2 , Jing Gong 2 , Akshay Patil 4 , Clara García-Sánchez 4 , Gerardo Zampino 2 , Ricardo Vinuesa 5 , Sotirios Xydis 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-14159

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Partner Project: Multi-GPU Performance Portability Analysis for CFD Simulations at Scale Thanks: This work has been partially funded by the EU Horizon 2022 program under grant agreement No 101096698 REFMAP (website: https://www.refmap.eu). We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LUMI, hosted by CSC (Finland) and the LUMI consortium through a EuroHPC Regular Access call. Panagiotis-Eleftherios Eleftherakis 1 , George Anagnostopoulos 1 , Anastassis Kapetanakis 1 , Mohammad Umair 2 , Jean-Yves Vet 3 , Konstantinos Iliakis 1 , Jonathan Vincent 2 , Jing Gong 2 , Akshay Patil 4 , Clara García-Sánchez 4 , Gerardo Zampino 2 , Ricardo Vinuesa 5 , Sotirios Xydis 1 Abstract As heterogeneous supercomputing architectures leveraging GPUs become increasingly central to high-performance computing (HPC), it is crucial for computational fluid dynamics (CFD) simulations, a de-facto HPC workload, to efficiently utilize such hardware. One of the key challenges of HPC codes is performance portability, i.e. the ability to maintain near-optimal performance across different accelerators. In the context of the REFMAP project, which targets scalable, GPU-enabled multi-fidelity CFD for urban airflow prediction, this paper analyzes the performance portability of SOD2D, a state-of-the-art Spectral Elements simulation framework across AMD and NVIDIA GPU architectures. We first discuss the physical and numerical models underlying SOD2D, highlighting its computational hotspots. Then, we examine its performance and scalability in a multi-level manner, i.e. defining and characterizing an extensive full-stack design space spanning across application, software and hardware infrastructure related parameters. Single-GPU performance characterization across server-grade NVIDIA and AMD GPU architectures and vendor-specific compiler stacks, show the potential as well as the diverse effect of memory access optimizations, i.e. 0.69 × \times - 3.91 × \times deviations in acceleration speedup. Performance variability of SOD2D at scale is further examined on the LUMI multi-GPU cluster, where profiling reveals similar throughput variations, highlighting the limits of performance projections and the need for multi-level, informed tuning. I Introduction Computational Fluid Dynamics (CFD) is an essential tool for modern engineering, enabling detailed analysis of fluid behavior in aerospace, automotive, maritime, and environmental systems without costly physical prototypes. Recently, CFD-generated datasets have also become central to training physics-informed neural models, which embed physical laws into machine learning models for improved predictive accuracy [ 18 , 4 ] . At its core, CFD solves the Navier–Stokes equations, expressing conservation of mass, momentum, and energy in fluid flows. Analytical solutions are rare, so numerical methods such as the Finite Difference Method (FDM) and Finite Element Method (FEM) are employed. FDM is efficient for structured grids, while FEM better handles complex geometries at higher computational cost. High-Performance Computing (HPC) has greatly advanced CFD, enabling high-fidelity simulations of complex phenomena. Heterogeneous architectures, particularly those leveraging Graphics Processing Units (GPUs), have driven progress toward exascale computing, supported by sophisticated solvers such as OpenFOAM, NekRS, and others [ 6 , 10 ] . However, optimizing CFD solvers for diverse hardware remains challenging due to differing GPU architectures, compiler ecosystems, and programming models. Solvers optimized for one platform often underperform on another, hindering performance portability—i.e., sustained efficiency across architectures. Within this context, the EU funded REFMAP project ( Reducing Environmental Footprint through transformative Multi-scale Aviation Planning ) targets low-latency, multi-fidelity urban-airflow prediction to support sustainable Innovative Air Mobility in urban environments. Its high-fidelity CFD component couples GPU-enabled simulations with machine-learning surrogates. High-resolution Direct Numerical Simulation (DNS) data train compact predictors deployable at the edge or in the cloud. Since such workloads are computationally intensive, REFMAP integrates GPU-accelerated CFD and autotuning to overcome data-generation bottlenecks. The Spectral Element solver SOD2D [ 7 ] is ported to GPUs using OpenACC [ 14 ] and MPI, enabling scalable, turbulence-resolving simulations across NVIDIA and AMD systems. Achieving performance portability across these heterogeneous HPC platforms is critical to REFMAP’s data-driven workflow, ensuring scalable, cost-effective surrogate training and deployment. CFD solvers adopting high-order numerical schemes and multi-architecture support—such as URANOS [ 3 ] , STREAmS [ 17 ] , and PyFR [ 19 ] —have achieved significant GPU speedups, though cross-vendor analysis remains limited. Reguly and Mudalige [ 16 ] provide a broad overview of performance portability strategies, noting that few works (e.g., CloverLeaf [ 13 ] , TeaLeaf [ 11 ] , Nekbone [ 8 ] ) use OpenACC across multiple platforms, with limited direct cross-vendor comparison. More recent efforts [ 2 ] similarly omit direct cross-vendor GPU evaluation. To our knowledge, this work is among the first to perform a detailed, full-stack cross-vendor study of GPU-accelerated CFD using OpenACC, spanning from solver configuration to hardware-level memory optimization. Fig. 1: Overview of REFMAP’s HPC to ML workflow, with emphasis on CFD acceleration. II The REFMAP Project Outline REFMAP targets low-latency, multi-fidelity urban-airflow prediction to support sustainable Innovative Air Mobility. As seen in Fig. 1 , the low-fidelity branch is driven by the OpenFOAM CFD simulator (and accelerated by the NVIDIA AmgX solver) executing a model that aims at evaluating the neighbourhood-scale wind-associated risk [ 15 ] . Regarding the high-fidelity framework, the SOD2D simulator [ 7 ] is employed (a Fortran-based SEM code with OpenACC offload and MPI partitioning, where key kernels —i.e. convection, diffusion— are parallelized across elements and nodes) enabling the coupling of GPU-enabled CFD with data-driven surrogates. High-resolution LES/DNS simulations generate turbulence-resolving data to train compact models [ 9 ] that undergo extensive optimizations such as compression and pruning in order to deploy at the edge or in the cloud, enabling UAV trajectory optimization and high-fidelity sensor placement, thus providing a robust substrate for efficient and safe UAV navigation. On CPU-only systems, such cases require weeks to compute, limiting the datasets for surrogate training. REFMAP overcomes this bottleneck by extensive autotuning and performance exploration of SOD2D. Initial studies have shown performance sensitivity when freely varying gang numbers and vector lengths using OpenTuner’s AUC-Bandit search technique, obtaining 1.2–1.5 \times speedup [ 12 , 1 ] . Moving towards REFMAP’s mature phase, multiple infrastructures were employed necessitating multi-level performance exploration [ 5 ] - i.e. spanning across hardware,software and application-specific parameters. On the surrogate model side, REFMAP employs fully convolutional networks (FCNs) that map wall-shear-stress and pressure inputs to velocity-fluctuation fields \hat{u},\hat{v},\hat{w} at selected wall-normal positions. Models are trained on DNS data (Re τ =180) with 6,000 snapshots per y^{+}=15,30,50,100 , using MSE loss and normalized targets to balance component errors. While additional GAN and transformer models are explored, FCNs remain the main surrogate coupled with accelerated CFD for wall-based flow reconstruction. Leveraging structured pruning, quantization, and compiler-level optimizations, the final turbulence-prediction models achieve up to 50× compression with negligible loss in accuracy, significantly improving latency and throughput. These advances enable fast, reliable flow prediction and establish a foundation for integrating high-performance simulation and data-driven surrogates into real-time UAV navigation and urban risk assessment workflows. The rest of this paper will focus on the performance portability analysis that enables efficient generation of datasets for surrogate model training within the REFMAP workflow. III SOD2D: Spectral FEM for compressible scale-resolving simulations III-A Implementation and governing equations The physical model considered is composed of the Navier–Stokes equations governing mass, momentum, and energy conservation in a compressible fluid [ 7 ] . In conservative form: \displaystyle{\color[rgb]{0,0.3906,0}\frac{\partial\rho}{\partial t}}{\color[rgb]{1,0.5,0}+\nabla\cdot(\rho\boldsymbol{u})}=0 (1) \displaystyle{\color[rgb]{0,0.3906,0}\frac{\partial(\rho\boldsymbol{u})}{\partial t}}{\color[rgb]{1,0.5,0}+\nabla\cdot(\rho\boldsymbol{u}\otimes\boldsymbol{u})}{\color[rgb]{0,0,1}-\nabla\boldsymbol{\tau}}=-\nabla p+\boldsymbol{f} (2) \displaystyle{\color[rgb]{0,0.3906,0}\frac{\partial E}{\partial t}}{\color[rgb]{1,0.5,0}+\nabla\cdot((E+p)\boldsymbol{u})}{\color[rgb]{0,0,1}-\nabla\cdot(\boldsymbol{\tau}\boldsymbol{u})-\nabla(\kappa\nabla

Read the complete text