Evaluating Application Characteristics for GPU Portability Layer Selection

Mohammad Atif 1, Meghna Bhattacharya 2, Mark Dewing 2, Zhihua Dong 1, Julien Esseiva 3, Oliver Gutsche 2, Matti Kortelainen 2, Ka Hei Martin Kwok 2, Charles Leggett 3, Meifeng Lin 1, Aleksei Strelchenko 2, Vakhang Tsulaia 3, Brett Viren 1, Tianle Wang 1, Haiwang Yu 1 · 2026

Read Evaluating Application Characteristics for GPU Portability Layer Selection by Mohammad Atif 1, Meghna Bhattacharya 2, Mark Dewing 2, Zhihua Dong 1, Julien Esseiva 3, Oliver Gutsche 2, Matti Kortelainen 2, Ka Hei Martin Kwok 2, Charles Leggett 3, Meifeng Lin 1, Aleksei Strelchenko 2, Vakhang Tsulaia 3, Brett Viren 1, Tianle Wang 1, Haiwang Yu 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17526.txt.

Introduction

Mohammad Atif and co-authors help teams pick GPU portability layers. Graphics processors now supply most high-end compute power. High energy physics uses them for a fast-growing share of tasks. NVIDIA leads the market, but AMD and Intel are gaining. Code must run well on all three. Portability layers promise that reach.

The team evaluates app traits that should drive the choice. Kernel shape, memory use, and team skill all matter. A wrong layer can lock a lab into poor speed. Their guide turns vendor claims into test-based picks. Big science codes live for decades. The layer choice shapes cost for years. This work makes that choice a studied one.

Catalog record

CREATORMohammad Atif 1, Meghna Bhattacharya 2, Mark Dewing 2, Zhihua Dong 1, Julien Esseiva 3, Oliver Gutsche 2, Matti Kortelainen 2, Ka Hei Martin Kwok 2, Charles Leggett 3, Meifeng Lin 1, Aleksei Strelchenko 2, Vakhang Tsulaia 3, Brett Viren 1, Tianle Wang 1, Haiwang Yu 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17526

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Application Characteristics for GPU Portability Layer Selection Mohammad Atif 1, Meghna Bhattacharya 2, Mark Dewing 2, Zhihua Dong 1, Julien Esseiva 3, Oliver Gutsche 2, Matti Kortelainen 2, Ka Hei Martin Kwok 2, Charles Leggett 3, Meifeng Lin 1, Aleksei Strelchenko 2, Vakhang Tsulaia 3, Brett Viren 1, Tianle Wang 1, Haiwang Yu 1 Abstract GPUs have become the dominant source of computing power for high performance computing and are increasingly being used across the High Energy Physics computing landscape for a wide variety of tasks. Though NVIDIA is currently the main provider of GPUs, AMD and Intel are rapidly increasing their market share. As a result, programming using a vendor-specific language such as CUDA can significantly reduce deployment choices. There are a number of portability layers such as Kokkos, Alpaka, SYCL, OpenMP and std::par that permit execution on a broad range of GPU and CPU architectures, significantly increasing the flexibility of application programmers. However, each of these portability layers has its own characteristics, performing better at some tasks and worse at others, or placing limitations on aspects of the application. In this presentation, we report on a study of application and kernel characteristics that can influence the choice of a portability layer and show how each layer handles these characteristics. We have analyzed representative heterogeneous applications from CMS (patatrack and p2r), DUNE (Wire-Cell Toolkit), and ATLAS (FastCaloSim) to identify key application characteristics that have different behaviors for the various portability technologies. Using these results, developers can make more informed decisions on which GPU portability technology is best suited to their application. 1 Introduction Over the past several years, great progress has been made by various vendors to support a broad range of GPU and other computational accelerator backends in a portable manner. Kokkos [ 6 , 15 ] , SYCL [ 12 ] , HIP [ 7 ] , OpenMP [ 2 ] , alpaka [ 10 ] and std::par [ 9 ] can all target NVIDIA, AMD and Intel GPUs, as well as multicore CPUs and FPGAs with various degrees of completeness (see Figure 1 ). However, just because an architecture can be targeted, doesn’t mean that all portability layers are equivalent. When developers are faced with the task of writing or converting code to run on GPUs, an informed decision needs to be made as to which technology to use, and which is best suited for their application. Understanding how different portability technologies handle various use cases, patterns and code structures is very useful in predicting how an application will perform before any code is written. The High Energy Physics Center for Computational Excellence (HEP-CCE) has spent several years porting a number of representative HEP testbeds from serial C++ code to a variety of portability technologies. We have used these testbeds to identify several code characteristics which can significantly affect an application’s performance with specific portability technologies. Figure 1: Hardware support of portability layers. Dark green indicates full support, light green indicates partial support or that the project is still under development, orange is a proof of concept with an uncertain future, and red indicates no support (though there may have been support in the past). 2 HEP-CCE Testbeds The testbeds that the HEP-CCE have chosen are from a number of HEP experiments, including ATLAS, CMS and DUNE. These testbeds are relatively small code bases that can be built and executed outside of the experiments’ full frameworks in order to simplify the development process. From ATLAS, we have FastCaloSim [ 4 ] [ 5 ] [ 1 ] , a standalone, parametrized simulation of the ATLAS Liquid Argon Calorimeter. From CMS we chose Patatrack [ 3 ] , a standalone version of the heterogeneous pixel reconstruction which spans 40+ kernels, implements multithreaded concurrency, and utilizes memory pools, as well as P2R [ 14 ] , which performs the propagate-to-radius track reconstruction in a single kernel. From DUNE we have extracted the Liquid Argon Time Projection Chamber simulation module [ 17 ] [ 8 ] , which is comprised of three kernels that perform a rasterization, scatter-add, and FFT convolution. We have also used the GPU ports of Sherpa (now called Pepper) and Madgraph, which are leading order event generators that make use of auto-generated code. 3 Kernel Runtime and Launch Latency Launching a kernel from a host device onto a GPU is not an instantaneous process. Even under the best of conditions, this can take from a few to many tens of microseconds. This launch latency is dependent on many factors, such as the GPU architecture (in general AMD GPUs have much higher launch latency than NVIDIA), the CPU architecture, the clock speed of the CPU and that of the bus between CPU and GPU, as well as GPU driver version and size of the parameter list passed to the kernel. Some portability layers can significantly increase this launch latency. Kokkos can add tens of microseconds, and even many tens of microseconds when executing on AMD GPUs, which in general have much larger launch latencies than NVIDIA or Intel GPUs. Large additional penalties were not observed for SYCL, HIP, OpenMP or Alpaka. We also observed that calling rocRAND (the hardware native random number generator from AMD) on AMD GPUs resulted in very large launch latencies, sometimes exceeding 1s, on the first call. If the runtime of a kernel, or a group of sequential kernels, is short and on the order of the launch latency, the performance of the application can suffer greatly. While there are techniques whereby the latency can be hidden, such as with asynchronous kernel launches where work continues on the CPU while the GPU kernel executes, this is not always possible. In these situations, where the kernel runtime is very short, it is best to avoid portability layers that exacerbate the problem. 4 Concurrency and Thread Pools In HEP codes, GPU applications are rarely standalone. Rather they exist within the context of a larger application framework, which often use concurrency mechanisms such as multi-threading or multi-processing to improve performance or make better use of resources. This means that not only can the GPU kernels be launched, sometimes concurrently, from different threads or processes, but that the portability layer itself also needs to support the larger concurrency mechanism. We found that Kokkos has significant incompatibilities with concurrency and thread pools such as Intel’s Threading Building Blocks (TBB). The serial backend of Kokkos implements a lock that serializes concurrent calls. The threads backend explicitly forbids calls from external threads. This results in very poor performance for multi-threaded applications that attempt to concurrently launch kernels. Furthermore, the concurrent launch of GPU kernels is only achievable with CUDA and HIP backends, using architecture specific APIs that limit portability. (Kokkos is however developing a prototype experimental feature of ”partitioned execution spaces” which may address this in the near future.) We also observed that the SYCL implementations were inconsistent in their ability to launch GPU kernels. Some implementations on the same architecture serialized concurrent kernel calls from different threads. This is often very dependent not only on the compiler manufacturer, but also on the version of the compiler itself, demonstrating how the SYCL implementations continue to evolve rapidly. Furthermore,

Read the complete text