A Microbenchmark Framework for Performance Evaluation of OpenMP Target Offloading
Mohammad Atif 1 , Tianle Wang 1 , Zhihua Dong 1 , Charles Leggett 2 , Meifeng Lin 1 · 2025
Read A Microbenchmark Framework for Performance Evaluation of OpenMP Target Offloading by Mohammad Atif 1 , Tianle Wang 1 , Zhihua Dong 1 , Charles Leggett 2 , Meifeng Lin 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00408.txt.
Introduction
Mohammad Atif and four co-authors built a test frame in two thousand twenty-five for OpenMP code that offloads work to graphics processors. Compilers for this offload model change fast, and each change can speed or slow real programs in hard-to-predict ways. Small focused tests, called microbenchmarks, can track those shifts.
Their frame, built on the Catch2 test tool, runs common operations on two leading supercomputers, Perlmutter and Frontier. Results let compiler teams see the effect of small changes, and let users pick the compiler version likely to run their code fastest. Such tracking matters as heterogeneous machines become the norm.
Catalog record
| CREATOR | Mohammad Atif 1 , Tianle Wang 1 , Zhihua Dong 1 , Charles Leggett 2 , Meifeng Lin 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00408
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Microbenchmark Framework for Performance Evaluation of OpenMP Target Offloading Mohammad Atif 1 , Tianle Wang 1 , Zhihua Dong 1 , Charles Leggett 2 , Meifeng Lin 1 Abstract We present a framework based on Catch2 to evaluate performance of OpenMP’s target offload model via micro-benchmarks. The compilers supporting OpenMP’s target offload model for heterogeneous architectures are currently undergoing rapid development. These developments influence performance of various complex applications in different ways. This framework can be employed to track the impact of compiler upgrades and compare their performance with the native programming models. We use the framework to benchmark performance of a few commonly used operations on leadership class supercomputers such as Perlmutter at National Energy Research Scientific Computing (NERSC) Center and Frontier at Oak Ridge Leadership Computing Facility (OLCF). Such a framework will be useful for compiler developers to gain insights into the overall impact of many small changes, as well as for users to decide which compilers and versions are expected to yield best performance for their applications. I Introduction In the past decade, the importance of portable parallel programming has increased due to the variety of accelerators in high-performance computing (HPC) systems. Most large scale supercomputers today are heterogeneous and employ Graphics Processing Units (GPUs) from various vendors like Intel, AMD, and NVIDIA, each of which has its own native application programming interface (API). In order to use the GPUs, one needs to either port compute-intensive parts of the application to a vendor-specific API, or write the application with a portable programming model. Several portable programming models have been developed in the last decade as a solution to avoid code duplication and diversion for different GPU backends. Examples of such portable programming models include Kokkos [ 1 ] , SYCL [ 2 ] , RAJA [ 3 ] , Alpaka [ 4 ] , OpenACC [ 5 ] and OpenMP [ 6 ] . This work focuses on OpenMP’s target offload model, which has undergone rapid development in the past few years and gained increasing vendor support [ 7 , 8 ] . OpenMP’s directive-based GPU offloading strategy is also a strong candidate for performance portability of Fortran codes [ 9 ] , although we focus our attention to C/C ++ in the following discussion. The quantification and reporting of HPC technologies and their evolution have traditionally been based on performance benchmarks [ 10 , 11 ] . Perhaps, the most famous benchmarks are the STREAM and High Performance Linpack (HPL) benchmarks [ 12 , 13 ] . There are collections of several excellent benchmark suites [ 14 , 15 , 16 , 17 ] that measure the run times of real-world and synthetic applications. Most of these benchmarks calculate the entire performance of a reasonably large application with several active components during various phases of execution [ 18 ] . To narrow down performance bottlenecks, micro-benchmarking frameworks have been developed concurrently with testing frameworks such as Google Benchmark, Nonius, and Catch2 [ 19 , 20 , 21 ] . Such micro-benchmarking frameworks execute a small snippet of code multiple times followed by statistical analyses to generate highly reliable metrics of computational performance. We demonstrate that micro-benchmarks can be useful tools for comparing programming models too. In this work, we present a framework based on Catch2 [ 21 ] for performance evaluation of OpenMP’s target offload model via microbenchmarks. The primary motivation is to develop the necessary tools for the better evaluation of specific operations in OpenMP target offload. The performance of the OpenMP target offload currently varies depending on the target backend and the application, as reported in [ 7 , 22 , 23 ] . We have encountered several performance issues [ 24 ] related to specialized, operations, such as atomic, memset or scan, when porting application codes to OpenMP. The key idea behind developing a micro-benchmarking framework is to generate high quality measurements by creating samples in contrast to running the same kernel a number of times in a loop and reporting the mean. As a demonstration, in this paper, we use the framework to benchmark the performance of several widely used operations on NERSC Perlmutter and OLCF Frontier (see [ 25 , 26 ] for architectures) and two servers with Nvidia V100 and A6000 housed at Brookhaven National Lab. There are several advantages to benchmarking with Catch2, such as: • Easy integration: While there are several similarities between Catch2’s benchmarks and Google Benchmarks in their sampling and iteration count evaluation, Catch2 is light-weight and enables a more streamlined integration of unit-testing and benchmarking framework. • Dynamic estimation of iteration count: Catch2’s micro-benchmarks create samples by accounting for the clock-resolution and dynamically estimating the iteration count of the kernel by estimating its runtime. Each sample can consist of more than one run of the kernel if the available clock lacks sufficient resolution. • Statistical bootstrapping: The statistical bootstrapping method first creates a set of samples and then creates several subsets of resamples to calculate the mean and standard deviation. The resampling technique reduces the impact of artifacts of measurement. This feature is quite important for our framework as it minimizes the chance of a fringe case biasing the measurements. It calculates confidence intervals for each run thus improving robustness and enhancing comparability across hardware, data types, threads per block, and programming models. • Catch2 also ensures that the compiler does not optimize away the kernel of interest in the case its output is not being used later in the program. The above features offer additional information as compared to standard tools such as Nsight Systems which instrument the code thus incurring profiler’s overheads. Indeed, the micro-benchmark results need to be combined with traces and profiles along with assembly to analyze the root causes of performance disparity. However, the idea behind this framework is to pinpoint the location of good/bad performance in the wide space of compilers, programming models, compiler flags, hardware, datatypes and array sizes for a few commonly used kernels. Therefore, the root cause analyses are kept out of scope of the present manuscript. The remainder of the paper is organized as follows: in Sec. III , we discuss OpenMP’s target offload model. In Sec. IV , we introduce our framework and discuss benchmarking with Catch2. In Sec. V we use the present framework to benchmark a few commonly used kernels and compare them with the native programming models across compiler version, hardware, data types, and compiler flags. Finally, in Sec. VI we summarize our findings and discuss avenues for further research. Fig. 1: A sketch of Catch2 based microbenchmarking suite’s workflow. II Related Work OpenMP micro-benchmarks [ 27 ] were developed by the Edinburgh Parallel Computing Centre (EPCC) to measure the overhead of synchronization, loop scheduling, array operations, and task scheduling. However, it does not yet support the OpenMP target offload constructs. The SOLLVE project of the US Exascale Computing Project (ECP) has developed a validation and verification suite [ 28 ] to test the correctness and conformance to the OpenMP specification of different compiler implementations. However, performance evaluation is not one of SOLLVE’s main goals. The SPEC HPC [ 29 ] and SPEC ACCEL [ 16 ] benchmarks evaluate the