Packaging HEP Heterogeneous Mini-apps for Portable Benchmarking and Facility Evaluation on Modern HPCs
\firstname Mohammad \lastname Atif \fnsep, \firstname Pengfei \lastname Ding \fnsep, \firstname Ka Hei Martin \lastname Kwok \fnsep, \firstname Charles \lastname Leggett \fnsep · 2025
Read Packaging HEP Heterogeneous Mini-apps for Portable Benchmarking and Facility Evaluation on Modern HPCs by \firstname Mohammad \lastname Atif \fnsep, \firstname Pengfei \lastname Ding \fnsep, \firstname Ka Hei Martin \lastname Kwok \fnsep, \firstname Charles \lastname Leggett \fnsep (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08933.txt.
Introduction
Atif and co-authors package small test programs so particle physics labs can try out new computing sites with little fuss, in this 2025 paper. Experiments now lean on graphics processors and shared supercomputers, whose hardware and software change fast. Picking where to invest needs quick, fair tests.
Earlier work made portable mini-apps that mimic real experiment workloads. Deploying them at a new site still took heavy manual effort. The team now wrap the apps in containers with automated build tools. Sites can be tested end to end quickly, and scores can be tracked over time as systems evolve.
Catalog record
| CREATOR | \firstname Mohammad \lastname Atif \fnsep, \firstname Pengfei \lastname Ding \fnsep, \firstname Ka Hei Martin \lastname Kwok \fnsep, \firstname Charles \lastname Leggett \fnsep |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 12 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08933
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Packaging HEP Heterogeneous Mini-apps for Portable Benchmarking and Facility Evaluation on Modern HPCs \firstname Mohammad \lastname Atif \fnsep, \firstname Pengfei \lastname Ding \fnsep, \firstname Ka Hei Martin \lastname Kwok \fnsep, \firstname Charles \lastname Leggett \fnsep Abstract High Energy Physics (HEP) experiments are making increasing use of GPUs and GPU dominated High Performance Computer facilities. Both the software and hardware of these systems are rapidly evolving, creating challenges for experiments to make informed decisions as to where they wish to devote resources. In its first phase, the High Energy Physics Center for Computational Excellence (HEP-CCE) produced portable versions of a number of heterogeneous HEP mini-apps, such as p2r , FastCaloSim, Patatrack and the WireCell Toolkit, that exercise a broad range of GPU characteristics, enabling cross platform and facility benchmarking and evaluation. However, these mini-apps still require a significant amount of manual intervention to deploy on a new facility. We present our work in developing turn-key deployments of these mini-apps, where by means of containerization and automated configuration and build techniques such as Spack, we are able to quickly test new hardware, software, environments and entire facilities with minimal user intervention, and then track performance metrics over time. 1 Introduction High energy physics experiments continue to increase their usage of HPC resources, with diverse workloads across a range of applications such as event generation, simulation, reconstruction and analysis, using both CPU and GPU hardware. However, deploying applications to HPCs is not as trivial as sending a job to generic grid hardware which tend to have much more uniform software and hardware stacks. HPCs are like snowflakes - each is unique at both the hardware and software levels, and so extensive modifications of the applications are usually required. Sometimes the process is so onerous, that it might not be worth the effort. Thus it is important to be able to quickly evaluate systems for suitability for various workflows, and to also track system performance with known benchmarks to understand the effects of hardware and software changes. While standard HPC benchmarks like High Performance Linpack or MLPerf exist, they do not readily extrapolate to performance of HEP specific applications. Thus it is critical to be able to run application benchmarks that are representative of real HEP workloads. We have extracted a number of such applications from HEP experiments such as CMS, ATLAS and DUNE, and now the challenge is to package them in a way that enables quick deployment by non-experts at a variety of facilities. Tracking both deployment success and application performance is very useful for monitoring a system as it evolves and changes are made to both software and hardware. Library incompatibilities or API changes often appear over time, and without regular monitoring can catch users by surprise when say an urgent production run is required. Regular, automated builds and benchmarks will help both users and facilities identify problems as they emerge. There are two obvious choices for packaging technologies - containerization and Spack. The first allows a self-contained deployment, but depending on the software dependencies, may require large base images, with different images required for different hardware backends (eg NVIDIA vs AMD GPUs). The second allows for the distribution of just the source code, but depending on what packages are available on site, may require significant compilation. We will explore using these two technologies with the p2r application for Spack, and FastCaloSim for containerization. 2 Packaging p2r Using Spack In collider experiments, reconstructing the tracks of charged particles propagating through the detectors under a magnetic field is one of the most arithmetically intensive task. p2r p2r is a lightweight mini-app that performs the core computations involved in track reconstruction, specifically propagating the track state to the next layer of detector in the radial direction and performing the Kalman update Kalman at the new layer. p2r has been implemented with various portability technologies that covers all three major GPU vendors, as well as multi-threaded CPU reference implementation. The kernel of p2r is abstracted sufficiently that it has minimal dependencies on other software packages or experimental framework. The wide coverage of platforms and minimal dependencies make p2r a prime target for portable benchmarking and evaluation. Spack spack is chosen to be the packaging technology for p2r primarily for its wide adoption in the HPC computing community. Its simplicity and flexibility for versioning and configurations are particularly suitable for integrating into the HPC computing environment. As the GPU portability technologies used in p2r are rapidly evolving, the flexibility of Spack will allow p2r-spack to be evaluated easily through these changes. 2.1 Spack implementation Prior to this work, p2r did not have a uniform build system for the range of implementations, partly due to the diverse needs of compilers for various backends. The first step for adopting to Spack is to create a build system for p2r using one that Spack supports, which is chosen to be CMake. As a CMake package, each p2r implementations is organized into individual sub-directories, containing the detailed compilation options specific to the implementation, while the top-level directory contains the controls of run parameters common to all implementations. Interfacing to p2r-spack , the run parameters, such as number of events and number of tracks, are fully exposed to users such that users can adjust the problem size suitable for each evaluation. The p2r implementations and backends are controlled with the variants feature of Spack. Two multi-valued variants options are used to specify the implementations ( cuda , kokkos , alpaka , alpaka , stdpar , sycl ) and backend ( nvidia , amd , cpu ). Detailed compiler options in each implementation and backend combination are hidden from for simplicity and reproducibility. Figure 1 shows the available p2r implementations and the currently supported combination of implementations and backends using Spack. Figure 1: Available p2r implementations on different execution backends (Green solid cells). This work implements the Spack installation method of the NVIDIA and CPU backends for all p2r implementations, and AMD backends for HIP, Kokkos and Alpaka implementation (Blue boarded cells). 2.2 External packages p2r depends on vendor software and compilers, such as CUDA and nvc++ or portability libraries, such as Kokkos and Alpaka, for specific implementations. Obtaining a consistent set of these software with compatible versions is a difficult task, especially across different platforms, and should be factored out of the p2r package. To address this issue, a Spack package can specify the compatible range of required software and the users can provide these external packages from system-installed locations. This approach avoids installing large, duplicated software from source via Spack while satisfying the package requirements. Instructions to specify these external packages for p2r is listed on the p2r repository ptorSpack . Overall, packaging with Spack provided a much easier path to deploy p2r across different HPC platforms. Multi-step manual installation are simplified into a single command. Given the wide coverage of p2r across portability technologies and GPU hardware, we envision p2r-spack to be a useful tool for benchmarking performance metrics on various modern HPCs for HEP. 3 Packaging FastCaloSim Using Containerization and Automating benchmarking with Continuous Integration tools FastCaloSim performs a parametrized simulation of the ATLAS Liquid Argon Calorimeter FastCaloSim , which allows for much faster simulation than the