exaCB: Reproducible Continuous Benchmark Collections at Scale Leveraging an Incremental Approach

Jayesh Badwaik, Mathis Bode, Michal Rajski, Andreas Herten · 2026

Read exaCB: Reproducible Continuous Benchmark Collections at Scale Leveraging an Incremental Approach by Jayesh Badwaik, Mathis Bode, Michal Rajski, Andreas Herten (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22251.txt.

Introduction

Badwaik and co-authors present exaCB, a framework for continuous benchmarking of high-performance computing systems. Software teams already use continuous integration to check correctness. Performance and energy checks usually sit outside that flow. exaCB brings benchmarking into the same pipelines, with shared parts and one reporting protocol. Teams can adopt it step by step, from basic run checks to full instrumentation. The framework was built for the JUPITER exascale system. Its early-access program ran continuous benchmarks for over 70 applications. The runs support regression tracking and energy-aware studies.

Catalog record

CREATORJayesh Badwaik, Mathis Bode, Michal Rajski, Andreas Herten
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57948
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22251

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

exaCB: Reproducible Continuous Benchmark Collections at Scale Leveraging an Incremental Approach Jayesh Badwaik, Mathis Bode, Michal Rajski, Andreas Herten Abstract The increasing heterogeneity of high-performance computing (HPC) systems and the transition to exascale architectures require systematic and reproducible performance evaluation across diverse workloads. While continuous integration (CI) ensures functional correctness in software engineering, performance and energy efficiency in HPC are typically evaluated outside CI workflows, motivating continuous benchmarking (CB) as a complementary approach. Integrating benchmarking into CI workflows enables reproducible evaluation, early detection of regressions, and continuous validation throughout the software development lifecycle. We present exaCB , a framework for continuous benchmarking developed in the context of the JUPITER exascale system. exaCB enables application teams to integrate benchmarking into their workflows while supporting large-scale, system-wide studies through reusable CI/CD components, established harnesses, and a shared reporting protocol. The framework supports incremental adoption, allowing benchmarks to be onboarded easily and to evolve from basic runnability to more advanced instrumentation and reproducibility. The approach is demonstrated in JUREAP, the early-access program for JUPITER, where exaCB enabled continuous benchmarking of over 70 applications at varying maturity levels, supporting cross-application analysis, performance tracking, and energy-aware studies. These results illustrate the practicality using exaCB for continuous benchmarking for exascale HPC systems across large, diverse collections of scientific applications. I Introduction High-performance computing (HPC) systems increasingly combine heterogeneous architectures with complex, tightly coupled software stacks to deliver optimal performance for diverse user workloads [ 9 , 1 ] . To ensure that systems and software perform as expected, reliable and continuous performance evaluation across a broad range of applications is essential. Such evaluation is critical during system procurement and acceptance, early access and bring-up phases, and throughout regular operation to ensure consistent performance over time. This manuscript presents exaCB , a framework for continuous benchmarking of scientific workloads on HPC systems, utilizing established software engineering practices to aid administrators, support staff, technology explorers, and application users in ensuring optimal performance. I-A Challenges of HPC Workloads Traditionally, system performance has often been addressed by using a set of well-known benchmarks, assuming that the results would generalize to a wide range of applications. These benchmarks would often be carefully designed and maintained by the system operators themselves, potentially independently of development of the upstream software, with only occasional updates during major events such as system upgrades or procurement of new systems. In an era where systems were relatively homogeneous, and applications were relatively simple, this approach was often sufficient to provide a reasonable estimate of the system performance for a wide range of applications. However, modern HPC hardware is increasingly heterogeneous [ 5 ] , with a variety of architectures becoming available, each with its own strengths and weaknesses. Furthermore, the applications themselves are becoming increasingly complex, often involving multiple components and large number of dependencies that interact in non-trivial ways. In such an environment, the traditional approach of using a small set of benchmarks is no longer sufficient to capture the performance characteristics of the full set of systems that a computing center may offer. At the same time, maintaining a large and diverse set of benchmarks that is broad enough to cover the full spectrum of systems and applications is a daunting task, and often infeasible without sustained support from the upstream software community. As a result, the traditional model of maintaining benchmarks independently of upstream development, and in a largely static manner, is no longer sufficient. Instead, benchmarking must be integrated more directly into the software development process so that application developers, system administrators, and vendors can rely on shared code paths to enable consistent, reproducible performance evaluation across systems. The complexity of the modern HPC ecosystem necessitates a shift in how benchmarking is approached in itself as well. It is often no longer sufficient to consider each benchmark independently, but rather as a part of a larger ecosystem of benchmarks that need to be maintained and executed in a collaborative manner. A change in a system component may have divergent effects on different benchmarks, and only by considering the full spectrum of benchmarks can system operators and computing centers make informed optimization decisions. Also the benchmarking ecosystem introduces its own challenges. The diversity of hardware architectures means that not all benchmarks are ready to run on all systems at all times [ 17 ] ; moreover, components throughout the benchmarking stack evolve at different paces across systems, leading to varying levels of benchmark readiness. A systematic benchmarking approach must therefore be flexible enough to accommodate this diversity, not only in methodologies but also in levels of readiness, while enabling benchmarks to continuously mature toward a high level of readiness. Finally, in the era of exascale computing, energy efficiency becomes a first-class concern [ 16 , 11 ] ; benchmarks must therefore be not only runnable, but also readily instrumentable and amenable to performance, power, and energy analysis and optimization. Such instrumentation should be supported from the outset and often includes performance-based measurements which, even when not directly required, facilitate analysis and significantly simplify sharing and reuse. These challenges require a conceptual change in how HPC workloads are executed, establishing new methodologies for reproducible workflows to focus on close collaboration between the application developers, support engineers, and system administrators. I-B Collaborative benchmarking The general concept of collaborative benchmarking is schematically represented in Figure 1 . Here, benchmarking is inherently collaborative, relying on a direct connection between developers, support engineers, and system administrators. Each of these actors has their own set of tasks that together form a pipeline, interleaved with the exchange of services. Reusable CI components, provided by support entities, enable developers to run benchmarks in a standardized way, while data produced by both developers and administrators is collected into a common meta-repository or result store. This data can then be post-processed and meta-analyzed to extract insights about application and system performance. The resulting information drives the reporting component, which on one hand allows developers to iterate and investigate the performance of their application, while on the other hand provides administrators with insights about the system they operate. Through this workflow, the benchmarking process is automated and ensures a rigorous and reproducible execution of the application in continuous fashion. Fig. 1: Collaborative benchmarking methodology for modern HPC systems I-C Continuous Benchmarking In this work, we present exaCB , a framework for continuous benchmarking on HPC systems that addresses the challenges outlined above, up to exascale class systems, providing a missing link in general concept of collaborative workflow. exaCB enables CI-based benchmarking by combining command-line tools, reusable CI/CD constructs such as GitLab CI/CD templates, and established benchmarking harnesses – currently using JUBE [ 3 ] , with planned support for ReFrame [ 15 ] and Ramble [ 13 ] . Several approaches to testing and benchmarking HPC applications have been proposed in the literature, including buildtest [ 24 ] , OLCF Test Harness [ 6 ] , Pavilion2 [ 19 ] , BenchPRO [ 25 ] , Benchpark [ 22 ] , and Thicket [ 4 ] , each tackling the aforementioned challenges in different ways. What distinguishes exaCB from these efforts is that its individual components are connected through a shared protocol and data model,

Read the complete text