A Performance Analysis of Task Scheduling for UQ Workflows on HPC Systems
Chung Ming Loi, Anne Reinarz, Linus Seelinger, William Hornsby, James Buchanan, Mikkel Lykkegaard · 2025
Read A Performance Analysis of Task Scheduling for UQ Workflows on HPC Systems by Chung Ming Loi, Anne Reinarz, Linus Seelinger, William Hornsby, James Buchanan, Mikkel Lykkegaard (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22645.txt.
Introduction
Chung Ming Loi and five co-workers wrote a 2025 paper on scheduling uncertainty studies on supercomputers. Such studies launch thousands of similar tasks with wildly varied runtimes. The total count is often unknown at the start. Standard batch scripts and built-in schedulers cope poorly with that load.
Their approach pairs a modeling bridge with a smart queue tool that rides on the native scheduler. No system-level changes are needed. Tests used a plasma turbulence simulator and a fast stand-in for it. Scheduling overhead fell by up to a thousandfold. Long jobs also used notably less processor time than the naive route.
Catalog record
| CREATOR | Chung Ming Loi, Anne Reinarz, Linus Seelinger, William Hornsby, James Buchanan, Mikkel Lykkegaard |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-22645
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Performance Analysis of Task Scheduling for UQ Workflows on HPC Systems Chung Ming Loi, Anne Reinarz, Linus Seelinger, William Hornsby, James Buchanan, Mikkel Lykkegaard Abstract Uncertainty Quantification (UQ) workloads are becoming increasingly common in science and engineering. They involve the submission of thousands or even millions of similar tasks with potentially unpredictable runtimes, where the total number is usually not known a priori. A static one-size-fits-all batch script would likely lead to suboptimal scheduling, and native schedulers installed on High-Performance Computing (HPC) systems such as SLURM often struggle to efficiently handle such workloads. In this paper, we introduce a new load balancing approach suitable for UQ workflows. To demonstrate its efficiency in a real-world setting, we focus on the GS2 gyrokinetic plasma turbulence simulator. Individual simulations can be computationally demanding, with runtimes varying significantly—from minutes to hours—depending on the high-dimensional input parameters. Our approach uses the UQ and Modelling Bridge , which offers a language-agnostic interface to a simulation model, combined with HyperQueue which works alongside the native scheduler. In particular, deploying this framework on HPC systems does not require system-level changes. We benchmark our proposed framework against a standalone SLURM approach using GS2 and a Gaussian Process surrogate thereof. Our results demonstrate a reduction in scheduling overhead by up to three orders of magnitude and a maximum reduction of 38% in CPU time for long-running simulations compared to the naïve SLURM approach, while making no assumptions about the job submission patterns inherent to UQ workflows. I Introduction Efficient scheduling of applications is critical for achieving high performance on large compute clusters. Traditional resource and job management software such as SLURM [ 1 ] , PBS [ 2 ] or LoadLeveler [ 3 ] are designed to schedule large, long-running, homogeneous applications. Ahn et al. [ 4 ] note that on production clusters at Lawrence Livermore National Laboratory 48.1% of jobs now involve submission of at least 100 identical jobs within a short time frame. In other words, workloads consisting of a large number of simple tasks are quickly becoming the norm. For these types of small tasks, traditional schedulers require users to manually balance their workloads, either by submitting large numbers of jobs quickly or by manually aggregating these tasks. Both approaches commonly lead to suboptimal resource utilisation. An area in which these types of workload are particularly prevalent is Uncertainty Quantification (UQ) . Whether it is uncertainty propagation or Bayesian inference [ 5 ] , the same simulation is run thousands of times with different input parameters. This is true for a wide range of algorithms; to list just a few Latin hypercube, quasi-Monte Carlo, ADVI, HMC, and NUTS [ 6 , 7 , 8 ] . In this paper, we will focus UQ workloads enabled by the the UQ and Modelling Bridge (UM-Bridge) framework [ 9 , 10 ] , a universal interface that makes numerical models accessible from any programming language or higher-level software, with a main focus on UQ . It consists of an Hypertext Transfer Protocol (HTTP) that can query the model for evaluations and several types of derivatives. As a realistic use-case, we apply our methods to simulations of plasma micro-turbulence. Plasma turbulence determines the heat confinement properties of fusion reactors. Direct numerical calculation of the transport coefficients and the micro-instabilities that form the turbulence is computationally expensive and is a significant bottleneck in fusion plasma modelling [ 11 ] . The considerable number of geometric and thermodynamic parameters, the interactions that influence these coefficients and the resolutions needed to accurately resolve these modes, make direct numerical simulation for parameter space exploration computationally extremely challenging. The runtime of the simulation varies strongly between a few minutes up to several hours on a single node with two AMD EPYC 7702 64-core processors. Furthermore, the simulation runtime is not easily predicted for a given set of inputs. Therefore, significant effort is being invested in the production of machine-learnt surrogate models built from datasets of thousands of simulations [ 12 , 13 , 14 , 15 , 16 , 17 ] . The end goal in our use case is the computation of multiple integrals for a given set of input parameters, as described in [ 18 ] . Such computations are common in UQ workflows, where objectives often include evaluating expectations of quantities of interest or determining the maximum a posteriori probability point. Efficient evaluation of these integrals requires numerous forward model evaluations at different integration points. These evaluations are not always embarrassingly parallel, e.g. due to dependencies between integration points, such as in adaptive quadrature methods. Additionally, significant variations in runtime across integration points complicate scheduling. On High-Performance Computing (HPC) systems, one approach to this scheduling problem would be to incorporate both the physical (forward) model and the UQ algorithm into one monolithic implementation. This approach can be facilitated by, e.g. Dask [ 19 ] and Ray [ 20 ] . Both are Python libraries for parallel computing which can be embedded as meta-schedulers within a larger job managed by traditional HPC schedulers. This allows for fine-grained scheduling of tasks but requires the user to manually handle resource allocation by manually creating new SLURM/PBS jobs in order to utilise more resources. This manual handling may lead to a worse user experience and resource underutilisation. Dask and Ray are specific to Python workloads, and do not work out of the box as a generalised solution for any tasks written in other languages. In many real world settings, the main bottleneck of Dask lies in its runtime overhead [ 21 ] . Furthermore, these approaches require the UQ specialist and the application specialist to collaborate throughout the project. The UQ algorithm and modelling software need to be written in compatible languages and use compatible parallelisation techniques, furthering the complexity of incorporating such an approach. Alternatively, it is possible to use workflow management tools such as Snakemake [ 22 ] that can aggregate multiple tasks into groups. This helps reduce the number of jobs submitted and thereby decreases the overhead introduced by HPC schedulers such as SLURM/PBS. The main limitation is that Snakemake performs this grouping eagerly, i.e. prior to the execution of tasks. This means that tasks are not dynamically balanced or scheduled in real-time, leading to sub-optimal node and core utilisation. Merlin [ 23 ] also offers load balancing and automatic task aggregation. However, the load balancing capabilities is constrained since Merlin requires users to define task queues with fixed concurrency—meaning no dynamic resource allocation. This tool is also complicated to set up, as it has dependencies on multiple external services, making it less accessible for many potential users. In contrast, HyperQueue (HQ) [ 24 ] has minimal dependencies, it is deployable to a large number of systems and allow for dynamical load balancing across all allocated nodes and CPU cores. HQ does this by deploying a lightweight server that manages the task scheduling and distribution across the computing cluster. This is similar to Kubernetes, where it coordinates a network of worker nodes that execute these tasks. These worker nodes can be manually designated or spawned on demand with a worker allocation —allowing for the scaling of compute resources with workload demands. This setup