Predicting the Performance of Scientific Workflow Tasks for Cluster Resource Management: An Overview of the State of the Art
Jonathan Bader, Kathleen West, Soeren Becker, Svetlana Kulagina, Fabian Lehmann, Lauritz Thamsen, Henning Meyerhenke, Odej Kao · 2025
Read Predicting the Performance of Scientific Workflow Tasks for Cluster Resource Management: An Overview of the State of the Art by Jonathan Bader, Kathleen West, Soeren Becker, Svetlana Kulagina, Fabian Lehmann, Lauritz Thamsen, Henning Meyerhenke, Odej Kao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20867.txt.
Introduction
Jonathan Bader and co-authors survey how to guess a scientific workflow's appetite before it runs in this 2025 book chapter. Big analyses are chopped into tasks spread over clusters, and the scheduler must feed each task the right memory and time. Guesses typed in by users are often wrong, wasting machines or killing jobs.
The chapter sorts prediction methods by what they use, from past runs of the same task to models of the input data, and compares the leading approaches in detail. Good forecasts pay off beyond speed: they enable energy-saving schedules, lower carbon plans, and honest cost estimates for researchers renting compute. The authors map where the field stands and where the open problems sit, offering the overview as a guide for system builders.
Catalog record
| CREATOR | Jonathan Bader, Kathleen West, Soeren Becker, Svetlana Kulagina, Fabian Lehmann, Lauritz Thamsen, Henning Meyerhenke, Odej Kao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20867
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Predicting the Performance of Scientific Workflow Tasks for Cluster Resource Management: An Overview of the State of the Art Jonathan Bader, Kathleen West, Soeren Becker, Svetlana Kulagina, Fabian Lehmann, Lauritz Thamsen, Henning Meyerhenke, Odej Kao Abstract Scientific workflow management systems support large-scale data analysis on cluster infrastructures. For this, they interact with resource managers which schedule workflow tasks onto cluster nodes. In addition to workflow task descriptions, resource managers rely on task performance estimates such as main memory consumption and runtime to efficiently manage cluster resources. Such performance estimates should be automated, as user-based task performance estimates are error-prone. In this book chapter, we describe key characteristics of methods for workflow task runtime and memory prediction, provide an overview and a detailed comparison of state-of-the-art methods from the literature, and discuss how workflow task performance prediction is useful for scheduling, energy-efficient and carbon-aware computing, and cost prediction. 1 Introduction When running a scientific workflow on a computing infrastructure, scientists usually have to define task resource consumption limits, such as the available time [ BLSHS05 ] or memory [ WvSL19 ] for task execution. Setting these resource limits correctly is necessary to enable a successful execution of workflows as exceeding runtime or memory limits results in failures, commonly through errors issued by cluster resource managers. Resource consumption estimates provided by the scientists are known to be error-prone and inaccurate [ PWCT21 , HCTY+12 , WvSL19 ] . To avoid a failure and timely re-execution of the workflow or parts of it, scientists tend to overprovision resources to ensure that their workflows are fully executed. The problem is further aggravated as different instances of the same workflow task consume varying amounts of resources [ PWCT21 , LBDM+24 ] when executed on different inputs. Assume, for example, that a task unzips its input files. Such a task’s runtime will heavily depend on the size of the input files. Hence, scientists have to conservatively configure consumption limits for the largest task instances. While a failed workflow execution due to an underestimated resource consumption directly affects scientists, an overestimation does so indirectly: It reduces the number of tasks running in parallel as more cluster resources than necessary are allocated to tasks. The cluster resource manager ensures that the requested memory is reserved for specific tasks, even if it is not fully used, leaving less memory for other tasks to run as well. This effectively decreases the level of overall parallelism and, hence, cluster throughput. Meanwhile, overestimating the runtime of tasks can lead to significantly suboptimal schedules as the scheduler assumes wrong runtimes [ RVTBJ+17 , IE18 ] . Predicting the performance of workflow tasks to set their resource consumption limits automatically can relieve scientists of the need to define limits manually. State-of-the-art approaches often focus on either runtime or memory prediction, as these are key examples of performance prediction where underestimation leads to task termination in resource managers [ RVTBJ+17 , WBGL19 , TLM+22 ] . Many approaches use machine learning to predict the performance of workflow tasks. Some of these approaches rely on historical data to model a task’s performance [ HRB18 , WvSL19 ] , while other approaches learn during the workflow execution and incrementally update their model online [ WWL19 , TdSJ+17 ] . In addition, some approaches use profiling to predict the performance of workflow tasks before the workflows are executed [ BLT+22 , BLT+24a ] . The approaches also differ in their ability to predict performance for different types of infrastructure, such as homogeneous or heterogeneous infrastructures, their used prediction model, or their model input. Figure 1: Interaction between a scientific workflow management system, a resource manager, and task performance prediction models. Figure 1 provides a high-level overview of the interactions among components within the scientific workflow execution environment. The Scientific Workflow Management System (SWMS) parses the abstract workflow and coordinates task submission to the resource manager, which then schedules and allocates tasks onto the cluster infrastructure, concurrently monitoring their resource usage. Workflow task performance prediction employs predictive models to generate runtime, memory, and other resource-related estimations for individual tasks. These predictions are accessible to both the SWMS and the resource manager. Utilizing predictions at the SWMS level offers the advantage of leveraging workflow graph information, which is typically unavailable to the resource manager. Conversely, the resource manager benefits from a global view, enabling it to optimize resource allocation decisions across multiple workflows and users. Recent work proposes unified solutions that leverage information from both components, enabling direct integration of task performance predictions [ ABC+20 , LBT+23 ] . In this chapter, we give an overview of state-of-the-art workflow task performance prediction methods, compare them, and discuss use cases as well as ongoing research. In particular, we compare methods that focus on workflow task runtime and workflow task memory prediction. That is, we do not include methods that only predict the total execution time or total memory consumption of entire workflow applications, as task-level predictions serve as fine-grained foundation for workflow-level predictions. Likewise, we do not include methods that are not specific to scientific workflows. The main contributions of this chapter are: • We identify key characteristics of workflow task runtime (Section 2.1 ) and memory prediction methods (Section 3.1 ), covering general aspects such as code availability, model-specific criteria like the prediction method and applicability to heterogeneous infrastructures, as well as model inputs and evaluation settings. • We provide an overview and a detailed comparison of state-of-the-art workflow task runtime (Section 2 ) and memory prediction methods (Section 3 ), analyzing nineteen state-of-the-art papers based on the previously defined characteristics and identifying shortcomings while highlighting future research directions. • We exemplify how workflow task performance prediction is useful for resource management, namely scheduling, energy-efficient and carbon-aware computing, and cost prediction, and present recent research in each of these fields while also highlighting limitations (Section 4 ). The remainder of this chapter is structured as follows. Section 2 describes state-of-the-art workflow task runtime prediction methods and compares them, while Section 3 describes state-of-the-art workflow task memory prediction methods. Section 4 presents applications of workflow task performance prediction for resource management and discusses the shortcomings of state-of-the-art methods. Lastly, Section 5 concludes the chapter. Table 1: Overview of state-of-the-art methods for workflow task runtime prediction. General Model Model Input Evaluation Publication Code Available Domain Specific Prediction Method Training Time Heterogeneous Infrastructure GPU Support Hardware Capacity Hardware Performance Task Input Size Task Resource Usage Workflow System # Workflows Experiment Type [ MF10 ] Diverse Offline - - Sim. Real [ MFP13 ] NN Offline Askalon 3 Sim. Real [ DSJD+13 ] Regr. & Clust. Online Pegasus 3 Sim. Real [ DSJR+15 ] Regr. & Clust. Online Pegasus 5 Sim. Real [ PDF17 ] Regr. Offline - 4 Sim. Syn. [ NAM+17 ] NN Offline - 3 Sim. Real [ HRB18 ] NN Online Diverse 2 Sim. Real [ BLT+22 ] Regr. Pre Nextflow 5 Sim. Real [ HCP+23 ] NN Offline - - Sim. Real [ BLT+24a ] Regr. Pre Nextflow 5 Sim. Real Yes — No — Partially 2 Workflow Task Runtime Prediction In this chapter, we explore the features of runtime prediction methods for workflow tasks and offer a detailed