WOW: Workflow-Aware Data Movement and Task Scheduling for Dynamic Scientific Workflows
Fabian Lehmann 1 , Jonathan Bader 2 , Friedrich Tschirpke 1 , Ninon De Mecquenem 1 , Ansgar Lößer 3 , Soeren Becker 2 , Katarzyna Ewa Lewińska 1 4 , Lauritz Thamsen 5 , and Ulf Leser 1 · 2025
Read WOW: Workflow-Aware Data Movement and Task Scheduling for Dynamic Scientific Workflows by Fabian Lehmann 1 , Jonathan Bader 2 , Friedrich Tschirpke 1 , Ninon De Mecquenem 1 , Ansgar Lößer 3 , Soeren Becker 2 , Katarzyna Ewa Lewińska 1 4 , Lauritz Thamsen 5 , and Ulf Leser 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13072.txt.
Introduction
Fabian Lehmann and co-authors speed up science workflows in a 2025 paper. Such workflows chain many tasks over clusters of machines. Files pass between tasks through a shared store, while a manager hands tasks to machines. In current stacks, files land wherever, with no thought for which task runs next. That split causes network jams and slow runs.
Their system, WOW, steers both file moves and task picks with the workflow in mind. It plants advance copies of mid-way files where later tasks will likely run. That cuts traffic when the task starts. Built on a known engine with a standard cluster manager, WOW cut total run time on all 16 test workflows. Gains reached 94.5 percent on pattern flows.
Catalog record
| CREATOR | Fabian Lehmann 1 , Jonathan Bader 2 , Friedrich Tschirpke 1 , Ninon De Mecquenem 1 , Ansgar Lößer 3 , Soeren Becker 2 , Katarzyna Ewa Lewińska 1 4 , Lauritz Thamsen 5 , and Ulf Leser 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13072
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WOW: Workflow-Aware Data Movement and Task Scheduling for Dynamic Scientific Workflows Fabian Lehmann 1 , Jonathan Bader 2 , Friedrich Tschirpke 1 , Ninon De Mecquenem 1 , Ansgar Lößer 3 , Soeren Becker 2 , Katarzyna Ewa Lewińska 1 4 , Lauritz Thamsen 5 , and Ulf Leser 1 Abstract Scientific workflows process extensive data sets over clusters of independent nodes, which requires a complex stack of infrastructure components, especially a resource manager (RM) for task-to-node assignment, a distributed file system (DFS) for data exchange between tasks, and a workflow engine to control task dependencies. To enable a decoupled development and installation of these components, current architectures place intermediate data files during workflow execution independently of the future workload. In data-intensive applications, this separation results in suboptimal schedules, as tasks are often assigned to nodes lacking input data, causing network traffic and bottlenecks. This paper presents WOW, a new scheduling approach for dynamic scientific workflow systems that steers both data movement and task scheduling to reduce network congestion and overall runtime. For this, WOW creates speculative copies of intermediate files to prepare the execution of subsequently scheduled tasks. WOW supports modern workflow systems that gain flexibility through the dynamic construction of execution plans. We prototypically implemented WOW for the popular workflow engine Nextflow using Kubernetes as a resource manager. In experiments with 16 synthetic and real workflows, WOW reduced makespan in all cases, with improvement of up to 94.5% for workflow patterns and up to 53.2% for real workflows, at a moderate increase of temporary storage space. It also has favorable effects on CPU allocation and scales well with increasing cluster size. ©2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. I Introduction Many scientific disciplines recently faced a steep increase in the size and amount of collected experimental data. Such data is typically analyzed using a scientific workflow, which is a set of independently developed analysis programs (called tasks) connected through input/output relationships (called dependencies) managed by a scientific workflow engine. For processing large data sets, these engines execute workflows on a distributed cluster by leveraging further infrastructure components [ 1 ] . The most important components are resource managers (RMs) to assign tasks whose input is available to nodes with free capacity and fitting resources, and distributed file systems (DFSs) to offer transparent access to data throughout the cluster (see Figure 1 ). Nextflow, Parsl, and Pegasus are examples of popular workflow engines. Kubernetes and Slurm are widely used RMs. HDFS, Ceph, and NFS are relevant DFSs. Current architectures decouple the functionality of these three components to achieve a clear separation of concerns, with the known benefits for system development and maintenance [ 1 , 2 ] . However, this decoupling also yields significant disadvantages. The disadvantage we focus on in this work is that the placement of intermediate data files created during workflow execution is determined independently of future workflow tasks. The reason for this is that the different components pursue different optimization goals. A DFS determines file locations with the goals of a balanced storage load across nodes and improved fault tolerance. In contrast, a typical RM prioritizes the optimal distribution of the computational load of all ready tasks in the cluster. Therein, the DFS is oblivious to future workloads, and the RM is oblivious to file locations. Consequently, tasks read their input data from the DFS upon start-up and write their output to the DFS when finished. This is not critical when clusters are equipped with high-speed, low-latency networks, such as InfiniBand, or if a workflow’s runtime is heavily dominated by compute times. However, it leads to suboptimal schedules in data-intensive workflows executed over commodity networks, where data movement determines runtimes and, therefore, should be minimized as much as possible [ 3 , 4 , 5 , 6 , 7 , 2 ] . Fig. 1 : Overview of a cluster setup with a scientific workflow management system running on a resource manager. Nodes have a local file system (LFS) and access a distributed file system (DFS) to exchange data. Consider a sequence of n tasks where t_{1} initiates the workflow by reading its input and tasks t_{2} to t_{n} each read output from their preceding tasks. Typically, with a decentralized file system (DFS) independently managing data, all tasks fetch their inputs from the DFS and store their outputs in the DFS. This often results in unnecessary data transfers. For example, colocating t_{2} with t_{1} could retain t_{1} ’s output on the same node, eliminating the need for data movement. To achieve this, a central component must be able to steer both data placement and task scheduling in an intertwined manner, which is absent in current workflow systems. Traditional workflow scheduling algorithms can jointly optimize data movement and task-node assignments but require that the entire workflow, task runtimes, and intermediate file sizes are known upfront [ 8 , 4 , 3 , 9 ] . However, modern dynamic workflow engines create the workflow tasks dynamically based on the results of predecessor tasks. Moreover, the systems treat tasks as black boxes with unpredictable resource requirements and output sizes. In this paper, we propose WOW, a novel workflow-aware approach for dynamic, data-intensive scientific workflows on commodity clusters that efficiently intertwines data and task placement. WOW features a three-step scheduler that connects decisions on data movement and task assignment such that (a) tasks are always assigned to nodes where all their input data is locally available (possibly after moving them there proactively using a speculative approach) and (b) data movement and task execution are performed in parallel (meaning that data possibly is moved before it is required). To this end, the scheduler interacts with a dedicated data placement service (DPS), controlling all data movements during workflow execution to avoid network bottlenecks. We implemented WOW for the popular combination of Nextflow workflows on Kubernetes clusters and evaluated it using 16 real-life workflows of varying complexity and from different scientific domains over two different DFSs, different commodity cluster sizes, and different network bandwidths. Our experiments show that WOW achieves makespan improvements of up to 53.2% for real-world workflows and 94.5% for common workflow patterns. WOW can thus significantly speed up the processing of data-intensive scientific workflows on commodity networks. II Background In this section, we examine how resource managers, distributed file systems, and workflow engines interact in current systems, leading to unnecessary network bottlenecks and longer execution times. II-A Scientific Workflow Engine A scientific workflow engine comprises a workflow language and an execution engine [ 10 , 1 , 11 ] . Workflow languages define tasks and dependencies, where tasks are custom program binaries or scripts for data analysis, and dependencies represent input-output relationships. The execution engine interprets the workflow iteratively, identifying ready tasks whose inputs are available and handing them to the RM’s job queue for cluster execution. The RM continuously signals back all tasks whose execution has finished