Nf-PEAK: Process-Based Energy Attribution for Nextflow Workflows on Kubernetes Clusters
Philipp Thamm, Somayeh Mohammadi, Kathleen West, Knut Reinert, Lauritz Thamsen, Ulf Leser · 2026
Read Nf-PEAK: Process-Based Energy Attribution for Nextflow Workflows on Kubernetes Clusters by Philipp Thamm, Somayeh Mohammadi, Kathleen West, Knut Reinert, Lauritz Thamsen, Ulf Leser (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22393.txt.
Introduction
Thamm and colleagues present Nf-PEAK, a method that attributes energy use to single Nextflow tasks on Kubernetes clusters. Node counters alone cannot split energy across tasks, and shared clusters add noise. Nf-PEAK finds workflow pods, maps them to host processes, and applies an energy-credit model to counter samples. On three nf-core workflows, mean error hits 6.6 percent alone and 10.9 percent under heavy co-located load. It beats Kepler on average, above all under shared load, and stays stable from 2 to 8 nodes.
Catalog record
| CREATOR | Philipp Thamm, Somayeh Mohammadi, Kathleen West, Knut Reinert, Lauritz Thamsen, Ulf Leser |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46469 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22393
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Nf-PEAK: Process-Based Energy Attribution for Nextflow Workflows on Kubernetes Clusters Philipp Thamm, Somayeh Mohammadi, Kathleen West, Knut Reinert, Lauritz Thamsen, Ulf Leser Abstract Scientific workflows are pipelines of interdependent tasks. They are increasingly executed on shared Kubernetes clusters via workflow engines such as Nextflow. Their energy consumption matters for both cost and sustainability. It is necessary to examine and optimize workflow tasks individually, because they can be very heterogeneous. However, estimating task-level energy on clusters is difficult: Intel RAPL counters report only node-level energy, access to counters and host process information is typically restricted, and concurrent workloads introduce resource contention and measurement noise. We present Nf-PEAK , a containerized method to attribute CPU-package and DRAM energy to individual processes and Nextflow tasks . Nf-PEAK (i) identifies workflow pods, (ii) maps pods to host processes via cgroup metadata, (iii) samples RAPL and per-process performance counters, and (iv) applies a non-linear energy-credit model before aggregating results at task level. On a Kubernetes cluster, we evaluate three nf-core workflows under controlled co-located CPU load. Nf-PEAK reaches an average Mean Absolute Percentage Error of 6.6% in isolated runs and 10.9% when an unrelated workload saturates 8 of 32 hardware threads per node, and remains stable across 2, 3, 4, and 8 nodes. Compared to the state-of-the-art Kubernetes tool Kepler, Nf-PEAK yields lower error on average, particularly under co-located load. 1 Introduction Extracting information from a dataset can require multiple interdependent processing steps. Such pipelines of tasks are referred to as scientific workflows [ 17 ] and are used in many areas, such as genomics [ 9 ] or remote sensing [ 20 ] . Workflow configurations are commonly optimized for fast execution, but recently energy-efficient computing has received more attention due to environmental concerns and rising energy costs [ 11 , 7 , 4 ] . For example, one run of the workflow Sarek 1 1 1 https://nf-co.re/sarek/3.5.1/, last accessed: March 22, 2026 for genome variant detection consumes more than 1.16 MJ of energy across 4 cluster nodes over a runtime of 56 minutes in our experiments with 10.6 GB of real-world input data. Of this energy, about 0.7 MJ is attributable to the workflow itself after subtracting static energy consumption of the cluster and overhead introduced by the Kubernetes scheduler, equivalent to drawing an average power of 208.3 W over the workflow’s runtime. Optimizing a workflow for energy efficiency requires feedback beyond node-level totals: workflow developers and users need to understand which tasks dominate energy consumption and how configuration changes affect them. A common approach to obtain energy information on Intel systems is to use Running Average Power Limit (RAPL) energy counters [ 6 , 15 ] , which provide low-overhead energy readings for CPU and (depending on the CPU model) DRAM. While access to RAPL is straightforward on local hardware through tools such as perf 2 2 2 https://perfwiki.github.io/main/, last accessed: March 22, 2026 or powercap 3 3 3 https://docs.kernel.org/power/powercap/powercap.html, last accessed: March 22, 2026 , extracting accurate task-level information on managed clusters is challenging for several reasons: • Limited access: User privileges are often limited, making it impossible to install supporting tools on nodes or to access RAPL counters and host-wide process information. • Containerization: Cluster workloads are commonly containerized, which restricts access to RAPL and to processes running outside the current container. • Resource manager: Workloads are scheduled on the cluster by a resource manager sitting between workflow engine and nodes, preventing knowledge about the location of individual tasks prior to execution. • Distributed execution: Workflows are executed across multiple nodes, so measurement must be coordinated and aggregated. • Complex node architecture: Multi-socket CPUs and NUMA memory require socket-aware monitoring and attribution. • Shared environment: Multiple workloads may run on the same nodes concurrently, but RAPL only counts at CPU package level, aggregating energy across unrelated workloads. In this paper, we present Nf-PEAK ( P rocess-based E nergy A ttribution on K ubernetes clusters targeting N ext f low workflows), a method to estimate the energy consumption of individual processes and workflow tasks on Kubernetes clusters. In summary, Nf-PEAK combines RAPL measurements with per-process performance counters and uses a non-linear attribution strategy inspired by EnergAt [ 14 ] . The tool is fully containerized and can be deployed through Kubernetes without the need to install special software on cluster nodes. Our contributions are: • Containerized task attribution on Kubernetes: a practical pipeline to discover Nextflow tasks, map them to host processes, and monitor CPU and DRAM energy on all involved nodes. • Process-level non-linear attribution: an energy-credit model adapted to process granularity, enabling accurate CPU and DRAM energy attribution for heterogeneous tasks and concurrent cluster workloads. • Evaluation on real workflows: experiments with three nf-core workflows across four cluster sizes (2, 3, 4, and 8 nodes) and under controlled co-located CPU load, including a comparison to Kepler [ 1 , 2 ] . In the following, we first provide background about the technologies used in our work. In Section 3 , we present how Nf-PEAK attributes energy consumption and integrates with Kubernetes. Our experiments and their results are presented in Section 4 and discussed in Section 5 . Section 6 presents related work in the area of software-based energy attribution. Finally, Section 7 concludes the paper. 2 Background This section summarizes the technologies and concepts most relevant to our work: Executing scientific workflows on Kubernetes, RAPL energy counters, and energy attribution strategies. 2.1 Scientific Workflows on Kubernetes Kubernetes [ 5 ] is a widely used container orchestration system that coordinates the execution of workloads, including scientific workflows, on compute clusters. It implements scheduling and re-scheduling of tasks on a distributed infrastructure, and also supports automatic resource allocation and load balancing. Workloads are executed on the nodes through containers, packaging programs together with their dependencies and execution environment. These containers are running in isolation on top of the node’s operating system’s kernel. The containers utilized by Kubernetes are commonly built and managed by Docker [ 19 ] . They are organized in Kubernetes pods. Each pod contains one or multiple containers and is scheduled independently. Scientific workflows decompose an analysis into separate logical tasks with explicit data dependencies [ 12 ] . Nextflow [ 8 ] expresses logical tasks in a DSL and orchestrates their execution in the form of physical tasks on a variety of backends, including Kubernetes. Here, each logical task is split into one or multiple physical tasks, depending on the task and workflow configuration. In a Kubernetes backend, Nextflow executes each physical workflow task in an isolated pod/container (e.g., Docker), using Kubernetes to orchestrate execution on a compute cluster [ 21 ] and to manage the available resources. 2.2 RAPL Energy Counters Intel RAPL exposes energy counters for different domains (depending on CPU model), such as Package (CPU package) and DRAM [ 15 ] . RAPL readings are low-overhead and widely used for power/energy estimation [ 13 ] . However, RAPL counters are coarse-grained : each CPU provides only aggregate energy for all workloads on that CPU and directly associated memory. In shared clusters, this makes it impossible to directly infer task-level energy from RAPL alone. Furthermore, energy counters are bounded registers and overflow after a certain