Mathematical Foundations of Modeling ETL Process Chains

L. Maier, L. Schulze, R. Lilow, L. Hahn, N. Krasowski, A. Barth, S. Gaebel, F. Güran, O. Hanau, G. Wagner, F. Borgmann, O. Arenz, J. Peters, [6pt] Deepshore GmbH, Germany · 2026

Read Mathematical Foundations of Modeling ETL Process Chains by L. Maier, L. Schulze, R. Lilow, L. Hahn, N. Krasowski, A. Barth, S. Gaebel, F. Güran, O. Hanau, G. Wagner, F. Borgmann, O. Arenz, J. Peters, [6pt] Deepshore GmbH, Germany (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29877.txt.

Introduction

Maier and co-authors build a math model of chained extract-transform-load processes. Thread counts tune throughput in each ETL phase. But random per-record times and chain bottlenecks complicate that link. They model the chain as a controlled discrete-time Markov process on a graph. Mean throughput rises with threads but with shrinking gains. A flow balance rule ties threads, throughput, and mean time. Heavy-tailed random times complete the model. It gives a sound simulator for resource control work.

Catalog record

CREATORL. Maier, L. Schulze, R. Lilow, L. Hahn, N. Krasowski, A. Barth, S. Gaebel, F. Güran, O. Hanau, G. Wagner, F. Borgmann, O. Arenz, J. Peters, [6pt] Deepshore GmbH, Germany
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25517
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.29877

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mathematical Foundations of Modeling ETL Process Chains L. Maier, L. Schulze, R. Lilow, L. Hahn, N. Krasowski, A. Barth, S. Gaebel, F. Güran, O. Hanau, G. Wagner, F. Borgmann, O. Arenz, J. Peters, [6pt] Deepshore GmbH, Germany Abstract Extract-Transform-Load (ETL) processes are core components of modern data processing infrastructures. The throughput of processed data records can be adjusted by changing the amount of allocated resources, i.e. the number of parallel processing threads for each of the three ETL phases, but also depends on stochastic variations in the per-record processing times. In chains of multiple consecutive ETL processes, the relation between allocated resources and overall throughput is further complicated, for example by the occurrence of bottlenecks affecting all subsequent ETL processes. We develop a mathematical model of ETL process chains that is accurate at the level of time-aggregated throughput and suitable for efficient simulation. The process chain is represented as a controlled discrete-time Markov process on a directed acyclic graph whose edges are individual ETL processes. We model the mean throughput as a bounded, monotone function of the number of parallel threads, to capture the diminishing benefit of allocating more threads. We furthermore introduce a Flow Balance postulate linking number of threads, mean throughput, and mean processing time. The stochastic processing times are then modeled by non-negative heavy-tailed distributions around the mean processing time. This framework provides a principled simulator for ETL networks and a foundation for learning- and control-based resource allocation. 1 Introduction In our data-driven world, most services we take for granted rely on massive volumes of data being transferred, processed and analyzed every day—bank transactions, public transport, electronic payments, social media, to name just a few. Although the specific ways of handling the data are as diverse as the services themselves, they generally follow the same universal pattern of an ETL process: E xtract data from a source system, T ransform it, and finally L oad it into a target system. Often, multiple ETL processes are performed consecutively, to achieve more complex data handling tasks. To increase processing throughput, each of the three phases in an ETL process can handle multiple data records in parallel. However, the degree of parallelization determines the required allocation of computing resources to the individual phases. Due to the ever growing volume of data being processed, striking the balance between performance and resource efficiency becomes an increasingly important challenge. This requires tuning the resource allocation to multiple consecutive processing steps in a way that avoids bottlenecks and idling resources. Up to now, this is mainly achieved by human expertise, which is, however, labor-intensive and suboptimal, in particular if the number of incoming records varies over time. Although there are existing auto-scaling mechanisms assisting with this, these do not account for the dependencies between consecutive steps and also require human expertise for their configuration. Furthermore, any real deployment of ETL processes is inevitably subject to fluctuations in the processing performance caused by interactions with other IT processes running on the same hardware or independently accessing the same data stores. Since these environmental interactions are out of our control, it is crucial to adapt to such fluctuations by dynamically optimizing the resource allocation, for example, to maximize throughput while minimizing resource usage. Such optimization could be performed using online or offline methods. Online methods would optimize resource allocation during operation by leveraging a mathematical model of the ETL process to predict future system behavior. A prominent example is Model Predictive Control (MPC) ( 4 ) , which optimizes actions (resource allocations) across a finite, receding time horizon. In this approach, the optimizer generates a sequence of allocations for several upcoming discrete time steps, but the controller applies only the action for the immediate next step. At each subsequent time interval, the optimization is repeated with a shifted horizon and an updated system state. Similarly, planning methods like Monte-Carlo Tree Search (MCTS) ( 1 ) can be applied by iteratively building a tree of action sequences using random roll-outs from the model to find the best next action. Offline optimization, on the other hand, would be performed prior to deployment. For example, Reinforcement Learning (RL) ( 5 ; 6 ) would learn a policy that maps the current system state to an action using the system model for simulation. By not relying on real-time optimization, such an approach can potentially produce better behavior by allowing for more extensive exploration of the state space during the training phase. The aim of this work is to construct a mathematical model suitable for future applications of the aforementioned optimization approaches. To achieve this, we develop a novel modeling framework that enables realistic simulation and admits a formulation in the language of sequential decision making. The main contributions of this work are as follows: • Flow Balance postulate for ETL phases. We introduce the Flow Balance postulate ( Section 3.2 ), linking parallel resource allocation, mean throughput, and mean processing time at the level of individual ETL phases. To the best of our knowledge, this relation has not been formalized previously in the context of ETL systems. • Bounded monotone throughput model. We propose a mathematically convenient and empirically motivated class of bounded, monotone throughput curves ( Section 3.2 ), which captures diminishing returns of parallelism while remaining suitable for gradient-based optimization and calibration against data. • Markovian formulation of ETL dynamics. We model the full ETL process chain as a controlled Markov process with explicit state, action, and transition definitions ( Section 3.1 ). This formulation captures bottlenecks and other interactions between linked ETL processes, enabling realistic simulation of complex ETL workflows. • A simulation model designed for learning and control. By combining the above modeling choices, we obtain a discrete-event simulation that is both faithful to real ETL system behavior and explicitly compatible with reinforcement learning and other sequential decision-making methods. While we do not address optimization in this work, the formulation provides the necessary foundation for future studies of adaptive resource allocation in ETL systems. Outline of the article. We begin in Section 2 by formalizing the problem setting and introducing a graph representation of ETL process chains that serves as the foundation for a broader research program, starting with the present work. Next, in Section 3 , we introduce a detailed mathematical model of ETL processes starting in Section 3.1 with a mathematical framework for discrete-event simulation of ETL systems. This is followed by Section 3.2 , where we model the processing times based on the two central postulates Section 3.2 and Section 3.2 regarding the mean throughput and mean processing times, supported by empirical tests. We close both subsections by discussing directions for future research. Acknowledgments. The authors thank Sven Behncke, Theresa Bick, Michael Brünker, Jan Langer, Giancarlo Meyer, Yvonne Reckmann, Henrik Riedel and Nora Stroetzel for their ongoing support of this project and insightful discussions. The authors acknowledge funding by the German Federal Ministry of Research, Technology and Space (BMFTR) under project ETL4Balance (FKZ 16IS23057A) and as part of the Robotics Institute Germany (RIG). Calculations for this research were conducted on the Lichtenberg high-performance computer of the TU Darmstadt. 2 Problem Setting: ETL Process Chains We consider process chains that iteratively process data records

Read the complete text