Scheduling Data-Intensive Workloads in Large-Scale Distributed Systems: Trends and Challenges
Georgios L. Stavrinides and Helen D. Karatza · 2025
Read Scheduling Data-Intensive Workloads in Large-Scale Distributed Systems: Trends and Challenges by Georgios L. Stavrinides and Helen D. Karatza (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25362.txt.
Introduction
Georgios L Stavrinides and Helen D Karatza survey workload scheduling. Their 2025 chapter concerns large-scale distributed systems. Big data growth makes workloads more complex. Systems are growing in scale and capacity. Data-intensive apps differ in parallelism and locality needs.
Workloads also impose quality of service requirements. Time constraints and failure resilience matter greatly. Energy efficiency is another common objective today. The chapter proposes a classification of workloads. It overviews common scheduling methods used in literature. Novel strategies from recent work are presented too. The authors highlight open challenges for scholars. Future directions for large-scale scheduling are discussed. Readers get a structured map of the field.
Catalog record
| CREATOR | Georgios L. Stavrinides and Helen D. Karatza |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-25362
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scheduling Data-Intensive Workloads in Large-Scale Distributed Systems: Trends and Challenges Georgios L. Stavrinides and Helen D. Karatza Abstract With the explosive growth of big data, workloads tend to get more complex and computationally demanding. Such applications are processed on distributed interconnected resources that are becoming larger in scale and computational capacity. Data-intensive applications may have different degrees of parallelism and must effectively exploit data locality. Furthermore, they may impose several Quality of Service requirements, such as time constraints and resilience against failures, as well as other objectives, like energy efficiency. These features of the workloads, as well as the inherent characteristics of the computing resources required to process them, present major challenges that require the employment of effective scheduling techniques. In this chapter, a classification of data-intensive workloads is proposed and an overview of the most commonly used approaches for their scheduling in large-scale distributed systems is given. We present novel strategies that have been proposed in the literature and shed light on open challenges and future directions. 1 Introduction The ever-increasing momentum of the Internet of Things, the rapid pace of technological advances in mobile devices and cloud computing, as well as the explosive growth of social media, produce an overwhelming flow of data of unprecedented volume and variety at a record rate. Such data are commonly referred to as big data and are characterized by the following attributes: (a) volume, i.e. they consist of very large datasets, (b) variety, i.e. they comprise diverse structured and unstructured data of various types and (c) velocity, i.e. the data are generated and streamed at staggering speeds Russom 2011 ; Hashem et al 2015 . Computationally intensive applications are employed in a wide spectrum of domains such as healthcare, science, engineering, business and finance, in order to unleash the power of big data, extract useful knowledge and gain valuable insights. Talia 2013 . The advent of big data has called for a paradigm shift in the computer architecture, and consequently the applications, required for their effective processing. Data-intensive applications are typically processed on interconnected computing resources that are geographically distributed, encompass various heterogeneous components, utilize virtualization, feature multi-tenancy and are able to scale up in the foreseeable future. Computer clusters, computational grids and clouds are examples of such platforms Foster et al 2008 . Furthermore, novel hybrid approaches have emerged, such as fog computing , which extends the cloud computing paradigm by bringing data processing at computational resources at the edge of the network, closer to where the data are generated, while sending selected data to the cloud for historical analysis and long-term storage Cisco 2015 ; Bonomi et al 2014 . Data-intensive applications may have different degrees of parallelism and must effectively exploit data locality. Furthermore, they may also impose several Quality of Service (QoS) requirements, such as time constraints and resilience against failures, as well as other objectives, like energy efficiency. These features of the workloads operating on big data, as well as the characteristics of the computing resources required to process them, present major challenges that require the employment of effective scheduling algorithms . Due to their inherent complexity, the performance of such algorithms is usually evaluated by simulation, rather than by analytical methods. Analytical modeling is difficult and often requires several simplifying assumptions that may have an unpredictable impact on the results Stavrinides and Karatza 2017b . This chapter is organized as follows: Sect. 2 gives a definition of the scheduling problem in large-scale distributed systems, as well as some of the most important scheduling objectives. In Sect. 3 , a classification of data-intensive workloads is proposed, according to their degree of parallelism. An overview of the most widely used strategies for the scheduling of each class of data-intensive applications in large-scale distributed systems is given. Sect. 4 presents some of the major challenges of data-intensive workload scheduling, covering topics such as data locality awareness, timeliness, fault tolerance and energy efficiency. Furthermore, novel strategies that have been proposed in the literature are presented in Sect. 5 . Finally, Sect. 6 concludes this chapter, shedding light on open challenges and future research directions. 2 Scheduling Problem In its general form, the scheduling problem in large-scale distributed systems concerns the mapping of a set of application tasks V=\{n_{1},n_{2},...,n_{N}\} to a set of processors P=\{p_{1},p_{2},...,p_{Q}\} , in order to complete all tasks under the specified constraints (e.g. complete each task within its deadline) Buttazzo 2011 ; Kolodziej 2012 . In this general form, the scheduling problem has been shown to be NP-complete Garey and Johnson 1979 . 2.1 Scheduling Objectives Some of the parameters that characterize a task n_{i}\in V are shown in Fig. 1 . These parameters are: • arrival time a(n_{i}) : it is the time at which the task arrives at the system. • start time s(n_{i}) : it is the time at which the task starts its execution. • finish time f(n_{i}) : it is the time at which the task finishes its execution. • deadline d(n_{i}) : it is the time before which the task should finish its execution. Figure 1: Typical parameters that characterize a task of an application submitted for execution in a large-scale distributed system. Based on the above parameters, some of the most commonly used scheduling objectives in large-scale distributed systems are: (a) To minimize the average response time \overline{R} of the tasks n_{i}\in V , where \overline{R} is given by: \overline{R}=\frac{1}{N}\sum_{n_{i}\in V}R(n_{i}) (1) where R(n_{i})=f(n_{i})-a(n_{i}) and N is the number of tasks in V . (b) To minimize the makespan (i.e. total execution time) M of the tasks n_{i}\in V , where M is defined as: M=\max_{n_{i}\in V}{\left\{f(n_{i})\right\}}-\min_{n_{i}\in V}{\left\{s(n_{i})\right\}} (2) (c) To maximize the task guarantee ratio TGR of the tasks n_{i}\in V , where TGR is given by: TGR=\frac{1}{N}\sum_{n_{i}\in V}guar(n_{i}) (3) where guar(n_{i})=\begin{cases}1&\text{if }f(n_{i})\leq d(n_{i})\\ 0&\text{otherwise}\end{cases} (4) (d) To minimize the average tardiness \overline{T} of the tasks n_{i}\in V , where \overline{T} is defined as: \overline{T}=\frac{1}{N}\sum_{n_{i}\in V}T(n_{i}) (5) where T(n_{i})=\begin{cases}f(n_{i})-d(n_{i})&\text{if }f(n_{i})>d(n_{i})\\ 0&\text{otherwise}\end{cases} (6) 3 Data-Intensive Workloads in Large-Scale Distributed Systems The data-intensive applications scheduled for execution in large-scale distributed systems, typically consist of numerous component tasks. At the one end of the spectrum, the tasks require frequent communication with each other during their execution. At the other end of the spectrum, the component tasks do not require any communication and are completely independent. Between these two ends, is the case where communication is required between the component tasks of an application, but only before or after their execution. Consequently, data-intensive workloads in large-scale distributed systems can be classified into the following categories: • fine-grained parallel applications , • coarse-grained parallel applications and • embarrassingly parallel applications . In the following paragraphs, each class of data-intensive applications is presented in more detail and their corresponding, most widely used scheduling heuristics are analyzed. 3.1 Fine-Grained Parallel Applications An application features fine-grained parallelism when it consists of frequently communicating parallel tasks. A proven and effective way to schedule such applications is gang scheduling . According to this approach, the parallel tasks of an application form a gang and are scheduled and executed simultaneously on different processors. Hence, all of the tasks of the application start execution at the same time. This way, the