RCOMPSs: A Scalable Runtime System for R Code Execution on Manycore Systems

Xiran Zhang, Javier Conejero, Sameh Abdulah, Jorge Ejarque, Ying Sun, Rosa M. Badia, David E. Keyes, Marc G. Genton · 2025

Read RCOMPSs: A Scalable Runtime System for R Code Execution on Manycore Systems by Xiran Zhang, Javier Conejero, Sameh Abdulah, Jorge Ejarque, Ying Sun, Rosa M. Badia, David E. Keyes, Marc G. Genton (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06896.txt.

Introduction

Xiran Zhang and co-authors present a system that runs the R language in parallel in this 2025 paper. R is a mainstay of science and statistics work, with a huge set of add-on packages. Yet its built-in support for parallel work is weak, which hurts as data and computing demands grow.

Their system, called RCOMPSs, splits code into tasks behind the scenes. Users write in a plain step-by-step style, while the system tracks what depends on what and schedules work across cores and machines. Tests on two supercomputers, with three common data tasks, show good scaling up to many cores and nodes. Two tasks kept efficiency above 70 percent in most settings.

Catalog record

CREATORXiran Zhang, Javier Conejero, Sameh Abdulah, Jorge Ejarque, Ying Sun, Rosa M. Badia, David E. Keyes, Marc G. Genton
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06896

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RCOMPSs: A Scalable Runtime System for R Code Execution on Manycore Systems Xiran Zhang, Javier Conejero, Sameh Abdulah, Jorge Ejarque, Ying Sun, Rosa M. Badia, David E. Keyes, Marc G. Genton Abstract . R has become a cornerstone of scientific and statistical computing due to its extensive package ecosystem, expressive syntax, and strong support for reproducible analysis. However, as data sizes and computational demands grow, native R parallelism support remains limited. This paper presents RCOMPSs, a scalable runtime system that enables efficient parallel execution of R applications on multicore and manycore systems. RCOMPSs adopts a dynamic, task-based programming model, allowing users to write code in a sequential style, while the runtime automatically handles asynchronous task execution, dependency tracking, and scheduling across available resources. We present RCOMPSs using three representative data analysis algorithms, i.e., K-nearest neighbors (KNN) classification, K-means clustering, and linear regression and evaluate their performance on two modern HPC systems: KAUST Shaheen-III and Barcelona Supercomputing Center (BSC) MareNostrum 5. Experimental results reveal that RCOMPSs demonstrates both strong and weak scalability on up to 128 cores per node and across 32 nodes. For KNN and K-means, parallel efficiency remains above 70% in most settings, while linear regression maintains acceptable performance under shared and distributed memory configurations despite its deeper task dependencies. Overall, RCOMPSs significantly enhances the parallel capabilities of R with minimal, automated, and runtime-aware user intervention, making it a practical solution for large-scale data analytics in high-performance environments. Conference: The International Conference for High Performance Computing, Networking, Storage, and Analysis; Nov 16–21, 2025; St. Louis, MO 1. Introduction Statistical computing is critical for extracting insights from data, enabling researchers and analysts to model uncertainty, test hypotheses, and make data-driven decisions. Among the available tools, the R programming language is a powerful and widely used platform for statistical analysis and data science ( R Core Team, 2023 ) . Its extensive ecosystem of packages, designed for statistical modeling, machine learning, data visualization, and reporting, makes R especially suitable for handling complex analytical tasks ( Wickham and Grolemund, 2017 ) . Furthermore, its strong focus on reproducibility and active community support have established it as a core tool in data analytics, scientific research, and academic use. Traditional sequential processing in R becomes a bottleneck as data volumes grow, particularly when dealing with large datasets or computationally intensive tasks. Parallel programming addresses this challenge by enabling concurrent execution, significantly reducing computation time, and improving efficiency, especially in workflows involving iterative algorithms, simulations, or large-scale transformations. By leveraging multicore and manycore architectures, R users can scale their analyses and extract insights from big data more effectively. Consequently, parallelism has become a vital enabler in extending its high-performance data analytics capabilities. Despite its popularity, R was originally designed as a single-threaded, memory-bound, interpreted language, which limits its scalability on HPC and manycore platforms ( Stojanovic et al., 2016 ) . Although performance-critical components are often implemented in compiled languages such as C, C++, or Fortran, R still faces challenges, including limited native support for parallelism, high memory overhead from copy-on-modify semantics, and weak integration with distributed memory systems ( Allaire et al., 2016 ) . These limitations restrict its ability to fully exploit modern HPC infrastructures without substantial low-level optimization or external parallel frameworks. Runtime systems can enable parallel processing by managing task execution, handling data dependencies, and efficiently utilizing hardware resources, particularly for task-based parallel algorithms. They abstract the complexity of thread management, task scheduling, and synchronization, allowing programmers to focus on the logic of their applications without dealing with low-level parallelism details. Some examples include OpenMP ( Chandra, 2001 ) , StarPU ( Augonnet et al., 2009 ) , PaRSEC ( Hoque et al., 2017 ) , and Kokkos ( Edwards et al., 2014 ) . COMPSs (COMP Superscalar) ( Badia et al., 2015 ; COMP Superscalar, 2025 ) is one of these advanced task-based runtime systems that automatically builds a task dependency graph at runtime and schedules tasks asynchronously based on data availability ( Badia, 2015 ) . It simplifies the work of programmers by allowing them to write code in a sequential style, while the runtime system transparently manages the parallel execution. Furthermore, COMPSs is designed to scale from desktops to large HPC clusters, making it suitable for a wide range of data-intensive and scientific applications. This work presents RCOMPSs, a runtime system that extends the capabilities of the R language by enabling efficient task-based parallel execution on multicore and manycore systems. Built on top of the COMPSs framework, RCOMPSs introduces a new binding for R, allowing developers to write sequential R code while transparently benefiting from parallel execution. By annotating functions as tasks, users can offload execution management to the underlying runtime, which dynamically constructs a task dependency graph, schedules tasks across available resources, and handles data movement and synchronization. RCOMPSs also supports fault tolerance by leveraging the built-in mechanisms of the COMPSs runtime, including automatic task resubmission and exception management. A key aspect of RCOMPSs is its infrastructure-unaware programming model. It allows users to develop applications once and run them seamlessly on different backends, whether shared-memory or distributed-memory systems, without modifying the original implementation. RCOMPSs simplifies parallel programming in R without requiring expertise in low-level concurrency or distributed computing. In this work, we demonstrate the effectiveness of RCOMPSs through three representative applications, namely, K-means clustering, K-nearest neighbors classification (KNN), and linear regression, and evaluate their performance on shared-memory and distributed-memory systems. Our results show that RCOMPSs enables scalable and efficient execution, significantly reducing runtime compared to sequential R code, and validating its potential as a high-level interface for parallel R programming on modern HPC platforms. The structure of this article is as follows. Section 2 provides an overview of the current state-of-the-art capability of the R language for exploiting multicore and manycore CPU systems, focusing on current challenges and existing solutions. Then, Section 3 describes RCOMPSs, elaborating on the architecture, programming model, and functionalities. Next, Section 4 overviews the three numerical algorithms developed in this work, while Section 5 offers an assessment of the performance and behavior of RCOMPSs on two supercomputers with them. Finally, Section 6 , provides the main conclusions on the work and guidelines for future work. 2. Related Work Parallel computing in R has advanced considerably by developing a diverse ecosystem of packages and frameworks that support both shared-memory and distributed-memory execution. At the low level, interfaces to OpenMP and Intel Thread Building Blocks (TBB) are exposed via packages such as RcppParallel ( Allaire et al., 2016 ) , enabling fine-grained control and high-performance parallelism within the compiled code. At a higher level of abstraction, R offers user-friendly tools such as parallel ( Team et al., 2012 ) , foreach ( Analytics and Weston, 2015 ) , and snow ( Tierney et al., 2007 ) that simplify the expression of parallel loops and independent task execution for multicore systems. These packages provide essential primitives for parallel execution but require users to explicitly manage parallel constructs, data partitioning, and synchronization, limiting scalability and ease of use for complex workflows or distributed environments. More recently, the future package ( Bengtsson, 2020 ) has introduced a unified and flexible framework for asynchronous and parallel evaluation across local

Read the complete text