BiJuTy: An Interactive HPC-Aware Big Data Cluster Lifecycle Manager and Performance Assessment Utility for JupyterHub

Apurv Deepak Kulkarni, Jan Frenzel, Siavash Ghiasvand · 2026

Read BiJuTy: An Interactive HPC-Aware Big Data Cluster Lifecycle Manager and Performance Assessment Utility for JupyterHub by Apurv Deepak Kulkarni, Jan Frenzel, Siavash Ghiasvand (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24412.txt.

Introduction

Kulkarni, Frenzel, and Ghiasvand bring big data clusters on supercomputers inside the notebook interface scientists already live in. High-performance computing systems could crush big data workflows. Standing up a cluster on one scares off beginners, though, and splits the workflow across tools. BiJuTy, their 2026 utility, is an interactive, HPC-aware lifecycle manager and performance check tool. It lives inside JupyterHub.

The interface walks the user from cluster setup through an initial performance check. Multiple clusters can be managed directly within the Jupyter notebook environment, without switching tools. Performance metrics gathered from several sources feed a simple tuning workflow. A worked example shows BiJuTy deployed to tune a real big data processing pipeline. Kulkarni's team targets the access gap explicitly. The same interface should serve first-time users and experienced hands. Supercomputer-scale data processing becomes a guided procedure, rather than a systems rite of passage.

Catalog record

CREATORApurv Deepak Kulkarni, Jan Frenzel, Siavash Ghiasvand
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24412

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BiJuTy: An Interactive HPC-Aware Big Data Cluster Lifecycle Manager and Performance Assessment Utility for JupyterHub Apurv Deepak Kulkarni, Jan Frenzel, Siavash Ghiasvand Abstract The increasing demand for data processing has created a pressing need for access to high-performance computing (HPC) systems. Nevertheless, leveraging these systems to execute complex big data processing workflows remains a significant challenge, especially for beginners. This work presents BiJuTy, a solution designed to bridge the accessibility gap for big data workflows on HPC systems within the Jupyter ecosystem. By providing an interactive and user-friendly interface, BiJuTy simplifies cluster lifecycle management and performance assessment, making it more accessible on HPC systems to beginners and experienced users alike. The solution is presented as an interactive interface that guides the user through the entire process, from setting up the cluster configuration to carrying out initial performance assessments. Additionally, the framework enables seamless management of multiple clusters directly within the Jupyter Notebook interface, eliminating the need to switch outside of working environment. The collection of performance metrics from various sources further simplifies the optimization workflow. Furthermore, an illustrative example is provided to demonstrate how BiJuTy can be deployed to optimize the performance of a big data processing application. This example showcases how the entire big data processing lifecycle can be iteratively executed and optimized in just a few clicks, helping to reach the goal of optimization easily and interactively. By facilitating such workflows, this work contributes in bringing the field of big data computing and high-performance computing one step closer to the goal of seamless interaction and usability. 1 Introduction The advancements in the field of machine learning and artificial intelligence (AI) increasingly motivate interdisciplinary researchers to use sophisticated analysis models. This usage necessitates the access to larger computational resources such as high performance computing (HPC) clusters. However, despite the wide-range of efforts and rapid advancements in democratizing HPC resources, there remains a significant gap which stems from the batch-processing nature of HPC clusters in contrast to the interactive approach of users who are coming from other fields. Currently, the most accessible computation resources are the commercial cloud providers which provide HPC resources with a minimal barrier via their own graphical user interface (GUI) [ 21 ] . However, despite the widespread availability and accessibility of cloud systems, HPC clusters remain a significant computation platform even for non-HPC workloads, due to reasons beyond user convenience. HPC clusters are purpose-built and have optimized architecture to provide extreme performance on specialized hardware within standardized environments while maintaining consistency across multiple runs. Additionally, increasing demands for data privacy and control motivate many institutions to maintain their own on-premise HPC clusters [ 1 , 24 ] . These systems mainly run on open-source software and are tightly-coupled into the research environment over decades. JupyterHub [ 28 ] is one of the successful software infrastructures that bridges the gap and provides interactive access to HPC resources, particularly for users without prior experience, and is being actively used on both cloud and HPC clusters. Its GUI requires only a modern web browser and serves as a single point of entry, while hiding all configurations from the user, which plays a significant role in the popularity of JupyterHub among the vast majority of users. Currently, many HPC centers provide JupyterHub as an alternative to the classical command line interface (CLI)-based access to their HPC clusters. Although this interface cannot fully replace the CLI-based approach, since it does not align with the batch-processing nature of HPC systems, it provides an easy access to computational resources and assists the new users to get onboard and familiarize themselves with the HPC environment with minimal efforts. Among the advanced analytical approaches, big data frameworks (BDF) [ 34 ] are not exceptions. BDFs are software environments and toolsets that are designed to efficiently ingest, store, process, and analyze massive volumes of data. These frameworks have a wide range of usability in various interdisciplinary domains and often they need HPC resources to properly handle the vast amount of data. However, setting up, managing, and monitoring BDF on HPC systems is itself a challenge for beginners due to a high level of complexity imposing a significant barrier on the path towards effective use of BDF on HPC systems. Beginners would significantly benefit from graphical representations of the configuration, guidance by predefined UI actions and simplified access to performance metrics. Thus, this work proposes BiJuTy 1 1 1 https://github.com/scads/bijuty a comprehensive infrastructure as code (IaC) toolbox that integrates into Jupyter ecosystem and extends its capabilities to (1) manage individual users’ configurations for various big data frameworks through a user-friendly interface, (2) control the lifecycle of standalone big data clusters (BDC), (3) visualize Spark/Flink job information, (4) visualize framework and system-level metrics along with external metric systems, and (5) manage configuration and workflow on a per-user and per-framework basis. BiJuTy streamlines interaction with big data frameworks on HPC systems, making it accessible to novice users and efficient for experts. It automates the set up, monitor, and tear down of BDCs. Users will obtain performance-metric data with a single click, facilitating rapid debugging and deeper insight into underlying operations. It enables users to manage configurations on an individual and per-framework basis. All of these are provided directly from within the familiar JupyterHub ecosystem. 2 Literature Review Several efforts focus on simplifying the access to HPC resources as a vital prerequisite for executing complex analysis pipelines which often need to handle large amount of data [ 8 , 32 , 20 , 39 ] . Since this work is focusing on BDF, once an intuitive access to HPC resources is established e.g., via JupyterHub, the next step would be deployment of BDCs on the underlying HPC environment. For deployment of BDCs on HPC clusters, many CLI tools are available for various HPC scheduling systems and either generic support for various BDC [ 14 , 13 , 19 ] or tailored solutions for Apache Spark-based clusters [ 22 , 10 , 31 , 7 ] . These tools reduce repetitive manual steps and streamline cluster setup for experienced practitioners. However, this work focuses on automated mechanisms that minimize user intervention, thereby enabling researchers to concentrate on their core scientific objectives while abstracting the complexities of BDC management through systematic automations. Various configuration management tools such as Chef [ 25 ] , Puppet [ 23 ] , Ansible [ 30 ] , and Salt [ 38 ] exist which can assist experienced users in installing and managing BDCs. However these tools are primarily designed for cloud environments with persistent infrastructure and full administrative access, rather than for HPC centers with resource scheduling and job-based execution environments. Consequently, they do not assist in provisioning BDCs within such environments and are targeted at experienced users. Other automation tools such as Pulumi [ 29 ] address the provisioning of underlying infrastructure, however most HPC clusters particularly in research institutes mainly rely on Slurm [ 33 ] or PBS [ 2 ] as job scheduler which therefore requires a hybrid IaC approach. Tools such as Open-OnDemand [ 6 ] on the other hand have their specific hardware and software requirements and need deep integration into the software stack of the HPC clusters, which is

Read the complete text