The National Research Platform: Stretched, Multi-Tenant, Scientific Kubernetes Cluster Conference: Practice and Experience in Advanced Research Computing; July 20–24, 2025; Columbus, OH, USA Practice and Experience in Advanced Research Computing (PEARC ’25), July 20–24, 2025, Columbus, OH, USA DOI:

Derek Weitzel, Ashton Graves, Sam Albin, Huijun Zhu, Frank Würthwein, Mahidhar Tatineni, Dmitry Mishin, John Graham, Elham E Khoda, Mohammad Firas Sada, Larry Smarr, Thomas DeFanti · 2025

Read The National Research Platform: Stretched, Multi-Tenant, Scientific Kubernetes Cluster Conference: Practice and Experience in Advanced Research Computing; July 20–24, 2025; Columbus, OH, USA Practice and Experience in Advanced Research Computing (PEARC ’25), July 20–24, 2025, Columbus, OH, USA DOI: by Derek Weitzel, Ashton Graves, Sam Albin, Huijun Zhu, Frank Würthwein, Mahidhar Tatineni, Dmitry Mishin, John Graham, Elham E Khoda, Mohammad Firas Sada, Larry Smarr, Thomas DeFanti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22864.txt.

Introduction

Weitzel and a large team described the National Research Platform in twenty twenty-five. The platform is a shared computing system for science, built on Kubernetes, a tool that runs software across many machines. It spans over seventy-five locations in the United States and abroad, run by many tenants at once.

Sites contribute anything from single nodes to large clusters of graphics and central processors. Users can work through friendly interfaces such as JupyterHub or control resources directly at a low level. The paper covers daily operations, including threat detection built into the system, wide monitoring, and detailed accounting of use. The authors present the platform as a scalable answer to rising demand for shared research computing.

Catalog record

CREATORDerek Weitzel, Ashton Graves, Sam Albin, Huijun Zhu, Frank Würthwein, Mahidhar Tatineni, Dmitry Mishin, John Graham, Elham E Khoda, Mohammad Firas Sada, Larry Smarr, Thomas DeFanti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22864

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The National Research Platform: Stretched, Multi-Tenant, Scientific Kubernetes Cluster Conference: Practice and Experience in Advanced Research Computing; July 20–24, 2025; Columbus, OH, USA Practice and Experience in Advanced Research Computing (PEARC ’25), July 20–24, 2025, Columbus, OH, USA DOI: 10.1145/3708035.3736060 ISBN: 979-8-4007-1398-9/2025/07 CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Machine learning CCS: Computer systems organization Dependable and fault-tolerant systems and networks Derek Weitzel, Ashton Graves, Sam Albin, Huijun Zhu, Frank Würthwein, Mahidhar Tatineni, Dmitry Mishin, John Graham, Elham E Khoda, Mohammad Firas Sada, Larry Smarr, Thomas DeFanti Abstract . The National Research Platform (NRP) represents a distributed, multi-tenant Kubernetes-based cyberinfrastructure designed to facilitate collaborative scientific computing. Spanning over 75 locations in the U.S. and internationally, the NRP uniquely integrates varied computational resources, ranging from single nodes to extensive GPU and CPU clusters, to support diverse research workloads including advanced AI and machine learning tasks. It emphasizes flexibility through user-friendly interfaces such as JupyterHub and low level control of resources through direct Kubernetes interaction. Critical operational insights are discussed, including security enhancements using Kubernetes-integrated threat detection, extensive monitoring, and comprehensive accounting systems. This paper highlights the NRP’s growing importance and scalability in addressing the increasing demands for distributed scientific computational resources. Figure 1. Map of NRP sites throughout the US. Not pictured are the several international locations. Map of NRP sites throughout the US. Not pictured are the several international locations. 1. Introduction The NSF supported research computing community has established infrastructures tailored to different levels of contributions. ACCESS ( Boerner et al. 2023 ) provides an umbrella organization primarily for large resources, each of which are located in a single data center and run their own batch schedulers. The OSG ( Pordes et al. 2007 ) federates clusters in different locations worldwide that each have their own batch scheduler. The OSG then overlays a fabric of services on top to federate the individual clusters to enable distributed High Throughput Computing ( Raman et al. 2000 ) . The National Research Platform (NRP) federates individual compute and storage nodes into a global scale resource cluster across an a priori unlimited number of locations. The NRP thus operates at levels in the vertical software stack below either OSG or ACCESS. The entirety of the NRP, encompassing all globally distributed hardware locations, currently presents as a single ‘batch cluster’ to users. The NRP has attracted a growing number of users. In calendar year 2024, there were 670 NRP Nautilus Namespaces that used 5M GPU-hours and 50M CPU Core-Hours on the NRP. These Namespaces can be divided into 4 main categories: Community Software (2), Observatories (13), NRP Operations (130), Individual campus researchers (525). These latter 525 Namespaces consumed >3/4 of the GPU-Hrs and 2/3 of the CPU-hrs used by all 670 NRP Nautilus Namespaces active in 2024. The individual PIs of these 525 researcher Namespaces came from 50 campuses, including 23 Minority Serving Institutions. These campuses include ones located in a dozen NSF designated Established Program to Stimulate Competitive Research (EPSCoR) ( NSF 2025b ) States. The campuses span from Carnegie R1/R2 campuses to community colleges. This broad national breadth of campuses are critical to the future development of a US trained AI/ML workforce. The NRP can federate resources ranging from a single node to large collections of resources located in the same data center, enabling groups to contribute hardware according to their capacity. NRP offers integration at either the IPMI or the Kubernetes ( Kubernetes 2025 ) layer, thus releasing the resource owners of the burden of system and cluster administration, in addition to offering higher level services. The National Research Platform (NRP) originated from the Pacific Research Platform ( Smarr et al. 2018 ) , which began as a testbed for evaluating network connectivity across multiple sites. It has since evolved into a specialized platform focused on GPU hosting for Artificial Intelligence (AI) workloads. Additionally, the NRP provides educational support through fully-managed Jupyter notebook environments, LLM as a service, training resources, and curated containers. It also fosters community formation through a chat platform implemented via Matrix support channels. Each institution contributes hardware to the NRP for its own reasons. The reduced burden of system administration attracts both institutions with limited resources as well as institutions for which research computing is not a primary focus (e.g. regional networking organizations and Internet2). Other institutions are attracted by the higher level services and the community building aspects of NRP. For example, institutions like the University of Missouri participate because their researchers already utilize the NRP and wish to expand their usage, including experimenting with new and/or unique hardware configurations ( Hurt et al. 2024 ) . For these researchers, the capability to run workloads on their dedicated nodes while seamlessly extending to additional research computing resources significantly amplifies their scientific impact. In addition, a number of projects use NRP as a distributed service deployment platform to build higher level services on top of NRP. Some sites contribute hardware to meet funding requirements, such as the 20% resource-sharing mandate in recent Campus Cyberinfrastructure (CC*) ( NSF 2025a ) solicitations. In these cases, the contributed hardware is shared among NRP users, and in return, the NRP user-support team actively assists these sites by training their local users. Additionally, the NRP systems team supports the site’s local management of the hardware and ensures compliance with any required institutional policies, monitoring security, and keeping software at all systems levels up-to-date. Increasingly, we see the community building aspects of NRP to lead to one institution hosting equipment bought by another institution, thereby contributing to the NRP community owned cluster by providing rackspace, power, and networking. The continued growth in the NRP user community lead to the implementation of a richer set of policies. This includes a resource allocation mechanism for the most heavily in demand GPUs, an Acceptable Use Policy, and enhancements to security, monitoring, and their automation, as detailed in Section 4 . Overall, the NRP continues to grow in both users and resources, while providing innovative offerings such as hosted LLMs 1 1 1 Including those that provided assistance in editing this paper! and other shared services. The remainder of this paper describes the operations of the NRP, a distributed, multi-tenant Kubernetes cluster designed for scientific computing, and shares our experiences in supporting users to effectively leverage the platform. 2. Distributed Deployment The Network Research Platform consists of over 420 nodes distributed across more than 70 sites as shown in Figure 1 . These sites include R1 institutions such as the University of California, San Diego, R2 institutions like the University of South Dakota, and network points-of-presence (PoPs) that provide high-quality network connectivity. Managing and deploying the NRP management software across multiple administrative domains poses significant challenges. To overcome these hurdles, we utilize Ansible ( HAt 2025 ) , a powerful automation platform, to streamline the deployment and maintenance of our services. Specifically, Ansible simplifies the installation process of new nodes by automatically configuring and installing necessary software, including: • Containerization : The NRP installs the container runtime which will run the user pods. • Kubernetes : The orchestration layer that the NRP uses to schedule workflows. • System settings : The NRP sets numerous settings on

Read the complete text