SuperSONIC: Cloud-Native Infrastructure for ML Inferencing Conference: Practice and Experience in Advanced Research Computing; July 20–24, 2025; Columbus, OH, USA Practice and Experience in Advanced Research Computing (PEARC ’25), July 20–24, 2025, Columbus, OH, USA DOI: 10.1145/3708035.3736049 ISBN

Dmitry Kondratyev, Benedikt Riedel, Yuan-Tang Chou, Miles Cochran-Branson, Noah Paladino, David Schultz, Mia Liu, Javier Duarte, Philip Harris, Shih-Chieh Hsu · 2025

Read SuperSONIC: Cloud-Native Infrastructure for ML Inferencing Conference: Practice and Experience in Advanced Research Computing; July 20–24, 2025; Columbus, OH, USA Practice and Experience in Advanced Research Computing (PEARC ’25), July 20–24, 2025, Columbus, OH, USA DOI: 10.1145/3708035.3736049 ISBN by Dmitry Kondratyev, Benedikt Riedel, Yuan-Tang Chou, Miles Cochran-Branson, Noah Paladino, David Schultz, Mia Liu, Javier Duarte, Philip Harris, Shih-Chieh Hsu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20657.txt.

Introduction

Big physics experiments now run machine learning on rivers of data, and the computing has to live somewhere elastic. Dmitry Kondratyev and nine co-authors describe SuperSONIC in 2025, a cloud-native server system that runs inference for several flagship experiments at once.

The design hosts models on Kubernetes clusters with graphics processors, served through the Triton inference server, so experiment workflows stay separate from the machines. Load balancing, monitoring, and standard messaging keep throughput high. The system already serves the CMS and ATLAS collider experiments, the IceCube neutrino observatory, and the LIGO gravitational-wave detectors, with test clusters at three United States sites. It grows out of an earlier approach that offloaded inference to nearby or remote coprocessors.

Catalog record

CREATORDmitry Kondratyev, Benedikt Riedel, Yuan-Tang Chou, Miles Cochran-Branson, Noah Paladino, David Schultz, Mia Liu, Javier Duarte, Philip Harris, Shih-Chieh Hsu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-20657

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SuperSONIC: Cloud-Native Infrastructure for ML Inferencing Conference: Practice and Experience in Advanced Research Computing; July 20–24, 2025; Columbus, OH, USA Practice and Experience in Advanced Research Computing (PEARC ’25), July 20–24, 2025, Columbus, OH, USA DOI: 10.1145/3708035.3736049 ISBN: 979-8-4007-1398-9/2025/07 CCS: Computer systems organization Heterogeneous (hybrid) systems CCS: Computer systems organization Client-server architectures CCS: Computing methodologies Machine learning CCS: Applied computing Physical sciences and engineering Dmitry Kondratyev, Benedikt Riedel, Yuan-Tang Chou, Miles Cochran-Branson, Noah Paladino, David Schultz, Mia Liu, Javier Duarte, Philip Harris, Shih-Chieh Hsu Abstract . The increasing computational demand from growing data rates and complex ML ( ML ) algorithms in large-scale scientific experiments has driven the adoption of the SONIC ( SONIC ) approach. SONIC accelerates ML inference by offloading it to local or remote coprocessors to optimize resource utilization. Leveraging its portability to different types of coprocessors, SONIC enhances data processing and model deployment efficiency for cutting-edge research in HEP ( HEP ) and MMA ( MMA ). We developed the SuperSONIC project, a scalable server infrastructure for SONIC , enabling the deployment of computationally intensive tasks to Kubernetes clusters equipped with GPU . Using NVIDIA Triton Inference Server, SuperSONIC decouples client workflows from server infrastructure, standardizing communication, optimizing throughput, load balancing, and monitoring. SuperSONIC has been successfully deployed for the CMS and ATLAS experiments at the CERN LHC ( LHC ), IceCube ( IceCube ), and LIGO ( LIGO ) and tested on Kubernetes clusters at Purdue University, the NRP ( NRP ), and the University of Chicago. SuperSONIC addresses the challenges of the Cloud-native era by providing a reusable, configurable framework that enhances the efficiency of accelerator-based inference deployment across diverse scientific domains and industries. 1. Introduction The impact of AI ( AI ) and ML on society in general and science in particular has been likened to the introduction of the steam engine or the internet by altering how we interact with machines and “do” science. Within a few years AI , and in particular ML , has revolutionized and accelerated scientific discovery across multiple fields to such a magnitude that the 2024 Nobel Prizes in Physics and Chemistry were awarded for fundamental work on AI (Physics) and new discoveries made using AI (Chemistry) ( Nobel Prize Committee 2024 ) . AI / ML requires a heterogeneous computing paradigm, using traditional (CPU) and specialized (generally GPU ) hardware for both training and evaluating (“inferencing”) the model. AI training has caught the headlines requiring ever larger heterogeneous clusters growing to over 100,000 GPU ( Ben Wodecki 2024 ) with plans reaching up to 1,000,000 GPU ( Anton Shilov 2024 ) . By comparison, inferencing an AI / ML model can be done with a single lower-performance GPU. This metric, however, is misleading. A given AI / ML model generally gets inferenced so often that 90% of the lifetime and computing cost of an AI / ML model are estimated to be spent on inferencing alone ( Kim Martineau 2023 ; Guo and Bhavani 2021 ) . Large-scale and data-intensive experiments, such as IceCube , the LHC ’s CMS ( CMS ) and ATLAS ( ATLAS ), and LIGO experiments, all of which are represented here, are adding and increasingly relying on AI / ML throughout the experiment ranging from data acquisition to data processing and final analysis ( Duarte 2024 ) . This trend is expected to continue as these experiments expand their science reach with AI -enhanced data analyses, detector upgrades, e.g. HL-LHC ( HL-LHC ) ( CERN 2025 ) and IceCube Upgrade ( Ishihara 2019 ) , and new experiments coming online, e.g. KAGRA ( KAGRA ) ( Akutsu et al. 2020 ) . Given the limited computational and human resources available to scientists, the growing use of AI in experiments, and the potentially high cost of AI inferencing, optimizing AI / ML deployment across the national cyberinfrastructure is crucial. The science community is taking several different approaches depending on the needs and requirements of the respective science driver. For high-throughput environments, such as the L1T ( L1T ) for the LHC experiments ( Duarte 2024 ) , the focus has been on integrating FPGA for AI inference to increase overall throughput compared to GPU . In less demanding environments, such as higher-level data processing, the development has not been as focused. Most scientists still co-locate heterogeneous compute resources for their AI / ML inference needs. However, co-location of heterogeneous resources limits the ability for the system to adapt to changes in the software stack that would require a different “mix” of resources (e.g., CPU/ GPU ), leading to significant inefficiencies in resource utilization. To address this issue, SONIC ( Duarte et al. 2019 ; Krupa et al. 2021 ; Rankin et al. 2020 ; Wang et al. 2021 ; Hayrapetyan et al. 2024 ; Zhao et al. 2025 ) were developed based on the “as-a-service” model in which the AI / ML compute is offloaded to remote accelerated resources. The idea of SONIC originated from the private sector, where the local hardware available is typically limited, e.g., smartphones, or insufficient to run large-scale models, e.g., ChatGPT ( OpenAI 2024 ) . As the SONIC approach was being implemented in ATLAS , CMS , IceCube , and LIGO experiments, it became apparent that, despite the differences in client workflows, the server infrastructure requirements are largely the same. Existing industry-grade inference platforms such as KServe ( The KServe Authors 2015 ) , vLLM ( The vLLM Authors 2015 ) , and NVIDIA Triton ( NVIDIA Corporation 2016 ) could not be used “off the shelf”, as their functionality did not meet all the needs of scientific workflows. Faced with the same challenge, the aforementioned experiments have joined forces under the NSF ( NSF ) funded HDR ( HDR ) A3D3 ( A3D3 ) Institute to develop SuperSONIC ( Kondratyev et al. 2025 ) , a common cloud-native inference framework designed to deploy ML -inference-as-a-service on national cyberinfrastructure. 2. Design SuperSONIC is designed as a scalable and portable system capable of managing inference workloads across diverse environments, from tiny GitHub Actions workers to small local clusters to large distributed HPC ( HPC ) centers. Its modular architecture relies on established industry-standard tools, with a preference given to open-source projects with guaranteed long-term maintainability and community support, such as the “graduated” projects of the CNCF ( CNCF ) ( Cloud Native Computing Foundation 2015 ) . SuperSONIC leverages Kubernetes ( The Kubernetes Authors 2014 ) to deploy its components as microservices, ensuring seamless workload orchestration and fault tolerance while abstracting infrastructure complexities into a simple, declarative configuration. To streamline installation and version control, it is distributed as a Helm chart ( The Helm Authors 2016 ) . Figure 1 illustrates the core components of the SuperSONIC architecture, which are described below in more detail. Figure 1. SuperSONIC architecture: client nodes (left) running specialized experimental software outsource the heterogeneous, often AI/ML, component of the software stack to a SuperSONIC server deployed on a Kubernetes cluster. The server handles dynamic GPU provisioning, optimized load balancing, and other operational complexities, and exposes only a single gRPC endpoint for inference requests. 2.1. Inference Servers SuperSONIC is built around NVIDIA Triton

Read the complete text