HeteroPod: XPU-Accelerated Infrastructure Offloading for Commodity Cloud-Native Applications
Bicheng Yang, Jingkai He, Dong Du ✉, Yubin Xia, Haibo Chen · 2025
Read HeteroPod: XPU-Accelerated Infrastructure Offloading for Commodity Cloud-Native Applications by Bicheng Yang, Jingkai He, Dong Du ✉, Yubin Xia, Haibo Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23952.txt.
Introduction
Bicheng Yang and co-authors present HeteroPod, a way to move cloud support services off the main computer and onto a helper chip. In cloud systems, services such as network meshes and watch agents fight user programs for the same resources. Moving them to a data processing unit gives strict separation and cuts cost.
A new network layer lets programs split across the two chips still talk as one, at near shared-memory speed. The full system was built on Linux and a standard cloud platform. Tests with two kinds of helper chips ran large, unmodified cloud programs well. Delay fell sharply against rival designs, and resource use dropped by a wide margin.
Catalog record
| CREATOR | Bicheng Yang, Jingkai He, Dong Du ✉, Yubin Xia, Haibo Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23952
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HeteroPod: XPU-Accelerated Infrastructure Offloading for Commodity Cloud-Native Applications Thanks: ✉ Corresponding author: Dong Du ( [email protected] ). Bicheng Yang, Jingkai He, Dong Du ✉, Yubin Xia, Haibo Chen Abstract . Cloud-native systems increasingly rely on infrastructure services (e.g., service meshes, monitoring agents), which compete for resources with user applications, degrading performance and scalability. We propose HeteroPod, a new abstraction that offloads these services to Data Processing Units (DPUs) to enforce strict isolation while reducing host resource contention and operational costs. To realize HeteroPod, we introduce HeteroNet, a cross-PU (XPU) network system featuring: (1) split network namespace , a unified network abstraction for processes spanning CPU and DPU, and (2) elastic and efficient XPU networking , a communication mechanism achieving shared-memory performance without pinned resource overhead and polling costs. By leveraging HeteroNet and the compositional nature of cloud-native workloads, HeteroPod can optimally offload infrastructure containers to DPUs. We implement HeteroNet based on Linux, and implement a cloud-native system called HeteroK8s based on Kubernetes. We evaluate the systems using NVIDIA Bluefield-2 DPUs and CXL-based DPUs (simulated with real CXL memory devices). The results show that HeteroK8s effectively supports complex (unmodified) commodity cloud-native applications (up to 1 million LoC) and provides up to 31.9x better latency and 64x less resource consumption (compared with kernel-bypass design), 60% better end-to-end latency, and 55% higher scalability compared with SOTA systems. 1. Introduction Cloud-native computing has emerged as a prominent paradigm in cloud and datacenter environments, utilizing containerization to distribute and deploy applications ( cnc, 2023a ; AWS, 2021 ; IBM, 2021 ; MSF, 2021 ; Goo, 2021 ) . The fundamental unit of the execution environment, such as a Pod in Kubernetes 1 1 1 In this paper, we consider the Pod as a generalized representation of the fundamental computational unit in cloud-native platforms. , typically comprises one or multiple containers, each containing a standalone image but sharing a network environment. This approach encapsulates an application in a portable and self-contained way that can be deployed and distributed across diverse nodes. One significant advantage of cloud-native apps compared to non-cloud-native apps is that they are developed with the support of extensive cloud infrastructure services, including storage, network, security, and more. These infrastructures are also deployed as containers. There are two categories of infrastructure containers: global infrastructure containers and per-Pod infrastructure containers, as shown in Figure 1 -a. For instance, the platform scheduler serves as one of the global infrastructure containers responsible for orchestrating requests to different Pods, while a sidecar proxy (e.g., Envoy ( env, 2023 ) ) represents a per-Pod infrastructure container present in each Pod to facilitate the networking functionalities of the application. Figure 1 . Cloud-native platform with infrastructure containers and user apps. While infrastructure containers play a crucial role in delivering advantages in cloud-native computing, they also introduce a non-trivial burden on resources, referred to as infra-burden in the paper. For example, service mesh ( ser, 2017 ) has emerged as a prominent pattern in cloud-native platforms for managing microservices, employing a dedicated proxy (known as a sidecar-proxy ) to handle network traffic on behalf of the applications. Since each microservice Pod contains a sidecar-proxy as an infrastructure container, these proxy containers can consume up to 25% of the CPU and 33% of memory resources on a node ( Liu et al., 2023 ) . The infra-burden challenge has persisted for a considerable time ( Kim et al., 2021 ) and presents substantial difficulty in mitigation ( Huang et al., 2024 ; Szekely et al., 2024 ; Cvetković et al., 2024 ) . A recent noteworthy trend that could potentially offer relief is that cloud vendors are increasingly deploying more heterogeneous devices to enhance scalability, performance, and cost-efficiency ( ama, 2021 ; Silberstein et al., 2013 ; Kim et al., 2014 ; Firestone et al., 2018 ; chi, 2021 ; hua, 2021 ; xil, 2021 ; ali, 2021 ; bai, 2021 ) . The Data Processing Unit (DPU) represents a novel category of heterogeneous devices that deliver scalable and energy-efficient computation capabilities ( Liu et al., 2019b ) , leveraging embedded processing units and an efficient NIC. DPUs have the potential to significantly enhance instance density in a node (e.g., 50% improvement in serverless computing ( Du et al., 2022 ) ). These devices also offer enhanced security features through isolated memory and cores, along with boosted packet processing facilitated by numerous on-path hardware threads, such as the 256 hardware threads supported in Bluefield-3 ( NVIDIA Mellanox, 2023 ) . The expectation is that heterogeneous processing units like DPUs will be interconnected with rapid interconnect technologies like PCIe 6.0 and CXL ( int, 2021 ) , providing further performance enhancements. Leveraging DPUs could potentially alleviate the infra-burden challenge. Goal: Offloading cloud-native infrastructure services to DPUs. In contrast to the conventional approach of colocating infrastructure containers and user apps on both the CPU and DPU ( red, 2023 ) , we propose a promising (albeit challenging) architecture that involves decoupling these two types of cloud-native containers onto separate Processing Units (PUs), as illustrated in Fig. 1 -b. This strategy can effectively alleviate the infra-burden by preventing interference between infrastructure containers and app containers. Nevertheless, separating infrastructure containers from app containers poses significant challenges. Challenge-1: Infrastructure and app containers in the same Pod are coupled with the same network namespace, dependent on the single-PU OS. Containers within a shared Pod utilize a will network namespace provided by the underlying OS kernel. Consequently, processes in different containers share common network devices and can communicate across containers using “localhost”. However, offloading infrastructure containers to DPUs disrupts this shared network namespace functionality. Sustaining local network abstraction while separating infrastructure and app containers onto different PUs poses a notable challenge. Challenge-2: Separating containers onto different PUs has a trade-off between communication performance and resource utilization. Infrastructure and app containers will communicate through network. The separation may significantly increase communication latency (by 10x to 100x) compared to a single-PU design with Linux network stack as network packets should go through two NICs. Advanced network solutions such as kernel-bypassed mechanisms ( Jeong et al., 2014 ; rso, 2023 ; f-s, 2023 ; Li et al., 2019 ; lib, 2024 ) like rsocket ( rso, 2023 ) can provide socket-level APIs for compatibility and optimal performance to directly leverage hardware features like RDMA. However, these solutions often encounter challenges related to suboptimal resource utilization ( Zhang et al., 2021 ) , including memory wastage from pinned memory allocation for user-space links and CPU cycle wastage from user-mode polling. Kernel-assisted approaches like SMC-R ( Fox et al., 2015 ) have performance drawbacks due to context switching and data copying overheads. Consequently, a cross-PU and network-based communication method that can deliver superior performance and maximize resource utilization is essential but currently unavailable. Challenge-3: Not all infrastructure containers are suitable for offloading to DPUs. The complexity of this challenge arises from the inherent nature of DPUs, which typically comprise a wimpy SoC (only) optimized for specific domain-centric tasks, such as network packet processing. A simple design for offloading containers to DPUs may carry significant performance overhead, as suggested by prior works ( Wei et al., 2022a ; Kim et al., 2021 ; Du et