OnePiece: A Large-Scale Distributed Inference System with RDMA for Complex AI-Generated Content (AIGC) Workflows

June Chen, Neal Xu, Gragas Huang, Bok Zhou, Stephen Liu · 2026

Read OnePiece: A Large-Scale Distributed Inference System with RDMA for Complex AI-Generated Content (AIGC) Workflows by June Chen, Neal Xu, Gragas Huang, Bok Zhou, Stephen Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20655.txt.

Introduction

June Chen and four colleagues present OnePiece, a large distributed inference system with RDMA networking for AI-generated content workflows. AI generation now produces high-quality creative work across many domains. Yet existing systems waste throughput, resources, and scalability when many multi-stage jobs run at once.

OnePiece is optimized for those staged workflows, where the output of one model feeds the next. RDMA networking moves data between machines with low delay. Faster shared infrastructure would lower the cost of image, video, and media generation services as demand grows. The aim is more work done on the same racks of chips, with less time lost in line.

Catalog record

CREATORJune Chen, Neal Xu, Gragas Huang, Bok Zhou, Stephen Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20655

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OnePiece: A Large-Scale Distributed Inference System with RDMA for Complex AI-Generated Content (AIGC) Workflows June Chen, Neal Xu, Gragas Huang, Bok Zhou, Stephen Liu Abstract . The rapid growth of AI-generated content (AIGC) has enabled high-quality creative production across diverse domains, yet existing systems face critical inefficiencies in throughput, resource utilization, and scalability under concurrent workloads. This paper introduces OnePiece, a large-scale distributed inference system with RDMA optimized for multi-stage AIGC workflows. By decomposing pipelines into fine-grained microservices and leveraging one-sided RDMA communication, OnePiece significantly reduces inter-node latency and CPU overhead while improving GPU utilization. The system incorporates a novel double-ring buffer design to resolve deadlocks in RDMA-aware memory access without CPU involvement. Additionally, a dynamic Node Manager allocates resources elastically across workflow stages in response to real-time load. Experimental results demonstrate that OnePiece reduces GPU resource consumption by 16 × 16\times in Wan2.1 image-to-video generation compared to monolithic inference pipelines, offering a scalable, fault-tolerant, and efficient solution for production AIGC environments. 1. Introduction The advent of AI-generated content ( AIGC ) has dramatically lowered barriers to creative expression and content production, empowering individuals and organizations to produce high-quality materials with unprecedented speed and scale. AIGC is being applied across diverse fields such as automated journalism, personalized advertising, video game design, and virtual reality content creation. Central to this progress are diffusion models, whose remarkable generative capabilities are increasingly surpassing those of GANs and auto-regressive Transformers. These models excel not only in image generation but are also driving innovation in video-related research and production. Video diffusion models support a wide range of applications—including video synthesis, editing, and enhanced video understanding. By providing non-experts with advanced generative tools, these technologies are helping democratize content creation. As a result, AIGC is fundamentally transforming traditional workflows within creative industries. AIGC workflows typically integrate a variety of heterogeneous models, such as Variational Autoencoders (VAEs) ( 9 ; 11 ) , diffusion models ( 8 ; 53 ; 55 ) , and text encoders ( 34 ) ,each fulfilling a distinct role within the content generation pipeline. In practice, specialized models like LoRA ( 16 ; 10 ) are often incorporated to further enhance output quality, leading to increasingly complex and multi-stage processing architectures. This multi-model design, however, often introduces significant latency and high GPU resource demands. For example, the WAN2.1 model ( 49 ) requires around 32GB of GPU memory distributed across 8 GPUs. Despite significant advances in reducing per-task latency, a critical challenge persists in real-world AIGC deployments: a disproportionate emphasis on optimization targets. Current research and industrial practices remain largely centered on minimizing latency at the request level, frequently at the expense of system-wide throughput and GPU utilization efficiency. This narrow focus leads to suboptimal GPU occupancy, constrained service capacity, and poor scalability under highly concurrent conditions. The identified inefficiencies arise from several interrelated factors: • First, the multi-stage architecture of AIGC pipelines introduces diverse computational and memory requirements that differ substantially across stages. Conventional monolithic system architectures often lack the granularity needed for fine-grained resource allocation tailored to these varying demands. • Second, the dynamic and often unpredictable nature of request patterns leads to significant load fluctuations, which undermines the effectiveness of static resource provisioning strategies. • Lastly, inefficient memory management exacerbates these challenges, as intermediate results and cached data consume excessive VRAM without corresponding gains in operational efficiency. Microservices-based architectures offer a promising solution to these challenges by enabling more elastic and efficient resource utilization. For instance, Ant Group’s deployment of the NVIDIA Triton Inference Server for managing multi-model pipelines achieved a 2.4× improvement in throughput, a 20% reduction in latency, and a 50% decrease in operational costs ( 43 ) . The principal benefits of a microservices architecture include: • Elastic Resource Allocation: Microservices can be deployed on hardware tailored to the specific computational requirements of each component. For example, compute-intensive tasks such as text encoding can utilize high-frequency GPUs, while memory-intensive operations like frame synthesis benefit from GPUs with larger VRAM capacities. • Independent Scalability: Each service can be scaled independently according to real-time workload fluctuations. This fine-grained scalability allows precise resource allocation, reduces the need for over-provisioning, and lowers operational costs compared to monolithic systems. • Fault Isolation and Enhanced Stability: The decoupled nature of microservices localizes failures within individual components, preventing cascading system failures and improving overall reliability. • Technological Heterogeneity: Each microservice can employ specialized frameworks, programming languages, or optimizations best suited to its task, promoting flexibility and performance gains. However, disaggregating pipelines into discrete stages often introduces significant latency overheads due to the substantial volume of data transferred between nodes. Some existing systems that disaggregate transformer models,such as Mooncake ( 38 ) , leverage NCCL ( 35 ) for message passing to mitigate this latency. Nevertheless, NCCL is primarily suitable for tensor-based messages with fixed payload sizes and cannot efficiently accommodate data with dynamic or variable dimensions. In this paper, we present OnePiece, a novel, large-scale distributed inference system designed for microservices-oriented deployment using RDMA and tailored to complex AIGC workflows. By strategically decomposing end-to-end pipelines into fine-grained microservices, OnePiece enables more flexible resource allocation and significantly improves GPU utilization. Additionally, we propose a systematic methodology for analyzing and identifying optimal pipeline partitioning strategies, thereby enhancing overall system performance and resource efficiency. OnePiece improves resource efficiency by sharing common pipeline components across multiple applications, such as text-to-video (T2V) and image-to-video (I2V) generation will go though the same VAE Encoder and Decoder services. To handle dynamic and unpredictable request loads, OnePiece integrates a Node Manager ( NM ) that dynamically redistributes GPU resources across microservices. This enables scalable capacity expansion during peak demand and contraction during low-traffic periods, ensuring high overall resource utilization. To address the high data transfer latency associated with traditional TCP-based sockets in large-volume data scenarios, OnePiece adopts one-sided RDMA for inter-service communication ( 19 ; 3 ; 46 ) . This approach supports direct memory-to-memory data transfer, substantially decreasing CPU overhead. However, due to the regional constraints inherent in RDMA connections, OnePiece organizes services into regionally autonomous sets, each of which can execute complete workflows independently. Incoming requests are distributed randomly across these sets. Beyond improving resource utilization, this multi-set design also increases fault tolerance by isolating failures within specific regional sets. As multiple instances deliver messages to a shared memory region within the same receiving instance via RDMA, OnePiece employs a ring buffer structure that supports dynamically sized messages. However, deadlocks may occur if an instance fails during memory access. To the best of our knowledge, existing deadlock resolution mechanisms—such as those in Redis ( 6 ) ,rely on CPU involvement. To address this under RDMA constraints, OnePiece introduces a novel double-ring buffer mechanism that resolves deadlocks without CPU intervention. Our system delivers high throughput and low latency while maintaining high-quality content generation, providing a scalable, reliable, and cost-effective platform for real-world AIGC services. In summary, this paper makes the following contributions: • We present the detailed design of OnePiece, a large-scale distributed system for orchestrating microservices-based AIGC pipelines. • We introduce a one-sided RDMA network architecture to enable high-performance inter-service communication to reduce the latency from transferring high volumn data. • We propose a deadlock-free multi-producer ring buffer data

Read the complete text