Adaptive Orchestration for Large-Scale Inference on Heterogeneous Accelerator Systems: Balancing Cost, Performance, and Resilience

Yahav Biran, Imry Kissos · 2025

Read Adaptive Orchestration for Large-Scale Inference on Heterogeneous Accelerator Systems: Balancing Cost, Performance, and Resilience by Yahav Biran, Imry Kissos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20074.txt.

Introduction

Yahav Biran and Imry Kissos propose a control system that routes AI inference requests across mixed fleets of GPUs and special accelerators. The surge in generative AI work calls for inference that scales. It must use varied hardware while it holds down running costs. Accelerator capacity swings and prices differ. So static allocation wastes costly compute power.

The paper proposes a control loop that does not rest on one hardware type. It allocates requests based on real-time cost and capacity signals. It shifts in a dynamic way between cost-based and capacity-based modes. That keeps latency low and throughput high when availability swings. Tested with the Stable Diffusion model, the frame met latency targets each time. It sent traffic to new routes on its own in capacity shortfalls. It also used lower-cost accelerators when it could.

Catalog record

CREATORYahav Biran, Imry Kissos
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-20074

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Orchestration for Large-Scale Inference on Heterogeneous Accelerator Systems: Balancing Cost, Performance, and Resilience Yahav Biran, Imry Kissos Abstract Abstract The surge in generative AI workloads has created a need for scalable inference systems that can flexibly harness both GPUs and specialized accelerators while containing operational costs. This paper proposes a hardware-agnostic control loop that adaptively allocates requests across heterogeneous accelerators based on real-time cost and capacity signals. The approach sustains low latency and high throughput by dynamically shifting between cost-optimized and capacity-optimized modes—ensuring the most efficient use of expensive compute resources under fluctuating availability. Evaluated using the Stable Diffusion model [ 1 ] , the framework consistently meets latency targets, automatically redirects traffic during capacity shortfalls, and capitalizes on lower-cost accelerators when possible. These results highlight how a feedback-driven deployment strategy, spanning the entire software and hardware stack, can help organizations efficiently scale generative AI workloads while maintaining resilience in the face of limited accelerator capacity. 1 Introduction The growing demand for generative applications in AI has driven advancements in compute accelerators, leading to a range of solutions designed to enhance performance and manage costs. Accelerators such as NVIDIA GPUs and purpose built chips like AWS Inferentia and Trainium now play a central role in enabling efficient inference at scale. However, integrating these diverse hardware architectures into existing workflows remains a challenge, requiring careful consideration of inference modes such as eager-based and graph-based, software compatibility, workload optimization, and cost-performance trade-offs. Inference, the process of using trained machine learning models to generate predictions or outputs, is distinct from training in its emphasis on dynamic, latency-sensitive operations. For generative AI applications like text-to-image models, this requirement becomes even more critical. These models demand a balance of high computational throughput and low response times, as shown in 4(a) , particularly in scenarios where user interactions are time-sensitive. At the same time, cloud-based inference must remain cost-efficient and adaptable to fluctuating availability of compute resources. Load balancer Prompt "portrait photo of a cat" Model app SD21 A10G CUDA Eager-mode Model app SD21 A10G Triton Graph-mode Model app SD21 L4 Triton Graph-mode Model app SD21 Trn1 Triton Graph-mode Model app SD21 Inf2 Neuron Graph-mode Karpenter NodeClass NodePool A10G & L4 NodeClass NodePool Trn1/Inf2 Figure 1 : The diagram illustrates a load balancer distributing traffic to five parallel model applications labeled "Model app SD21". Each application consists of compute framework and hardware accelerator components. The first two apps use CUDA (eager-mode) and Triton (graph-mode) with A10G accelerators. The third uses Triton with L4. The fourth and fifth run on Triton and Neuron frameworks, powered by Trn1 and Inf2 accelerators, respectively. Karpenter provisions resources through NodePools and NodeClasses managing NVIDIA (A10G, L4) and Neuron (Trn1, Inf2) instances. Arrows show the hierarchical relationships. This work presents a framework for scalable and hardware-agnostic inference that leverages Kubernetes-based [ 4 ] tools to dynamically allocate resources across heterogeneous compute architectures. By integrating AWS Elastic Kubernetes Service (EKS), Kubernetes Event-Driven Autoscaling (KEDA) [ 3 ] , and Karpenter [ 2 ] , this approach enables a flexible and efficient deployment of generative AI models across different accelerators, such as GPUs, AWS Inferentia, and Trainium. While this implementation highlights Kubernetes and specific open-source tools, the methods described are not limited to these technologies. The principles demonstrated can be extended to other container orchestration platforms and equivalent model invocations methods. Similarly, the framework is compatible with alternative compute hardware, including devices from vendors like AMD and Intel, beyond the specific accelerators evaluated here. The framework adopts two key deployment strategies: • Cost-Optimized Configuration: This strategy minimizes costs by prioritizing accelerators with lower inference costs, adjusting traffic distribution accordingly. • Capacity-Optimized Configuration: This approach ensures resilience by automatically switching to alternative accelerators during capacity constraints while maintaining latency and throughput requirements. These strategies are evaluated in the context of Stable Diffusion, a generative AI model for text-to-image synthesis. Performance and cost metrics are analyzed across multiple compute accelerators that are invoked in various ways such as PyTorch eager-mode with GPU, graph-mode with GPU and XLA, demonstrating the effectiveness of the framework in managing heterogeneous workloads. The primary contributions of this work are as follows: • A scalable framework for integrating and managing heterogeneous compute accelerators in inference workflows, adaptable to various container orchestration platforms and compute architectures. • A detailed evaluation of Stable Diffusion performance on various accelerators, including NVIDIA GPUs and AWS AI chips, Inferentia and Trainium. • Practical guidance for deploying and optimizing generative AI workloads in cloud environments using AWS services and Kubernetes tools. This framework offers a practical pathway for organizations to deploy generative AI applications for serving inference requests effectively, balancing the competing demands of cost, performance, and resource availability in diverse computing environments. 2 Background and Related Work Transformer Architecture & Foundation Models The emergence of the Transformer architecture has fundamentally reshaped both natural language processing and computer vision. Models such as GPT-3 and the Vision Transformer (ViT) have demonstrated that self-attention-based architectures can scale effectively across modalities, enabling state-of-the-art performance in text generation, image classification, and multimodal reasoning [ 22 , 6 , 10 ] . As these models continue to grow in size and adoption, scalable and latency-aware inference infrastructure has become critical for deploying them in production environments. Hugging Face Transformers To address this challenge, several software building blocks have emerged to simplify and abstract the deployment of Transformer-based models. Hugging Face’s transformers library provides the widely used pipeline API, which bundles pretrained models with tokenizers and pre/post-processing steps for common tasks. While this abstraction accelerates prototyping and experimentation, it is designed for single-node execution and lacks orchestration features such as autoscaling, load balancing, or multi-accelerator support [ 24 ] . PyTorch Execution & Backend Support At the execution layer, PyTorch plays a foundational role by supporting both eager-mode and graph-mode inference. With backends like CUDA for NVIDIA GPUs and XLA for AWS Trainium and Inferentia, PyTorch enables developers to deploy the same model across a variety of accelerators without changing core logic [ 17 , 5 ] . These hardware-agnostic abstractions are essential for decoupling model development from infrastructure-specific execution details. Ray Serve Ray Serve offers a general-purpose serving framework that abstracts inference deployments into autoscaling actors running on Ray clusters. It supports composable DAGs, multi-model workloads, and heterogeneous infrastructure [ 14 ] . However, while Ray Serve provides flexibility at the application layer, it does not natively handle cost-performance trade-offs or real-time hardware-aware scheduling—capabilities needed to fully optimize inference across diverse accelerators. vLLM and PagedAttention For high-throughput serving of large language models specifically, vLLM introduces several inference-time optimizations tailored to the autoregressive decoding pattern. Its core innovation, PagedAttention , allows the key–value cache to be paged efficiently in GPU memory, reducing fragmentation and maximizing memory reuse. These techniques enable 2–4× higher throughput at equivalent latency compared to conventional implementations [ 13 ] . vLLM also supports tokenizer parallelism and asynchronous scheduling for efficient batching. While originally designed for GPU inference, vLLM now includes support for AWS Neuron devices, enabling deployment on Inferentia and Trainium hardware [ 23 ] . However, its focus remains on LLMs, and it does not yet extend to vision or multimodal models. Our work builds on these capabilities

Read the complete text