Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs
Yixuan Mei, Zikun Li, Zixuan Chen, Shiqi Pan, Mengdi Wu, Xupeng Miao, Zhihao Jia, K. V. Rashmi · 2026
Read Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs by Yixuan Mei, Zikun Li, Zixuan Chen, Shiqi Pan, Mengdi Wu, Xupeng Miao, Zhihao Jia, K. V. Rashmi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04357.txt.
Introduction
Yixuan Mei and seven co-authors build a serving system for running many different large language models at once on mixed fleets of cloud graphics processors. No single model dominates the field, and older or mid-range chips are cheaper and easier to get than top hardware.
Their system jointly decides how resources are split among models and how each model's copies should serve requests. A two-stage method keeps that joint choice optimal while cutting the time to recompute it from hours to tens of seconds as demand shifts. Across six models and twenty hardware setups, it cut serving costs sharply against the strongest baseline.
Catalog record
| CREATOR | Yixuan Mei, Zikun Li, Zixuan Chen, Shiqi Pan, Mengdi Wu, Xupeng Miao, Zhihao Jia, K. V. Rashmi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04357
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs Yixuan Mei, Zikun Li, Zixuan Chen, Shiqi Pan, Mengdi Wu, Xupeng Miao, Zhihao Jia, K. V. Rashmi Abstract The usage of large language models (LLMs) has grown increasingly fragmented, with no single model dominating. Meanwhile, cloud providers offer a wide range of mid-tier and older-generation GPUs that enjoy better availability and deliver comparable performance per dollar to top-tier hardware. To efficiently harness these heterogeneous resources for serving multiple LLMs concurrently, we introduce Coral, an adaptive heterogeneity-aware multi-LLM serving system. The key idea behind Coral is to jointly optimize resource allocation and the serving strategy of each model replica across all models. To keep pace with shifting throughput demand and resource availability, Coral applies a lossless two-stage decomposition that preserves joint optimality while cutting online solve time from hours to tens of seconds. Our evaluation across 6 models and 20 GPU configurations shows that Coral reduces serving cost by up to 2.79 × \times over the best baseline, and delivers up to 2.39 × \times higher goodput under scarce resource availability. 1 Introduction Large language models (LLMs) are being deployed across an ever-widening range of tasks, including interactive chatbots [ 41 , 7 ] , automated code generation [ 5 , 55 , 39 ] , and agentic workflows [ 50 , 47 ] . No single model dominates this landscape. Models from different families excel at different tasks [ 26 , 6 , 22 ] , and within each family, models of different sizes target different cost–quality trade-offs [ 49 , 18 , 30 ] . Recent industry data further illustrates this fragmentation: no single model handles more than a quarter of queries [ 37 ] . As a result, providers must serve many models concurrently. For example, Perplexity serves 46 models behind its LLM-powered answer engine [ 37 ] , and Microsoft Office 365’s AI features are backed by seven different LLMs from three families [ 17 ] . To avoid the prohibitive capital expenditure of dedicated AI hardware, organizations increasingly turn to public clouds to host these diverse models [ 33 , 3 ] . As Table 1 shows, these clouds are heterogeneous, offering GPUs that span multiple generations and hardware specifications. Top-end GPUs (e.g., B200, H100) deliver the highest per-GPU capability but are often supply-constrained. Older and mid-tier GPUs (e.g., A100, L4) are more widely available and substantially cheaper, and they offer comparable or even better cost efficiency per unit of compute or memory than the latest generation. Together, model fragmentation and cloud GPU heterogeneity motivate a new problem: cost-efficient, adaptive multi-LLM serving on heterogeneous cloud resources . In this problem, a serving system is given a set of models with latency service-level objectives (SLOs), the current per-model throughput demand, and the real-time price and availability of different GPU configurations across cloud regions. From these inputs, it must jointly produce two outputs. The first is a resource allocation that decides which nodes to provision. The second is a model placement that decides how to partition each model across those heterogeneous nodes. The goal is to satisfy every model’s throughput demand and latency SLO at minimum total cost. Furthermore, because demand, availability, and prices shift over time, the system must re-solve the problem periodically to adapt the cluster to these changes. GPU A G R Rel. Cost Mem (GB) BW (TB/s) TF LOPS Perf. Per Cost Mem BW TF H100 ✓ ✓ ✓ 7.6 80 3.35 989 10.5 0.44 129.8 A100 ✓ ✓ ✓ 3.5 80 2.04 312 22.8 0.58 88.9 L40S ✓ ✗ ✓ 2.2 48 0.86 362 21.5 0.39 162.3 L4 ✓ ✓ ✓ 1.0 24 0.30 121 24.0 0.30 121.0 A10G ✓ ✗ ✗ 1.2 24 0.60 70 19.7 0.49 57.4 Table 1 : GPU specs and availabilities on AWS (A), GCP (G), and RunPod (R). Relative cost is mean hourly price normalized to L4. “Perf. Per Cost” divides metrics by relative cost. Criterion Coral SkyS. [ 28 ] Sage. [ 17 ] Cauchy [ 53 ] Helix [ 29 ] HexG. [ 21 ] Resource Alloc. Joint Opt. ✓ ✓ ✓ ✗ ✗ Model Placement ✗ ✗ ✗ ✓ ✓ Latency SLO ✓ ✗ ✓ ✓ ✗ ✓ Multi-LLM ✓ ✗ ✓ ✓ ✗ ✗ Table 2 : Comparison with prior work on heterogeneous LLM serving. Existing systems optimize either resource allocation [ 28 , 17 , 53 ] or model placement [ 29 , 21 ] , but never jointly. Coral is the first to co-optimize both for multiple models under per-model latency SLOs on heterogeneous hardware. Prior work tackles this problem only in pieces. Existing heterogeneity-aware allocation systems target orthogonal goals, such as spot preemption resilience via cross-region replica spreading [ 28 ] and mixed interactive/batch workloads via forecast-driven replica scaling [ 17 ] . They share a common limitation: each model replica is treated as a black box on homogeneous hardware, which collapses model placement out of the optimization space. Cauchy [ 53 ] relaxes this for Prefill-Decode disaggregated serving by letting prefill and decode use different GPU configurations, but each phase remains internally homogeneous. As both the examples in Sec. 2.2 and our evaluation results in Sec. 6 show, the result is not only substantial cost savings left on the table, but outright infeasibility under constrained resource availability. Helix [ 29 ] and HexGen [ 21 ] take the opposite stance, optimizing placement for a single model on a given heterogeneous node set. Wrapping them in an outer enumeration over allocations is intractable. The allocation space is exponential, and each inner placement solve is already expensive. Helix alone reports four hours for a single model on just 24 nodes. As Table 2 summarizes, no existing system addresses the problem end-to-end. Solving the heterogeneity-aware multi-LLM serving problem end-to-end poses two key challenges. First, resource allocation and model placement are tightly coupled, and each has an exponential solution space. The throughput and SLO attainable on a node set depend on how the model is placed across it. Conversely, the optimal model placement can only be determined once the node set is fixed. Neither dimension can be pruned without committing to the other. Brute-forcing the joint problem is equally infeasible, as the combined search space is exponential in both dimensions. Second, the solution must be produced in minutes, not hours. Both GPU availability [ 46 , 43 ] and throughput demand [ 42 ] shift quickly. A solution that arrives too late is invalidated by the very changes it was meant to adapt to. To address these challenges, we present Coral 1 1 1 Coral stands for C ost-efficient O rchestration of R esources for A daptive L LM-serving , an adaptive, heterogeneity-aware multi-LLM serving system. Coral builds on a key observation: given a model and its latency SLO, the throughput-optimal model placement on any node combination depends only on that combination. It is independent of how the rest of the cluster is allocated. The optimal placement is therefore a reusable artifact that can be pre-computed offline and cached . This decouples placement from allocation without sacrificing joint optimality. We call this artifact a Serving Template . The space of node combinations is unbounded in