SkyServe: Serving AI Models across Regions and Clouds with Spot Instances
Ziming Mao ∗† Tian Xia ∗† Zhanghao Wu † Wei-Lin Chiang † Tyler Griggs † Romil Bhardwaj † Zongheng Yang † Scott Shenker †⋄ Ion Stoica † † UC Berkeley ⋄ ICSI · 2024
Read SkyServe: Serving AI Models across Regions and Clouds with Spot Instances by Ziming Mao ∗† Tian Xia ∗† Zhanghao Wu † Wei-Lin Chiang † Tyler Griggs † Romil Bhardwaj † Zongheng Yang † Scott Shenker †⋄ Ion Stoica † † UC Berkeley ⋄ ICSI (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01438.txt.
Description
A 2024 arXiv research paper by Ziming Mao ∗† Tian Xia ∗† Zhanghao Wu † Wei-Lin Chiang † Tyler Griggs † Romil Bhardwaj † Zongheng Yang † Scott Shenker †⋄ Ion Stoica † † UC Berkeley ⋄ ICSI. Recent years have witnessed an explosive growth of AI models. The high cost of hosting AI services on GPUs and their demanding service requirements, make it timely and challenging to lower service costs and guarantee service quality.
Catalog record
| CREATOR | Ziming Mao ∗† Tian Xia ∗† Zhanghao Wu † Wei-Lin Chiang † Tyler Griggs † Romil Bhardwaj † Zongheng Yang † Scott Shenker †⋄ Ion Stoica † † UC Berkeley ⋄ ICSI |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01438
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SkyServe: Serving AI Models across Regions and Clouds with Spot Instances Thanks: ∗ Equal Contributions Ziming Mao ∗† Tian Xia ∗† Zhanghao Wu † Wei-Lin Chiang † Tyler Griggs † Romil Bhardwaj † Zongheng Yang † Scott Shenker †⋄ Ion Stoica † † UC Berkeley ⋄ ICSI Abstract . Recent years have witnessed an explosive growth of AI models. The high cost of hosting AI services on GPUs and their demanding service requirements, make it timely and challenging to lower service costs and guarantee service quality. While spot instances have long been offered with a large discount, spot preemptions have discouraged users from using them to host model replicas when serving AI models. To address this, we propose a simple yet efficient policy, SpotHedge, that leverages spot replicas across different failure domains (e.g., regions and clouds) to ensure availability, lower costs, and high service quality. SpotHedge intelligently spreads spot replicas across different regions and clouds to improve availability and reduce correlated preemptions, overprovisions cheap spot replicas than required as a safeguard against possible preemptions, and dynamically falls back to on-demand replicas when spot replicas become unavailable. We built SkyServe, a system leveraging SpotHedge to efficiently serve AI models over a mixture of spot and on-demand replicas across regions and clouds. We compared SkyServe with both research and production systems on real AI workloads: SkyServe reduces cost by 43% on average while achieving high resource availability compared to using on-demand replicas. Additionally, SkyServe improves P50, P90, and P99 latency by 2.3 × \times , 2.1 × \times , 2.1 × \times on average compared to other research and production systems. Figure 1 . SkyServe Overview. SkyServe leverages SpotHedge to intelligently provisions and manages a mixture of spot and on-demand replicas across regions and clouds to minimize preemptions, improve availability, and reduce cost. thoncode [yamlcode]yaml baselinestretch=1.2 Conference: Twentieth European Conference on Computer Systems; March 30–April 3, 2025; Rotterdam, Netherlands Twentieth European Conference on Computer Systems (EuroSys ’25), March 30–April 3, 2025, Rotterdam, Netherlands DOI: 10.1145/3689031.3717459 ISBN: 979-8-4007-1196-1/25/03 CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Distributed computing methodologies 1. Introduction Generative AI has experienced explosive growth in the past several years, which has enabled a plethora of new applications, such as large language model (LLM) chatbots ( Goo, 2024 ; ope, 2024b ) , programming assistants ( git, 2024 ) , image generation ( ope, 2024a ) , and writing assistants ( gra, 2024 ) . Many companies ( any, 2024 ; ope, 2024c ; tog, 2025 ; goo, 2025 ; ant, 2025 ) offer these models as hosted services on the cloud. A service is composed of multiple model replicas; each replica runs on one or more GPU instances. However, serving these models is challenging: not only do they need to be highly available and serve user requests under tight latency constraints, but they are also expensive to operate. Each request served by these models can take several seconds, if not tens of seconds to process. ( Spo, 2024 ; Seery, 2024 ; Li et al., 2023 ) . Compared to traditional web services, AI systems have much higher compute requirements ( OpenAI, 2024 ) and costs ( ai-, 2024 ) . There are two reasons for the high cost of serving AI models. First, these models require expensive GPU accelerators. As a result, processing a request can be 10 \times more expensive than a traditional search engine query ( ai-, 2024 ) . Second, real-life AI workloads have frequent and unpredictable traffic spikes (up to 50 \times on average ( Li et al., 2023 ) ) and service latency fluctuations ( ope, 2024d ) . This results in organizations over-provisioning (provisioning more than required by traffic) more replicas than needed to serve average user traffic or sometimes even provisioning for the peak load, both of which exacerbate the high serving cost. Spot instances have long been offered by cloud providers as a cost-saving option (8-50% cost of on-demand; Table 1 ). However, serving AI models on spot GPU instances (or "spot replicas") has a few key challenges. First, spot instances can be preempted by the provider at any time, and preemptions of spot GPUs are much more common than spot CPUs (§ 2.3 ). Second, when preemptions happen, service quality can degrade due to fewer replicas responding to requests. Naively placing spot replicas in a single region can lead to limited availability and correlated preemptions, where multiple spot instances are preempted simultaneously, potentially resulting in service downtime (§ 2.2 ). Finally, recovery of spot replicas can be slow due to the long cold start and provisioning delays, on the order of minutes (§ 2.3 ), or even infeasible, due to the immediate unavailability of spot instances after preemption, in the same zone or region. Most prior work ( Yang et al., 2023a ; Wagenländer et al., 2020 ; Harlap et al., 2017 ; Harlap et al., 2018 ; Zhang et al., 2019 ; Gunasekaran et al., 2022 ) focuses on the more available spot CPU instances (§ 2.3 ) or on training workloads ( Thorpe et al., 2023 ; Athlur et al., 2022 ; Yang et al., 2021 ) . However, serving AI models on spot GPUs requires the system to be robust to frequent preemptions, spot unavailability, and significant cold start delays. Spot preemption warnings cannot address the problem, as the time to find available instances, provision, and load models typically exceeds the best-effort preemption warnings (2 minutes on AWS and 30 seconds on GCP and Azure). Spot instances can also be simultaneously unavailable or preempted in practice (§ 2.2 ). As such, serving model replicas on spot GPUs while maintaining high service quality has not been widely considered viable in practice. We show that leveraging spot replicas in AI model serving is not only feasible, but can ensure high availability, lower costs, and improve service quality. We propose a simple yet effective policy, SpotHedge, that uses a dynamic mixture of spot and on-demand replicas across regions and clouds to minimize the cost and improve service latency. First, SpotHedge improves availability and decorrelates preemptions by spreading spot replicas across wider failure domains (regions or clouds), compared to the common practice of launching in the same zone or region ( goo, 2024a ; aws, 2024d ; Zhang et al., 2019 ; Miao et al., 2023 ) . Using more regions and clouds enlarges the search space where spot instances can be provisioned, significantly improving availability and speeding up recovery. We have observed that a single-region deployment of spot replicas is often not viable due to instance unavailability, leading to service downtime (§ 2.2 ). Second, to avoid degraded service on preemptions, and to ensure high availability, SpotHedge proactively uses on-demand replicas (model replicas running on on-demand instances) as a fallback . On-demand instances are typically available if we search and provision across regions and clouds. Third, SpotHedge mitigates preemption by over-provisioning with cheap spot replicas instead of expensive on-demand replicas. Even when some spot replicas are preempted, these additional over-provisioned replicas will mitigate the impact of preemption. Thus, a service is backed by a dynamic mixture of spot and on-demand replicas. We implemented SpotHedge in SkyServe, a real system that