Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking Journal: PACMMOD Volume: 4 3 (SIGMOD) 230 6 DOI: 10.1145/3802107 CCS: Computer systems organization Cloud computing
Zizhao Mo, Junlin Chen, Huanle Xu, Chengzhong Xu · 2026
Read Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking Journal: PACMMOD Volume: 4 3 (SIGMOD) 230 6 DOI: 10.1145/3802107 CCS: Computer systems organization Cloud computing by Zizhao Mo, Junlin Chen, Huanle Xu, Chengzhong Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12831.txt.
Introduction
Zizhao Mo and co-authors serve mixed language model loads in shared clusters without breaking latency promises. Latency-sensitive services have strict limits, while best-effort services get squeezed by memory and interference. Reserving headroom is a blunt fix that hurts both sides. Their OmniServe offloads attention work from best-effort jobs to CPUs on the fly. Async links keep GPUs from stalling while results return. Tests showed far better SLO attainment and best-effort throughput than prior systems.
Catalog record
| CREATOR | Zizhao Mo, Junlin Chen, Huanle Xu, Chengzhong Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79510 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12831
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking Journal: PACMMOD Volume: 4 3 (SIGMOD) 230 6 DOI: 10.1145/3802107 CCS: Computer systems organization Cloud computing Zizhao Mo, Junlin Chen, Huanle Xu, Chengzhong Xu Abstract . Nowadays, service providers often deploy multiple types of LLM services within shared clusters. While the service co-location improves resource utilization, it introduces significant interference risks for latency-sensitive (LS) services—which have strict SLO requirements for inference latency—and severely constrains the service capacity of best-effort (BE) services due to limited available memory. To address interference, existing systems typically rely on reserving headroom to constrain BE resource usage. However, this approach’s coarse granularity compromises the SLO compliance of the latency-sensitive service and unnecessarily restricts the generation potential of the best-effort service. In this paper, we propose OmniServe, a novel LLM serving system that efficiently harnesses both CPU and GPU resources to mitigate interference and improve throughput. Central to OmniServe is the Attention Piggybacking mechanism, which effectively offloads the Attention computation of BE services to CPUs on the fly. This mechanism also facilitates asynchronous communication between CPU and GPU streams, preventing GPUs from being blocked while aggregating Attention results. Additionally, OmniServe incorporates a dynamic batching control policy to adapt to fluctuating request arrivals, facilitating Dense module computation using layer-wise batching. Experimental results show that OmniServe improves the SLO attainment rate for LS services by up to 1.48 × 1.48\times while enhancing BE serving throughput by up to 9.85 × 9.85\times compared to state-of-the-art systems. † † cc-license: by 1. Introduction Nowadays, service providers aim to deliver a diverse range of services in datacenters using large language models (LLMs), to offer versatility and flexibility across various domains such as programming assistance and and question answering. Current foundational LLM models ( 48 ; 14 ) are typically trained on vast amounts of real-world data, to capture knowledge from all aspects of human life. While a variety of LLM services are deployed together in clusters to enhance resource utilization, their service requirements vary significantly. For example, online chatting is a latency-sensitive application ( 3 ) where users expect stringent service level objectives (SLOs) to ensure fast responses and high token generation rates. In contrast, many services focus on back-office tasks, such as benchmarking, form processing, and data wrangling ( 25 ; 7 ; 13 ) . These workloads have more lenient SLOs and relatively low priority, requiring service provisioning on a best-effort basis. Furthermore, even within the same service type, non-uniform service provisioning for different users is a commonly adopted strategy. For example, service providers may offer free users best-effort access while ensuring rapid responses for paid users. As LLM-based services become increasingly popular in data centers, it is critical to simultaneously optimize performance for different types of services ( 11 ; 18 ; 38 ) . This often requires maximizing generation throughput of best-effort (BE) service requests without negatively impacting SLO guarantees for latency-sensitive (LS) services. To achieve this goal, Llumnix ( 47 ) proposes a priority-based scheduling strategy to dynamically schedule various requests across multiple LLM serving instances. For ensuring performance isolation between different types of services, Llumnix limits the KV cache usage for BE services by reserving headroom for LS services, recognizing the positive correlation between memory usage and the level of interference introduced to LS services. Despite its operational simplicity in production environments, Llumnix’s GPU memory-centric paradigm inadequately sustains balanced quality-of-service for hybrid workloads. First, its coarse-grained memory controls fail to effectively address compute resource contention—experiments reveal persistent 50% LS latency variance even with fixed memory reservations for LS services (Fig. 2 (a)), attributable to unmanaged GPU SM/Memory bandwidth competition. Second, the framework’s rigid prioritization during LS traffic surges triggers BE starvation, reducing BE throughput significantly due to GPU memory monopolization. Fortunately, our analysis reveals that the underutilized CPU resources in LLM serving clusters can be strategically leveraged to enhance BE computation while minimizing interference with LS services. Specifically, idle CPU cores can compute the offloaded BE Attention workloads to reduce contention on GPUs, while the abundant CPU memory accommodates significantly larger KV caches for BE requests. However, fully exploiting CPU for BE throughput faces a critical bottleneck: the 498.1 \times performance gap in Dense module computation between GPUs and CPUs. This disparity necessitates reserving Dense computations for GPUs, creating a dependency where GPUs must wait for Attention results computed on CPUs. To avoid GPU blocking, a robust synchronization mechanism is required to orchestrate heterogeneous compute streams across devices. This challenge is further compounded by the dynamic nature of LLM workloads. Fluctuating request arrival rates widen the CPU-GPU performance gap under larger batch sizes, making synchronization increasingly complex during traffic bursts. To address these challenges associated with harnessing CPU resources, this paper proposes OmniServe, an efficient LLM system for serving hybrid loads with SLO-awareness. The first innovation of OmniServe is the introduction of Attention Piggybacking, a new CPU inference mechanism designed to effectively decouple CPU Attention computation from GPU inference. Specifically, Attention Piggybacking enables an asynchronous execution flow between CPUs and GPUs, allowing the GPU stream to perform inference without waiting for gathering immediate Attention results from the CPU stream. Dense computations for these overflowed BE requests can be opportunistically piggybacked later using layer-wise batching, introducing minimal interference to LS services with fluctuating request arrival rates. Unlike token-wise batching ( 56 ) , the set of requests executed within a batch under layer-wise batching can vary across layers during a single token iteration. The second innovation of OmniServe lies in the dynamic batching control through explicit latency quantification. OmniServe leverages the high predictability of service latency at the module level to dynamically determine the appropriate number of BE requests that can be piggybacked with LS inference using layer-wise batching during decoding. This approach not only ultimately ensures the SLO for LS services, but also enhances the BE service throughput. The Attention Piggybacking mechanism and explicit batching control approach are highly flexible and can be seamlessly integrated with various parallelism techniques for distributed inference, as well as into state-of-the-art LLM serving systems, such as Prefill-Decode disaggregation ( 63 ; 36 ; 35 ) and chunk prefill ( 1 ) . We have developed a prototype of OmniServe based on the vLLM framework ( 22 ) . Additionally, we conduct extensive experiments to evaluate OmniServe in a cluster consisting of one host with four A100 GPUs and four additional CPU-only hosts. All hosts features Intel Xeon(R) Gold 6342 CPUs. The results show that OmniServe can improve the SLO attainment rate for LS service by up to 1.48 \times while enhancing the BE serving throughput by up to 9.85\times , compared to existing hybrid serving systems. To summarize, we have made the following contributions in this paper: • We conduct a comprehensive analysis to examine the interplay between BE and LS services, highlighting the need for a more efficient interference mitigation approach in hybrid serving. • We design a novel Attention piggybacking mechanism for efficient service co-location. With asynchronous CPU-GPU interaction, GPUs do not need to wait for CPU results, representing a fundamental departure from prior offloading approaches. • We introduce a dynamic scheduling policy that builds on the Attention piggybacking mechanism. Its core innovation is a fine-grained, layer-level