HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location

Ting Sun, Penghan Wang · 2025

Read HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location by Ting Sun, Penghan Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14808.txt.

Introduction

Ting Sun and co-authors present, in a 2025 paper, a serving system that lets chatty and batch AI work share the same machines. Language model services split into two kinds: live chats that must answer fast, and offline jobs, like data making, that prize volume. Giving each its own machines wastes capacity. Their system, HyGen, mixes the two on one fleet while keeping live replies within their speed promises.

Two tools make the mix safe. A predictor estimates how long each batch will run, and a profiler measures how much the batch work slows the live work. Scheduling rules then feed offline jobs into the gaps without starving them. On real production loads, HyGen lifted total throughput several-fold over standard setups. The code is public.

Catalog record

CREATORTing Sun, Penghan Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14808

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location Ting Sun, Penghan Wang Abstract Large language models (LLMs) have facilitated a wide range of applications with distinct service-level objectives (SLOs), from latency-sensitive online tasks like interactive chatbots to throughput-oriented offline workloads like data synthesis. The existing deployment model, which dedicates machines to each workload, simplifies SLO management but often leads to poor resource utilization. This paper introduces HyGen, an interference-aware LLM serving system that enables efficient co-location of online and offline workloads while preserving SLOs. HyGen incorporates two key innovations: (1) performance control mechanisms, including a latency predictor to estimate batch execution time and an SLO-aware profiler to quantify latency interference, and (2) SLO-aware offline scheduling policies that maximize serving throughput and prevent starvation. Our evaluation on production workloads shows that HyGen achieves up to 3.9-5.8 × \times throughput gains over online and hybrid serving baselines, while ensuring latency SLOs. The code of HyGen is publicly available at https://github.com/UIUC-MLSys/HyGen . 1 Introduction Large language models (LLMs) have emerged as transformative tools across diverse domains, handling both latency-critical online requests (e.g., chatbot interactions [ 32 , 39 ] ) and throughput-oriented offline tasks (e.g., document summarization [ 18 ] ). Online serving demands low and stable response times, measured by Time to First Token (TTFT) and Time Between Tokens (TBT), while offline tasks prioritize high throughput and resource utilization, often processing large batches with relaxed latency constraints. The disparity in these requirements has led most production deployments to segregate online and offline serving onto separate clusters to avoid interference [ 41 , 51 , 54 , 69 ] . However, real-world LLM workloads demonstrate significant temporal variations in request load. Our analysis of production traces shows that in addition to the popular diurnal request arrival patterns, online request rates can vary by up to 3 \times within minutes (Section 3 ). To meet latency requirements under such bursty loads, service providers have to provision GPU resources for peak demand, as minute-level resource scaling is often impractical due to infrastructure complexity and engineering overhead [ 15 , 64 , 66 ] —leading to substantial underutilization during off-peak hours. This resource overprovisioning suggests an opportunity to improve resource efficiency via hybrid serving—co-locating online and offline workloads on the same inference engine instance. By opportunistically padding online requests with offline requests during periods of low online load, the system could maintain high GPU utilization while preserving latency guarantees for online requests. However, realizing this opportunity requires addressing several fundamental challenges. First, LLM services exhibit diverse latency requirements across applications [ 32 ] . Interactive chatbots require consistent response times with both low initial latency (TTFT) and smooth token generation (TBT), while batch processing tasks prioritize throughput over latency. These requirements often manifest in different statistical metrics—from strict P99 latency bounds to mean performance targets—making it difficult to establish unified resource-sharing policies. Second, LLM workloads are inherently unpredictable in both their arrival patterns and resource demands. Request arrival rates exhibit both diurnal patterns and unpredictable short-term fluctuations. This variability is further complicated by uncertainty in resource demands—input sequences vary widely in length, and the number of output tokens can hardly be predicted until generation completes. Third, co-locating online and offline workloads introduces interference. For instance, offline requests may delay time-sensitive online requests; using large batch sizes to improve the throughput of offline requests can increase the latency of online requests. Effectively managing this interference while ensuring latency guarantees demands meticulous orchestration of resource sharing. This paper presents HyGen, an interference-aware LLM serving system that elastically co-locates online and offline workloads. HyGen introduces several key techniques: (1) a latency predictor that accurately estimates the execution time of different request batches, (2) an interference-aware profiler that quantifies the performance interference of co-location, and (3) an adaptive scheduler that maximizes offline throughput while maintaining strict latency guarantees for online requests. Our evaluation on production workloads shows that HyGen improves serving throughput by 3.87-5.84 \times over existing advances [ 2 ] , while guaranteeing strict SLO compliance. To summarize, this paper makes the following contributions: • Key insights on the feasibility and benefits of co-locating online and offline LLM workloads, derived from systematic characterization of production traces. • A statistical latency prediction model that accurately captures the relationship between batch composition and execution latency, accounting for quadratic complexity in prefill and linear scaling in decode phases. • A novel scheduling system that dynamically co-locates online and offline workloads, formulated as a constrained optimization problem that maximizes throughput with prefix sharing while preserving strict latency SLOs and providing theoretical guarantees for fairness. • Experimental evaluation on real-world workloads demonstrating up to 5.84 \times throughput improvement over state-of-the-art alternatives. 2 Related Work LLM Inference Optimization. Recent advances in LLM inference through kernel optimization [ 12 , 61 ] , compilation frameworks [ 33 , 59 , 72 ] , and scheduling algorithms [ 30 , 31 , 55 ] have substantially improved LLM serving performance. Among these, a particularly important development is predictive scheduling, which aims to estimate request difficulty [ 13 , 38 ] or generation length [ 16 , 23 , 27 , 43 , 45 ] . Contrasting these efforts, HyGen uniquely predicts batch execution time, providing precise control over request interference during co-located workloads. Additionally, the concept of request prefix sharing has been explored by [ 17 , 28 , 71 ] , where shared prefixes across requests optimize resource use. HyGen takes this a step further by applying a prefix sharing maximization strategy to opportunistically schedule offline requests, utilizing residual capacity from a primary, SLO-bound online workload. Moreover, HyGen introduces a fairness-aware extension to the standard prefix-sharing maximization (PSM) method, addressing the issue of starvation often encountered in naive designs. In parallel, the increasing demand for offline LLM inference has driven two key trends. First, platforms like Huggingface Accelerate [ 25 ] , DeepSpeed ZeRO-Inference [ 4 ] , and FlexGen [ 51 ] enable efficient inference on commodity hardware through memory offloading across GPUs, CPUs, and disk. Second, cloud providers, including OpenAI’s Batch API [ 40 ] , have launched specialized services optimized for throughput rather than latency, processing millions of requests daily for large-scale data processing tasks. Workloads Co-location. In data center environments, co-locating latency-sensitive applications with batch applications has been explored to improve resource utilization [ 8 , 9 , 42 , 68 ] . However, these approaches generally overlook the unique characteristics of LLM inference. In the context of LLM inference, several works have investigated the co-location of various model types and tasks. For instance, Punica [ 7 ] , S-LORA [ 49 ] , and dLoRA [ 58 ] batch requests from different LoRAs, while MuxServe [ 14 ] multiplexes resources across multiple LLMs. In contrast, HyGen focuses on batching online and offline requests, addressing distinct optimization problems that are not interchangeable. 3 Background and Motivation This section first introduces the background of LLM serving deployment (Section 3.1 ), then illustrates how these characteristics motivate our system designs (Section 3.2 ). 3.1 LLM Serving Large-scale inference clusters consist of multiple serving instances, with a router intelligently directing incoming requests to the most suitable instances [ 53 , 65 ] . Each

Read the complete text