AccelGen : Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications
Haiying Shen, Tanmoy Sen · 2025
Read AccelGen : Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications by Haiying Shen, Tanmoy Sen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13737.txt.
Introduction
Serving many language-model apps on one fleet means juggling short and long prompts. Each carries its own deadline for the next token. Haiying Shen and Tanmoy Sen built AccelGen around that juggling act. It sizes work chunks dynamically to keep graphics processors busy without breaking any deadline. It orders tasks by tightness of deadline, and batches requests to fill both compute and memory.
On real traces it beat rival systems by wide margins: up to eleven times the throughput, far fewer missed deadlines, and latency cut by as much as twelvefold. It nearly matched an all-knowing oracle scheduler. Shared serving, done this way, wastes much less hardware.
Catalog record
| CREATOR | Haiying Shen, Tanmoy Sen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13737
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AccelGen : Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications Haiying Shen, Tanmoy Sen Abstract In this paper, we consider a mixed-prompt scenario for a large language model (LLM) inference serving system that supports diverse applications with both short prompts and long prompts and heterogeneous SLOs for iteration time. To improve throughput when handling long prompts, previous research introduces a chunking method, but has not addressed heterogeneous SLOs. To address the limitation, we propose AccelGen , a high-throughput LLM inference serving system with heterogeneous SLO guarantees for diverse applications. AccelGen introduces three core components: (1) SLO-guaranteed dynamic chunking, which dynamically adjusts chunk sizes to maximize GPU compute utilization while meeting iteration-level SLOs; (2) Iteration-level SLO-based task prioritization, which prioritizes tight-SLO requests and batches requests with similar SLOs; (3) Multi-resource-aware batching, which selects queued requests to maximize the utilizations of both GPU compute resource and key-value cache (KVC). Trace-driven real experiments demonstrate that AccelGen achieves 1.42-11.21 × \times higher throughput, 1.43-13.71 × \times higher goodput, 37-90% higher SLO attainment, and 1.61-12.22 × \times lower response latency compared to the state-of-the-art approaches. It achieves performance near the Oracle , which optimally maximizes goodput. 1 Introduction Transformer-based generative large-language models (LLMs) have garnered considerable attention in Natural Language Processing (NLP) applications such as text generation [ 1 ] , text summarization [ 2 ] , code generation [ 3 ] , and chatbot [ 4 ] . In the model inference, each request begins with a user-entered prompt, comprising a sequence of tokens. The model processes this prompt in the first iteration to generate the initial token, and subsequently generates tokens in an autoregressive manner [ 5 ] in each iteration until the entire response text is produced. Requests are processed in batches, including the prompt processing (PP) tasks and the token generation (TG) tasks. Given the diverse applications running on an LLM inference serving system, prompt lengths can vary significantly, ranging from just a few tokens for chat applications to 4K-100K tokens for more extensive content like electronic books used in text summarization, code generation, and legal document analysis [ 6 ] . Additionally, user experience is critical; users may have different iteration time service-level-objectives (SLOs) in online applications based on their reading speed and application requirements, with a typical reading speed around 0.1875 seconds per token [ 7 , 8 , 9 ] . Offline applications, on the other hand, may impose job completion time (JCT) SLOs. Thus, in this paper, we consider a mixed-prompt scenario, consisting of short prompts and long prompts (containing \geq 4K tokens) and heterogeneous SLOs. To improve throughput when handling long prompts, previous research introduces chunking [ 10 , 11 , 12 , 13 ] . The approaches select chunks or requests from the waiting queue until the target forward size (or token budget) – defined as the desired total number of tokens in a batch – is reached, to fully utilize the GPU compute resource. However, our trace-driven measurement analysis ( § 3 ) reveals that these systems fail to meet heterogeneous iteration-level SLOs, potentially degrade user experience, and fall short of maximizing GPU compute and key-value cache (KVC) utilization. Specifically, our findings indicate that: (1) Heterogeneous iteration-level SLOs cannot be met in current LLM systems, making it necessary to account for SLO diversity in scheduling. (2) Focusing solely on maximizing GPU compute resources does not guarantee maximum throughput for a workload. It is essential to jointly maximize both GPU compute utilization and KVC utilization at each iteration to improve throughput. (3) We should limit the number of concurrently running long-prompt requests to free up KVC resources more quickly to improve throughput. (4) The first-come-first-serve (FCFS) policy used in current LLM systems [ 14 , 12 ] underutilizes GPU resources when meeting iteration-level SLOs, but batching requests with similar SLOs enhances throughput. (5) The varying available GPU compute resource and unallocated KVC space after each iteration, coupled with the diverse GPU and KVC demands of different prompts, provide an opportunity to identify prompts or prompt chunks to be added to the batch to maximize both GPU compute and KVC utilizations, thus improving throughput. Building on these key insights, based on the chunking method of FastGen, we propose AccelGen , a high-throughput LLM inference serving system designed to meet heterogeneous SLOs while maximizing throughput for diverse applications. AccelGen incorporates the following methods to achieve the goal: (1) SLO-guaranteed dynamic chunking. AccelGen determines the token budget and batches requests with similar SLOs to maximize GPU compute utilization as much as possible while satisfying the iteration-level SLOs of the requests in the batch. In addition, it limits concurrently running long-prompt requests to improve throughput. (2) Iteration-level SLO-based task prioritization. AccelGen enables users to specify the SLOs for Time-To-First-Token (TTFT) in PP and for Time-Between-Tokens (TBT) in TG of a job, or specify the JCT SLO for a job. It converts a JCT SLO into iteration-level SLOs. Based on iteration-level SLOs, it orders the waiting requests to enhance SLO compliance and facilitate batching requests with similar SLOs to enhance throughput. (3) Multi-resource-aware batching. After each iteration, AccelGen selects requests and adjusts prompt chunk lengths from the prioritized waiting queue to maximize both GPU compute utilization and KVC utilization in the subsequent iteration. Our tace-driven real experiments show that AccelGen achieves 1.42-11.21 \times higher throughput, 1.43-13.71 \times higher goodput, 37-90% higher SLO attainment, and 1.61-12.22 \times lower response latency compared to the state-of-the-art approaches. This is the first paper on heterogeneous iteration-level SLOs, a topic of interest to industry for practical applications. 2 Background and Theoretical Founcation 2.1 Background As a representative of the transformer model, we illustrate the OPT structure in Figure 1 . OPT is a decoder-only transformer model and its layers fall into two categories: 1) forward computation layers (FCL), and 2) attention layer. The attention layer takes three input components: Query ( \mathbf{Q} ), Key ( \mathbf{K} ), and Value ( \mathbf{V} ). These components are obtained by projecting the embeddings ( \mathbf{E} ) using three learned matrices \mathbf{W_{q}} , \mathbf{W_{k}} , and \mathbf{W_{v}} in the transformer: \mathbf{Q}=\mathbf{W_{q}}\mathbf{E}\text{, }\mathbf{K}=\mathbf{W_{K}}\mathbf{E}\text{, }\mathbf{V}=\mathbf{W_{v}}\mathbf{E}\vskip-3.61371pt At the layer level, each matrix multiplication is a fully connected layer with input and output neurons. Then its number of operations equals: 2\times S_{f}\times H^{2},\vskip-3.61371pt (1) where S_{f} is forward size and H is the model dimension. Figure 1: Transformer layer of OPT. The attention module includes the dot products of the query tokens in \mathbf{Q} with all the keys of previous tokens in \mathbf{K} to measure the similarity of previous tokens from the new token’s perspective: Similarity(\mathbf{Q,K})=\mathbf{QK}^{T} . The attention operation produces the output: Self-attention(\mathbf{Q,K,V})=softmax(\frac{\mathbf{QK}^{T}}{\sqrt{d_{k}}})\mathbf{V},\vskip-3.61371pt (2) where d_{k} is the dimension of the \mathbf{K} or \mathbf{Q} . The number of operations at the layer level equals: \vskip-0.72229pt4\times S_{f}^{2}\times H.\vskip-3.61371pt (3) The FC2 layer outputs logits, which are passed through the Add to generate the probabilities of words in the vocabulary. In the Transformer model, the above operations are executed by each attention head in parallel. Finally, the results from all attention heads are concatenated. (a) W/o prompt chunking. (b) W/ prompt chunking. Figure 2: Illustration of chunking a long prompt. 2.2 Theoretical Foundation Throughput . For the Transformer layer, the primary throughput contributors