SLO-Aware Scheduling for Large Language Model Inferences Conference: The International Conference for High Performance Computing, Networking, Storage, and Analysis; Nov 16-Nov 21, 2025; St. louis, Missouri

Jinqi Huang, Yi Xiong, Xuebing Yu, Wenjie Huang, Entong Li, Li Zeng, Xin Chen · 2025

Read SLO-Aware Scheduling for Large Language Model Inferences Conference: The International Conference for High Performance Computing, Networking, Storage, and Analysis; Nov 16-Nov 21, 2025; St. louis, Missouri by Jinqi Huang, Yi Xiong, Xuebing Yu, Wenjie Huang, Entong Li, Li Zeng, Xin Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14966.txt.

Introduction

Huang and six co-authors plan request queues for language model hosts by each ask's time pledge in this twenty twenty-five paper. Hosts field mixed asks, from code help to chat, each with its own speed bar. Past queue plans gave all asks one bar, which wastes gear and misses marks.

Inside, a heat-style hunt sets the queue rank from pledge, in length, and out length. That rank aims to lift pledge hits while it cuts the mean wait per ask. Tests on two chat and code sets beat top host frames by wide gaps. Pledge hits rose up to five-fold and mean waits fell by near a third. The work is the first queue plan built for mixed pledges.

Catalog record

CREATORJinqi Huang, Yi Xiong, Xuebing Yu, Wenjie Huang, Entong Li, Li Zeng, Xin Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14966

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SLO-Aware Scheduling for Large Language Model Inferences Conference: The International Conference for High Performance Computing, Networking, Storage, and Analysis; Nov 16-Nov 21, 2025; St. louis, Missouri Jinqi Huang, Yi Xiong, Xuebing Yu, Wenjie Huang, Entong Li, Li Zeng, Xin Chen Abstract . Large language models (LLMs) have revolutionized applications such as code completion, chatbots, and online classification. To elevate user experiences, service level objectives (SLOs) serve as crucial benchmarks for assessing inference services capabilities. In practice, an inference service processes multiple types of tasks, each with its own distinct SLO. To ensure satisfactory user experiences, each request’s distinct SLOs should be considered in scheduling. However, existing designs lack this consideration, leading to insufficient hardware utility and suboptimal performance. This paper analyzes scenarios to process tasks with varying SLOs, and introduces a simulated annealing-based scheduler to decide request priority sequence based on a request’s SLO, input lengths, and possible output lengths. As the first specialized scheduler for multi-SLO scenarios, this work improves SLO attainment by up to 5 × \times and reduces average latency by 31.6% on Python-Code-23k-ShareGPT and ShareGPT_Vicuna_unfiltered datasets, compared to current state-of-the-art framework vLLM and a new framework LMDeploy. 1. Introduction The rapid revolution of LLMs has been witnessed since the debut of ChatGPT ( AI 2022 ) , with a succession of LLMs being unveiled, including Llama ( Meta 2024 ) by Meta, Gemini ( Sundar Pichai 2024 ) by Google, and Qwen ( Qwen et al. 2025 ) by Alibaba. Since then, LLMs have gained extensive adoption across a multitude of industry applications such as fault maintainence, customer service, recommendation system, code completion ( Tang et al. 2023 ; Chen et al. 2023 ) , question answering ( Arefeen et al. 2024 ; Koohborfardhaghighi et al. 2024 ) , knowledge extraction ( Dagdelen et al. 2024 ; Duranti et al. 2025 ) , document summarization ( Liu et al. 2024b ; Van Veen et al. 2023 ) , search engines ( Liu et al. 2024a ; Chowdhury and Chowdhury 0 ) , etc. Figure 1. Illustration of multiple SLOs existing in LLM inference: (A) distinct SLOs for multiple sub-tasks within an application; (B) diverse SLOs for applications sharing resources; (C) multiple SLOs introduced by scenarios (A) and (B), with ‘/’ indicates unimportance for this task. LLMs are integrated into inference serving systems to provide services. These systems, such as vLLM ( Kwon et al. 2023 ) , fastTransformer ( Nvidia 2024a ) , and Triton ( Nvidia 2024b ) , queue, schedule, and batch the incoming queries before feeding them into LLMs. SLOs are introduced to evaluate if a LLM inference service deliver a satisfactory user experience due to LLM services’ interactive nature. As the result of diverse user experiences across tasks, SLOs typically vary by applications. For an instance, offline processing emphasizes throughput over latency to reduce costs, while online classifiers require real-time responses, making the end-to-end (e2e) latency critical. For chatbots, focusing on time-to-first-token (TTFT) and time-per-output-token (TPOT) ensures quick initial responses and subsequent token generation with human-like reading speed. In practical industry applications (e.g., fault maintenance, customer service, or recommendation system), a set of LLM inference services are employed for various tasks, each with its own SLO. Figures A and B in 1 show examples of varying SLO requirements for different tasks. In Scenario 1, as depicted in Figure 1 (A), an application may involve multiple subtasks, each with varying input and output lengths, request concurrency and specific SLOs. Similarly, as shown in Figure 1 (B), Scenario 2 presents a case where multiple applications with diverse SLOs utilize a shared resource pool, which encompasses more than one LLM inference service. Figure 1 (C) gives an example of multiple SLOs introduced by Scenario 1 or 2. To fully utilize the hardware resources and to fulfill application SLOs as much as possible, it is important to account for the varying SLOs across different applications. Mainstream inference services, including vLLMs ( Kwon et al. 2023 ) , Triton ( Nvidia 2024b ) , and fastTransformer ( Nvidia 2024a ) , focus on high-throughput low-latency scheduling with first-come-first-serve (FCFS) policies. Besides, other high-performance schedulers enhance throughput with specialized techniques: Sarathi-Serve ( Agrawal et al. 2024 ) introduces chunked prefill to avoid stalls during transitions between prefill and decode phases; FastServe ( Wu et al. 2024b ) implements skip-join multi-level feedback queues and iteration-level preemption to mitigate head-of-line blocking caused by longer sequences processed before shorter ones; Splitwise ( Patel et al. 2024 ) and DistServe ( Zhong et al. 2024 ) isolate prefill and decode phases into distinct inference instances and optimizing them separately; Llumnix ( Sun et al. 2024 ) leverages load balancing, prioritization, de-fragmentation, and auto-scaling scheduling strategies to achieve high throughput. All these existing designs are tailored to scenarios where queries originate from a single task or from various tasks with identical SLOs. When immigrated to multi-SLO scenarios, these existing systems lack awareness of the unique SLOs associated with different tasks, leading to suboptimal hardware utilization and SLO attainment. To address the complexity of multi-SLO scheduling, this paper propose a SLO-aware priority mapping algorithm. Aiming at optimizing the system SLO attainment and minimizing the average latency, we optimize each request’s priority and the actual batch size, based on task-specific SLOs, input lengths, and output lengths prediction. To mitigate substantial searching overhead for the optimal priority sequence and request batch sizes, we employ a simulated annealing-based SLO-aware scheduling approach with an overhead of 1ms, while maintaining a similar effectiveness to an exhaustive search-based method. To the best of our knowledge, this work is the first system designed specifically to address SLO-aware scheduling challenges. To evaluate the system effectiveness, we benchmark the performance against the state-of-the-art framework vLLM and a newly-introduced framework LMDeploy ( Contributors 2023 ) with ShareGPT_Vicuna_unfiltered ( teams 2023 ) and Python-Code-23k-ShareGPT ( HuggingFace 2025a ) datasets, and prove that our system boosts the SLO attainment by up to 5 \times and reduces the average latency by up to 31.6%, respectively. The contributions of this paper are summarized below: • We propose a priority mapping algorithm for prioritizing each request based on its SLO, input length, and expected output length if available, to achieve highest SLO attainment with average latency as low as possible. • We develop the first specialized simulated annealing-based SLO-aware scheduler with low overhead to tackle the complexities of SLO-aware scheduling. • We demonstrate the scheduling system on Python-Code-23k-ShareGPT and ShareGPT_Vicuna_unfiltered datasets and prove that our method outperforms vLLM and LMDeploy in SLO attainment by up to 5 \times and reduces average latency by up to 31.6%. 2. Background Figure 2. Prefill and decode phases in LLM inferences. KV cache is dynamically stored, loaded, and updated during the inference. 2.1. Preliminaries LLMs evolve from transformers ( Vaswani et al. 2017 ) with an attention-based architecture. Unlike their deep neural networks (DNNs) predecessors, transformer-based LLMs follow the principle of ‘next token prediction’ to construct output sequences: LLM inference generates one token per iteration based on the user prompt and previous tokens. This working principle forms the characteristic of variable and unpredictable output lengths in LLM. Another important feature of LLMs is the significant diversity between prefill and decode phases. As shown

Read the complete text