SLICE: S LO-Driven Scheduling for L LM I nferen c e on E dge Computing Devices

Pan Zhou 1 , Yiming Lei 1 , Ling Liu 1 , Xiaoqiong Xu 1 , Ying Cai 1 , Daji Ergu 1 , Hongfang Yu 2 , Yueyue Dai 3 · 2025

Read SLICE: S LO-Driven Scheduling for L LM I nferen c e on E dge Computing Devices by Pan Zhou 1 , Yiming Lei 1 , Ling Liu 1 , Xiaoqiong Xu 1 , Ying Cai 1 , Daji Ergu 1 , Hongfang Yu 2 , Yueyue Dai 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18544.txt.

Introduction

Will Chow, writing alone, presents a scheduler called SLICE. It runs large language models on edge devices. Such devices include robots and smart vehicles. Each app sets its own service goals. One goal is how fast the first token must appear. Another is how steady later tokens must flow. Old schedulers chase raw output speed alone. Deadline-style goals then get missed a lot. Real-time edge tasks suffer most from those misses.

SLICE pairs two tools. One schedules requests to raise a utility score. The other is a control loop. It adjusts output rates as conditions change. Tests pitted SLICE against Orca and FastServe. SLICE beat both by wide margins on goal attainment. In some tests it reached thirty-five times their score. It also finished tasks several times faster. Chow argues edge inference should serve stated goals. Raw throughput alone is the wrong target.

Catalog record

CREATORPan Zhou 1 , Yiming Lei 1 , Ling Liu 1 , Xiaoqiong Xu 1 , Ying Cai 1 , Daji Ergu 1 , Hongfang Yu 2 , Yueyue Dai 3
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18544

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SLICE: S LO-Driven Scheduling for L LM I nferen c e on E dge Computing Devices Corresponding E-mail: [email protected] Pan Zhou 1 , Yiming Lei 1 , Ling Liu 1 , Xiaoqiong Xu 1 , Ying Cai 1 , Daji Ergu 1 , Hongfang Yu 2 , Yueyue Dai 3 Abstract Large Language Models (LLMs), as the foundational architecture for next-generation interactive AI applications, not only power intelligent dialogue systems but also drive the evolution of embodied intelligence on edge devices, including humanoid robots, smart vehicles, and other scenarios. The applications running on these edge devices impose differentiated Service Level Objectives (SLO) requirements on LLM services, specifically manifested as distinct constraints on Time to First Token (TTFT) and Time Per Output Token (TPOT) as well as end-to-end latency. Notably, edge devices typically handle real-time tasks that are extremely sensitive to latency, such as machine control and navigation planning. However, existing scheduling service systems still prioritize maximizing output token throughput as the sole optimization objective, failing to adequately address the diversity of SLO requirements. This ultimately results in persistently high violation rates for end-to-end latency or TPOT related SLOs. This paper proposes SLICE, an innovative scheduling solution designed for edge computing scenarios with differentiated SLO requirements. By combining a utility-maximizing request scheduling algorithm with a dynamic iterative control mechanism for generation rates, SLICE significantly improves LLM inference service SLO attainment. Experimental results demonstrate that compared to state-of-the-art solutions Orca and FastServe, SLICE achieves up to 35× higher SLO attainment and 3.4× advantage in task completion time than the other two solutions. I Introduction Large language models (LLMs) such as GPT and DeepSeek series [ 1 ] [ 2 ] are raising artificial general intelligence to an unprecedented heights. With their human-like generation capabilities, these models are being rapidly adopted across diverse domains, most notably in edge intelligence. This development has sparked new possibilities for intelligent edge devices and is poised to fundamentally transform lifestyles and work patterns. A prominent example is Google’s SayCan [ 3 ] , which significantly enhances robots’ ability to comprehend and execute linguistic commands. Leading technology companies including Boston Dynamics [ 4 ] , Tesla [ 5 ] , and Apple [ 6 ] are actively developing and implementing LLM technologies in their products. As a result, integrating the vast knowledge capabilities of large language models into edge devices—spanning smartphones, IoT systems, and automotive platforms—represents a transformative opportunity in modern edge computing ecosystems [ 7 ] [ 8 ] [ 9 ] . (a) Decode latency varing with batch size (b) Token throughput varing with batch size Fig. 1: We conducted a batch processing performance evaluation of the ChatGLM2-6B model on an NVIDIA RTX 4060 Ti 16GB GPU. Experimental results demonstrate that as the batch size increases from 1 to 9, the per-token latency shows near-linear growth while throughput improves only marginally. When batch size exceeds 9, although the per-token latency stabilizes and throughput continues to scale linearly with batch size, the absolute latency spikes above 120ms, reducing the token generation rate per task to below 10 tokens/s. This performance bottleneck will significantly impact latency-sensitive real-time tasks, making it impossible to meet strict SLO. Deploying large language models on edge devices requires supporting diversified application services, where different applications exhibit significant variations in service-level objectives (SLOs) for inference performance. Real-time applications like navigation planning and device control demand strict end-to-end deadline constraints [ 10 , 11 , 12 ] . Non-real-time interactive applications focus on Time To First Token (TTFT) and Time Per Output Token (TPOT) [ 13 , 14 , 15 ] . For example, voice interaction must guarantee output speed not less than human speech rate [ 16 ] , while text-based question answering requires matching reading speed to maintain interaction fluency [ 17 ] [ 18 ] . Given the constrained hardware resources of edge devices and the large scale of LLMs, even a single-unit increase in batch size can lead to significant single-token latency spikes (as illustrated in Fig. 1 ). This tension between resource constraints and multi-SLO requirements places scheduling system design as the key enabler for efficient LLM inference in edge computing scenarios. Mainstream inference services such as Orca [ 19 ] , vLLM [ 20 ] , FastLLM [ 21 ] , and FasterTransformer [ 22 ] predominantly employ First-Come-First-Served (FCFS) scheduling to compose a processing batch, which may cause queuing delay for later-arriving tasks under ultra heavy workloads in cloud computing scieneros. Meanwhile, alternative high-performance schedulers adopt specialized techniques to alleviate this queuing delay. Sarathi Serve [ 23 ] implements chunked pre-filling to eliminate stalls between prefill and decode phase transitions. FastServe [ 24 ] deploys multi-level feedback queues with iteration-level preemption to mitigate head-of-line blocking, while Splitwise/DistServe [ 25 ] isolates pre-filling and decoding stages for independent optimization. Llumnix [ 26 ] further enhances throughput through load balancing, prioritization, defragmentation, and auto-scaling. All these designs indiscriminately batch tasks regardless of their SLO requirements during inference. During each decoding iteration, all scheduled tasks are indiscriminately batched together and processed through the LLM’s decoding forward pass. This coarse-grained decoding scheduling works well for homogeneous SLO scenarios without strict deadline constraints or TPOT requirements, while our experiments reveal its inadequate when handling edge computing workloads with various strict SLO requirements, as illustrated in Fig. 2a . This experimental case demonstrates that existing decoding scheduling can cause a long decoding latency that exceeds TPOT requirements when the batch size surpasses the critical threshold, resulting in severe SLO violations. More critically, if the deadline of real-time tasks cannot be guaranteed, it will directly impair the normal operation of edge devices, such as those performing delay-sensitive and mission-critical tasks like machine control and navigation planning, potentially leading to system malfunctions or even task failures. Motivated by the above insights, can we design an intelligent scheduling system that dynamically adjusts decoding rates according to tasks’ SLO requirements, thereby improving overall SLO attainment while ensuring effective guarantees for real-time tasks’ deadline in edge computing scenarios? (a) Existing coarse-grained decoding scheduling (b) SLO-aware decoding scheduling Fig. 2: (a) The current scheduling system cannot perform fine-grained, decode-level scheduling. Each decoding process batches all scheduled tasks together, which increases decoding latency and consequently leads to violations of task-level SLOs. (b) By adopting decode-level batch scheduling and implementing an on-demand allocation approach, more tasks can meet their SLO requirements. In this paper, we propose a SLO-driven scheduling strategy called SLICE for LLM inference services. The core concept of SLICE revolves around ”real-time task prioritization and on-demand resource allocation”. Specifically, SLICE gives priority to scheduling real-time tasks to ensure they meet their deadlines. Additionally, it allocates decoding rates in accordance with SLO requirements. This approach prevents low-demand tasks from excessively consuming computational resources, thereby enabling more concurrent tasks to run even with limited computing power (as shown in Fig. 2b ). By adhering to this principle, SLICE achieves a synergistic improvement in both SLO attainment and the successful completion of real-time tasks within their deadlines. To implement this scheduling concept, SLICE employs a two-stage scheduling strategy. In the first phase (i.e., the task selection phase), each newly arrived task is assigned a utility value that reflects its importance. Notably, real-time tasks

Read the complete text