DeepServe : Serverless Large Language Model Serving at Scale

Junhao Hu, Jiang Xu, Zhixia Liu, Yulong He, Yuetao Chen, Hao Xu, Jiang Liu, Jie Meng, Baoquan Zhang, Shining Wan, Gengyuan Dan, Zhiyu Dong, Zhihao Ren, Changhong Liu, Tao Xie, Dayun Lin, Qin Zhang, Yue Yu, Hao Feng, Xusheng Chen, Yizhou Shan · 2025

Read DeepServe : Serverless Large Language Model Serving at Scale by Junhao Hu, Jiang Xu, Zhixia Liu, Yulong He, Yuetao Chen, Hao Xu, Jiang Liu, Jie Meng, Baoquan Zhang, Shining Wan, Gengyuan Dan, Zhiyu Dong, Zhihao Ren, Changhong Liu, Tao Xie, Dayun Lin, Qin Zhang, Yue Yu, Hao Feng, Xusheng Chen, Yizhou Shan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14417.txt.

Introduction

Junhao Hu and co-authors describe DeepServe, a platform that runs large language models for many customers on shared cloud hardware, in a 2025 paper. Serving these models at scale strains resource allocation, efficiency, and start-up delays when new capacity spins up. DeepServe attacks them with a serverless design built around a request-job-task abstraction that handles fine-tuning, agent, and model-serving work in one frame.

Its engine, FlowServe, uses a microkernel-inspired layout with accelerator-centered execution and data-parallel splitting. Scheduling policies handle clusters that mix split and combined prefill-decode setups. Pre-warmed pods, memory pre-loading, and accelerator forking let it scale to sixty-four instances within seconds. The authors report the platform has run in production for over a year on an Ascend NPU cluster, offering standard interfaces. The paper details lessons from operating language models as a utility.

Catalog record

CREATORJunhao Hu, Jiang Xu, Zhixia Liu, Yulong He, Yuetao Chen, Hao Xu, Jiang Liu, Jie Meng, Baoquan Zhang, Shining Wan, Gengyuan Dan, Zhiyu Dong, Zhihao Ren, Changhong Liu, Tao Xie, Dayun Lin, Qin Zhang, Yue Yu, Hao Feng, Xusheng Chen, Yizhou Shan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14417

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DeepServe : Serverless Large Language Model Serving at Scale Junhao Hu, Jiang Xu, Zhixia Liu, Yulong He, Yuetao Chen, Hao Xu, Jiang Liu, Jie Meng, Baoquan Zhang, Shining Wan, Gengyuan Dan, Zhiyu Dong, Zhihao Ren, Changhong Liu, Tao Xie, Dayun Lin, Qin Zhang, Yue Yu, Hao Feng, Xusheng Chen, Yizhou Shan Abstract In this paper, we propose DeepServe , a scalable and serverless AI platform designed to efficiently serve large language models (LLMs) at scale in cloud environments. DeepServe addresses key challenges such as resource allocation, serving efficiency, and cold start latencies through four main design components. First, DeepServe uses a simple serverless abstraction called the request-job-task model, which helps manage diverse AI workloads across post-training and model-serving tasks. Second, DeepServe integrates an in-house serving engine named FlowServe using a microkernel-inspired design, NPU-centric execution, and SPMD-based parallelism to optimize LLM serving. Third, DeepServe includes novel scheduling policies tailored for a configuration with both PD-disaggregated and PD-colocated instances. Fourth, DeepServe includes optimizations such as pre-warmed pods, DRAM pre-loading, and NPU-fork, which allow DeepServe to scale up to 64 instances in seconds. DeepServe has been in production for over a year, operating on a large Ascend NPU cluster and providing industry-standard APIs for fine-tuning, agent serving, and model serving to our customers. 1 1 footnotetext: This work was completed during his internship at Huawei Cloud. 2 2 footnotetext: Co-corresponding authors. 1 Introduction The rapid rise of generative AI, exemplified by the success of platforms such as ChatGPT, has transformed the landscape of AI and increased the demand for scalable systems capable of serving large language models (LLMs). Model-as-a-Service (MaaS) platforms, such as OpenAI’s offerings, enable millions of users to access powerful AI capabilities through a simple API interface. Consequently, LLM serving has become one of the most crucial workloads in modern data centers. Ensuring such services’ performance, efficiency, and cost-effectiveness is complex, and we identify the following three challenges to achieve optimal performance and resource utilization while ensuring Service Level Objectives (SLOs) for multi-tenant users. First, AI workloads vary significantly in duration, from fine-tuning that can last hours or even days, to agent serving and LLM serving, which typically range from seconds to minutes. This variation makes it difficult to dynamically allocate shared resources without either underutilizing or overloading the system. Second, as LLM serving becomes increasingly distributed and stateful, managing resource allocation, synchronization, and fault tolerance across multiple instances is more complex. A single inference request may span multiple distributed instances and involve cached states, creating additional challenges in ensuring efficient operation. Third, the variability in LLM serving demands leads to fluctuating resource needs, further complicating resource optimization and the handling of cold-start latencies [ 10 ] . To address the preceding challenges, in this paper, we propose DeepServe , Huawei Cloud’s fully-hosted and serverless platform, offering industry-standard fine-tuning, agent serving, and model-serving APIs [ 31 ] . DeepServe has been running for over a year atop a large Ascend NPU cluster [ 24 ] . We report how DeepServe addresses the preceding challenges using four major design components: serverless abstraction and infrastructure, serving engine, scheduling algorithms, and scaling optimizations. Serverless Infrastructure. DeepServe introduces a developer-facing serverless abstraction called the request-job-task model. In this model, a request is an external trigger (e.g., a user-sent HTTP call). A job (or multiple jobs ) matching the request type (e.g., chat, fine-tuning) handles the request. A task is a fine-grained operation within a job (e.g., prefill task, decode task). Users send HTTP requests that trigger internal jobs , which are broken down into smaller tasks . This request-job-task abstraction allows for the dynamic scaling of workloads and efficient resource sharing across post-training and model-serving tasks. DeepServe ’s architecture, as shown in Figure 1 , is built around this abstraction and consists of three core components: Job Executors (JEs), Task Executors (TEs), and a cluster manager. JEs handle incoming requests and decompose them into manageable tasks that are then distributed to TEs for execution. The cluster manager ensures the health and scaling of the job and task executors. DeepServe deploys JEs and TEs dedicated to post-training and serving. We focus on serving in this paper. Serving Engine . In DeepServe , we design an efficient serving engine called FlowServe , built on three fundamental principles. The first is a microkernel-inspired design, which divides system functionality into modular components that can scale independently. This separation ensures that different system parts can evolve and operate asynchronously. The second principle is NPU-centric execution, which aims to keep the NPU busy to minimize delays caused by other resources such as CPUs, DRAM, or storage. Finally, FlowServe adopts a Single-Program-Multiple-Data (SPMD)-based design, enabling efficient parallel processing and scaling across multiple NPUs. The FlowServe engine has six core functionalities: tokenization, model execution, scheduler, memory management, caching management, and networking management. The tokenizer operates independently. Adhering to the SPMD design principle, FlowServe follows a master-executor architecture: the master oversees scheduling, caching, and networking decisions, while per-NPU executors carry out these decisions on their respective NPUs. FlowServe ’s scheduler is centralized at the master module, using both asynchronous KV cache prefetch and asynchronous execution to keep an NPU busy. FlowServe has a Relational Tensor Cache (RTC) module to manage the relationship between tensors (primarily on the KV cache) and a distributed flow (DistFlow) module to transfer tensors across tiered storage within a single engine or across engines in a peer-to-peer manner. Scheduling Algorithm. DeepServe presents three designs to tackle the challenges introduced by prefix caching and disaggregation. First, we implement a locality-aware algorithm to maximize KV cache reuse, a design shared by previous work [ 39 , 50 , 11 ] . Second, we extensively compare PD-disaggregated and PD-colocated TEs in a controlled environment with varying configurations. Based on the profiling results, we develop a PD-aware scheduling policy that accounts for the dynamics of online serving and decode-length uncertainty. Finally, we propose a combined scheduling algorithm that integrates load-aware, locality-aware, and PD-aware strategies. Scaling Optimization. DeepServe achieves fast scaling by quickly adjusting to fluctuating workloads. It employs multiple key techniques, including reserving pre-warmed Pods and TEs, pre-loading models into DRAM, and leveraging high-speed NPU-to-NPU links for efficient model loading. Combined with optimizations such as parallel initialization and predictive model pre-loading, these techniques significantly reduce initialization time. DeepServe can scale up to 64 instances in parallel within seconds. This paper makes the following main contributions: • Design of DeepServe , a large-scale serverless AI platform for LLM serving (§ 3 ). • Design of FlowServe , an efficient and modular serving engine architecture (§ 4 ). • Study of scheduling techniques for PD-disaggregated and PD-colocated setups (§ 5 ). • Detailed description of optimization techniques for fast scaling in LLM serving (§ 6 ). 2 Background We describe LLM and Ascend NPU chips in this section. LLM. LLM inference consists of two main stages: the prefill stage and the decode stage . In the prefill stage, the model processes the full input prompt (x_{1},x_{2},\dots,x_{n}) , computes Key-Value (KV) vectors for each token, and stores them in the KV cache [ 22 , 14 , 15 ] . The model then generates the next token x_{n+1} to initiate the decode stage. This stage is compute-bound

Read the complete text