Ascendra: Dynamic Request Prioritization for Efficient LLM Serving

Azam Ikram, Xiang Li, Sameh Elnikety, Saurabh Bagchi · 2025

Read Ascendra: Dynamic Request Prioritization for Efficient LLM Serving by Azam Ikram, Xiang Li, Sameh Elnikety, Saurabh Bagchi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20828.txt.

Introduction

Azam Ikram, Xiang Li, Sameh Elnikety, and Saurabh Bagchi present a serving system that keeps large language model replies fast and steady, in work from twenty twenty-five. Hosts must meet two goals at once: a quick first word and a steady stream after. Most systems trade one goal for the other.

Their system, called Ascendra, splits hardware into low and high priority groups. The low group runs requests out of order for max output, with a risk of starvation. A model spots requests near their deadline and moves them to the high group, which is tuned for low delay. Tests beat two known serving systems on throughput by up to about seventy parts in a hundred. Both speed goals held at the same time.

Catalog record

CREATORAzam Ikram, Xiang Li, Sameh Elnikety, Saurabh Bagchi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20828

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ascendra: Dynamic Request Prioritization for Efficient LLM Serving Azam Ikram, Xiang Li, Sameh Elnikety, Saurabh Bagchi Abstract . The rapid advancement of Large Language Models (LLMs) has driven the need for more efficient serving strategies. In this context, efficiency refers to the proportion of requests that meet their Service Level Objectives (SLOs), particularly for Time To First Token (TTFT) and Time Between Tokens (TBT). However, existing systems often prioritize one metric at the cost of the other. We present Ascendra, an LLM serving system designed to meet both TTFT and TBT SLOs simultaneously. The core insight behind Ascendra is that a request’s urgency evolves as it approaches its deadline. To leverage this, Ascendra partitions GPU resources into two types of instances: low-priority and high-priority. Low-priority instances maximize throughput by processing requests out of arrival order, but at the risk of request starvation. To address this, Ascendra employs a performance model to predict requests at risk of missing their SLOs and proactively offloads them to high-priority instances. High-priority instances are optimized for low-latency execution and handle urgent requests nearing their deadlines. This partitioned architecture enables Ascendra to effectively balance high throughput and low latency. Extensive evaluation shows that Ascendra improves system throughput by up to 1.7× compared to vLLM and Sarathi-Serve while meeting both TTFT and TBT SLOs. 1. Introduction In recent years, large language models (LLMs) have become integral to various applications—from answering user queries to aiding software development. As the demand for LLM-powered services continues to grow, companies are racing to integrate LLM capabilities into their platforms. On the research front, significant effort has been devoted to making LLM serving more efficient. Recent work has focused on improving execution speed ( Dao et al., 2022 ; Hong et al., 2024 ) , optimizing I/O access patterns ( Kwon et al., 2023 ) , and ensuring fairness across concurrent requests ( Sheng et al., 2024 ) . In this paper, we study the problem of efficient LLM serving and propose a system that achieves high throughput while respecting latency service-level objectives (SLOs), without relying on expensive network interconnects. (a) Ascendra offloads urgent requests to high-priority (HP) instances to prevent SLO violations, while baselines like vLLM strictly follow arrival order and result in SLO violation. (b) Goodput and normalized throughput when serving Mistral-7B on the ShareGPT4 dataset using three NVIDIA A100-80G GPUs. Ascendra consistently maintains higher SLO attainment and lower latency compared to existing baselines, even at higher QPS. Figure 1 . Top : Out-of-order scheduling and proactive offloading of Ascendra avoid SLO violations. Bottom : Ascendra improves SLO attainment and latency compared to baselines. LLM inference typically consists of two phases: prefill and decode. In the prefill phase, the model processes the input prompt and generates the first token; the latency from request arrival to this token is referred to as Time-to-First-Token (TTFT). In the decode phase, the model autoregressively generates subsequent tokens, with Time-Between-Tokens (TBT) measuring the latency between two consecutive token generations. Real-world applications often impose strict SLOs on both TTFT and TBT. A serving system’s effectiveness is evaluated by its goodput —the fraction of requests meeting both SLOs. Several prior works have explored the trade-off between optimizing TTFT and TBT. For instance, vLLM ( Kwon et al., 2023 ) prioritizes prefill to reduce TTFT, at the cost of increased chance of generation stalls. More recently, Sarathi-Serve ( Agrawal et al., 2024 ) introduced chunked prefill, which improves TBT by interleaving decoding with prefill but suffers from higher TTFT due to frequent I/O access. Another approach, DistServe ( Zhong et al., 2024 ) , decouples prefill and decode by running them on separate GPUs. While simple, this disaggregation leads to resource underutilization - prefill is compute-heavy but memory-light, and decode is compute-light but memory-heavy - and requires expensive hardware (e.g., NVLink, NVSwitch, InfiniBand) to transfer large amounts of data (KV caches) between GPUs and nodes. In this work, we propose a hybrid approach that achieves high goodput without relying on expensive network interconnects. We make three key observations: (1) the delay a request can tolerate before violating its TTFT SLO depends on the prompt length and how long the request has been in the system, (2) First-Come-First-Serve (FCFS) scheduling causes head-of-line blocking, and (3) continuous execution of decoding requests is critical for meeting TBT targets. Based on these insights, we design Ascendra, a system that: (1) assigns dynamically evolving priorities to requests in system based on their proximity to violating the TTFT SLO, (2) enables out-of-order execution to avoid head-of-line blocking, using the current priority as the scheduling policy, and (3) utilizes two distinct types of instances—each optimized for either throughput or latency—to specialize execution based on workload characteristics. This separation enables workload-aware resource allocation and better SLO adherence. To illustrate the impact of these insights, consider an example involving five requests and a TTFT SLO of five time units as shown in 1(a) . Under vLLM, the first four requests arrive at time t=0 and are distributed round-robin across two GPUs. Because vLLM processes requests in strict FCFS order, a fifth request (E) arriving at t=1 must wait until all earlier requests have completed, incurring a four-unit delay before starting execution. By the time request E begins its prefill, it has only one time unit remaining to meet its TTFT SLO. However, as the prefill of E is expected to take two time units, E will inevitably violates its TTFT target. In contrast, consider the same scenario under Ascendra, GPU resources are partitioned into Low Priority (LP) and High Priority (HP) instances. The HP instance accepts a new request only if its queue is empty; otherwise, requests are routed to LP instances. LP instances use an internal value function to determine execution order, optimizing throughput rather than strictly following arrival order. At t=1 , request E is forwarded to the idle HP instance. At t=3 , the LP instance holds requests C and D, and chooses to execute D while offloading C to HP, as C is projected to miss its TTFT SLO if delayed. This proactive offloading ensures urgent requests are handled promptly, while LP instances maintain high throughput. We validate this behavior in Figure 1 , showing that Ascendra improves both throughput and goodput by flexibly managing latency-sensitive requests. In this paper, we make the following contributions: (1) We identify key challenges in existing LLM serving systems, highlighting inefficiencies in handling latency-sensitive workloads in FCFS fashion. We provide an in-depth analysis of the TTFT-TBT trade-off. (2) We design and implement a hybrid serving architecture that partitions GPU resources into specialized instance types. We introduce a priority-aware scheduling policy that dynamically assigns request priorities at batch-level granularity. This design significantly improves goodput by ensuring timely service for urgent requests while maintaining high system throughput. (3) We extensively evaluate our system on different models and data sets to show that Ascendra improves goodput compared to state-of-the-art baselines, without requiring additional hardware resources. 2. Background 2.1. Phases in LLM Inference LLM serving typically involves two distinct phases: the prefill phase and the decode phase . Upon receiving a request, the model performs a single forward pass over the entire input prompt, generating a Key-Value (KV) cache and producing

Read the complete text