Efficient and Workload-Aware LLM Serving via Runtime Layer Swapping and KV Cache Resizing
Zhaoyuan Su, Tingfeng Lan, Zirui Wang, Juncheng Yang, Yue Cheng · 2025
Read Efficient and Workload-Aware LLM Serving via Runtime Layer Swapping and KV Cache Resizing by Zhaoyuan Su, Tingfeng Lan, Zirui Wang, Juncheng Yang, Yue Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02006.txt.
Introduction
Zhaoyuan Su and co-authors present MorphServe, a system that reshapes a running language model to match the workload arriving at each moment. Serving these models is hard because demand arrives in bursts. Running full precision through a burst breaks response-time promises, while shrinking the model permanently costs accuracy even in quiet times.
MorphServe adapts on two fronts at once. During heavy load it swaps some layers of the model for smaller, quantized versions, choosing the layers whose loss hurts least. It also grows or shrinks the memory cache that holds conversation context, following the pressure of the moment. In tests on two model families with realistic traffic, broken response promises fell by over ninety percent. Reply speed at the slow tail improved two to four times over. Output quality held steady, which the authors offer as proof that elastic serving can be practical.
Catalog record
| CREATOR | Zhaoyuan Su, Tingfeng Lan, Zirui Wang, Juncheng Yang, Yue Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-02006
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient and Workload-Aware LLM Serving via Runtime Layer Swapping and KV Cache Resizing Zhaoyuan Su, Tingfeng Lan, Zirui Wang, Juncheng Yang, Yue Cheng Abstract Efficiently serving large language models (LLMs) under dynamic and bursty workloads remains a key challenge for real-world deployment. Existing serving frameworks and static model compression techniques fail to adapt to workload fluctuations, leading to either service-level objective (SLO) violations under full-precision serving or persistent accuracy degradation with static quantization. We present MorphServe , a dynamic, workload-aware LLM serving framework based on morphological adaptation . MorphServe introduces two asynchronous, token-level runtime mechanisms: quantized layer swapping , which selectively replaces less impactful layers with quantized alternatives during high-load periods, and pressure-aware KV cache resizing , which dynamically adjusts KV cache capacity in response to memory pressure. These mechanisms enable state-preserving transitions with minimum runtime overhead and are fully compatible with modern scheduling and attention techniques. Extensive experiments on Vicuna and Llama family models with real-world workloads demonstrate that MorphServe reduces average SLO violations by 92.45% and improves the P95 TTFT latency by 2.2 × 2.2\times – 3.9 × 3.9\times compared to full-precision serving, without compromising generation quality. These results establish MorphServe as a practical and elastic solution for LLM deployment in dynamic environments. 1 Introduction The rise of large language models (LLMs) has made efficient and reliable serving a core challenge in modern AI infrastructure. Systems like vLLM [ 33 ] and TGI [ 29 ] optimize throughput via PagedAttention [ 33 ] and continuous batching [ 75 , 62 , 24 ] , but assume fixed-precision execution and stable workloads. In contrast, real-world LLM workloads are dynamic and bursty [ 67 , 4 ] , with fluctuating request rates and context lengths. Even brief load spikes can cause memory exhaustion or queueing delays, leading to SLO violations—e.g., higher time-to-first-token (TTFT) and time-per-output-token (TOPT)—that degrade user experience and system throughput. One naive solution is to statically over-provision GPU resources to accommodate worst-case traffic spikes. However, over-provisioning leads to substantial cost inefficiencies during underutilized periods [ 30 , 19 ] . Moreover, edge deployments lack the flexibility for dynamic scaling altogether [ 6 ] . Thus, the inability to elastically match model resource usage to real-time demand results in either SLO violations under pressure, or significant resource waste during low-load intervals. Model compression techniques, such as quantization [ 38 , 17 , 37 , 57 ] , pruning [ 43 , 59 , 21 ] , or low-rank approximation [ 26 , 70 ] , offer an alternative approach by statically reducing the resource footprint of deployed LLMs. While these methods are effective in lowering memory and compute demands, they introduce irreversible accuracy degradation that persists even during periods of low load, when full-precision inference could be served without penalty. This results in a rigid, suboptimal quality–efficiency tradeoff that fails to align with workload variability. Key-value cache (KVC) compression [ 7 , 78 , 35 ] and eviction [ 41 , 16 ] methods have been proposed to further reduce memory usage. However, these techniques often rely on fixed heuristics, cannot adapt to different workloads, lack compatibility with modern attention variants like Grouped Query Attention (GQA) [ 3 , 9 , 10 ] and Multi-Head Latent Attention (MLA) [ 45 , 40 ] , and remain inflexible to runtime serving conditions. In this paper, we present MorphServe , a dynamic, workload-aware LLM serving framework based on morphological adaptation. MorphServe continuously monitors system load and morphs model components—transformer layers and KVC blocks—on the fly in response to real-time memory pressure. When resource usage surges, MorphServe reduces model footprint by replacing selected full-precision layers with lightweight quantized alternatives and expands KVC capacity by dynamically attaching additional memory blocks. These adaptations are reversed as pressure subsides, restoring full precision and reclaiming memory from KVC without interrupting inference. MorphServe contributes the following: (1) A runtime layer swapping mechanism that enables workload-aware mixed-precision serving , allowing quantized and full-precision layers to coexist and be dynamically reconfigured based on runtime pressure without model flushing or architectural changes. (2) A pressure-aware KVC resizing mechanism that elastically adjusts KV cache capacity, supporting efficient batch prefilling and decoding under bursty traffic. (3) A tunable runtime policy that navigates the accuracy–latency Pareto frontier , balancing high-fidelity and low-latency objectives. (4) Full compatibility with existing KVC compression and eviction schemes, enabling further efficiency gains with minimal accuracy degradation. To achieve this, MorphServe introduces two complementary morphing mechanisms, both designed to support asynchronous and compatible kernel executions with minimal overhead: LayerSwapper identify low-impact transformer layers by a sensitivity-based profiling, selectively and asynchronously replacing them with lower-precision alternatives at runtime. KVResizer adaptively adjusts KVC capacity under memory pressure and runs in parallel with decoding using separate CUDA streams, ensuring seamless execution. Across extensive experiments on Llama 2 [ 64 ] , Llama 3 [ 22 ] , CodeLlama [ 56 ] , and Vicuna [ 63 ] using four datasets [ 27 , 77 , 23 , 15 ] under Azure LLM Inference [ 4 ] and BurstGPT [ 67 ] traces, MorphServe reduces average SLO violations by 92.45% and P95 TTFT latency by 2.2 \times –3.9 \times over full-precision serving, while preserving comparable accuracy. Compared to static quantization via AWQ [ 38 ] , MorphServe reduces F1 and Rouge-L degradation by up to 88.85% and improves memory utilization by 29.29%. These results demonstrate MorphServe ’s ability to adapt to dynamic workloads while balancing performance and responsiveness. 2 Background and Motivation (a) (b) (c) (d) Figure 1 : Motivation for dynamic adaptation design in LLM serving. (a) Real-world LLM workloads are highly dynamic and bursty in request and token volume. (b) Full-precision serving suffers TTFT spikes and SLO violations when workload exceeds the saturation point. (c) Statically quantized model causes constant accuracy degradation even during low-load periods when it is possible to serve full-precision models. (d) MorphServe dynamically adapts to resource pressure and consistently achieves an optimal balance between SLO compliance and accuracy. Real-world LLM workloads are highly bursty. LLM serving systems face highly dynamic and bursty traffic patterns in real-world scenarios. As shown in Figure 1(a) , the production workloads of Microsoft Azure LLM services [ 4 , 61 ] and BurstGPT [ 67 ] reveal rapid fluctuations in both the request arrival rates (i.e., request bursts) and the volumes of tokens. These fluctuations reflect the non-stationary nature of practical LLM inference workloads, which deviates from the traditional assumptions of most serving schemes [ 33 , 75 , 30 ] . Request burst leads to long TTFT and SLO violation . As system load increases, even small surges can cause sharp spikes in time-to-first-token (TTFT) latency. In this work, we set the TTFT SLO threshold to 2 seconds, consistent with prior work [ 72 , 20 , 53 ] . As shown in Figure 1(b) , full-precision serving quickly exceeds the SLO threshold once it reaches the saturation point — defined as the load level at which available GPU memory becomes insufficient to schedule new requests for prefilling or to continue decoding for the ongoing batch . At this point, incoming requests are forced to wait until memory is reclaimed, incurring significant queueing latency with SLO violation . Static quantization trades