OrbitFlow : SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration

Xinyue Ma, Heelim Hong, Taegeon Um, Jongseop Lee, Seoyeong Choy, Woo-Yeon Lee, Myeongjae Jeon · 2026

Read OrbitFlow : SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration by Xinyue Ma, Heelim Hong, Taegeon Um, Jongseop Lee, Seoyeong Choy, Woo-Yeon Lee, Myeongjae Jeon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10729.txt.

Introduction

Xinyue Ma and co-authors present OrbitFlow, a serving system for long-context language models. Serving such models is tricky because request lengths and batch mixes shift while text is generated. Memory use swings with them. Moving key-value caches to host memory helps, but fixed offloading plans cannot keep up with the swings.

OrbitFlow reconfigures the cache at a fine grain while watching service-level goals. The aim is to meet response targets without wasting scarce accelerator memory. Static plans either hoard memory or stall requests. An adaptive plan can do better on both counts. The work speaks to data centers that run long-document chat and analysis under tight latency promises.

Catalog record

CREATORXinyue Ma, Heelim Hong, Taegeon Um, Jongseop Lee, Seoyeong Choy, Woo-Yeon Lee, Myeongjae Jeon
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-10729

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OrbitFlow : SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration Xinyue Ma, Heelim Hong, Taegeon Um, Jongseop Lee, Seoyeong Choy, Woo-Yeon Lee, Myeongjae Jeon Abstract . Serving long-context LLMs is challenging because request lengths and batch composition vary during token generation, causing the memory footprint to fluctuate significantly at runtime. Offloading KV caches to host memory limits effective memory usage, but existing static and predetermined offloading strategies cannot adapt to the rapidly shifting memory demands of long-context serving. This often leads to excessive CPU-to-GPU KV transfers that translate into latency spikes and frequent SLO violations. To address these challenges, we introduce OrbitFlow , a fine-grained and adaptive KV cache management system that meets latency SLOs in long-context LLM serving. OrbitFlow employs a lightweight ILP solver to decide which layers’ KV caches to retain on the GPU for each request, within memory capacity constraints. It continuously refines KV placements based on runtime feedback when the active plan becomes suboptimal during token generation. Under heavy load, OrbitFlow invokes a fallback mechanism to temporarily defer in-flight requests with large memory footprints , preserving overall SLO attainment. Our experiments demonstrate that OrbitFlow improves SLO attainment for TPOT and TBT by up to 66% and 48%, respectively, while reducing the 95th percentile latency by 38% and achieving up to 3.3 × 3.3\times higher throughput compared to existing offloading methods. † † authors: . PVLDB Reference Format: PVLDB, 19(1): XXX-XXX, 2025. doi:XX.XX/XXX.XX † † This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing [email protected] . Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 19, No. 1 ISSN 2150-8097. doi:XX.XX/XXX.XX PVLDB Artifact Availability: The source code, data, and/or other artifacts have been made available at https://github.com/Heelim-Hong/ORBITFLOW . 1. Introduction Large language models (LLMs) have gained significant prominence across various natural language processing tasks, including text generation ( 11 ; 7 ; 16 ; 9 ; 5 ; 27 ) , language translation ( 53 ; 31 ; 34 ) , summarization ( 38 ; 25 ) , and question answering ( 41 ; 21 ; 15 ) . Their capabilities have greatly improved over the past few years, largely empowered by dramatic increases in context window size—i.e., the maximum number of tokens in words or subwords a model can process at once. For instance, GPT-2 ( 42 ) supported a context window of 1K tokens, while its successor GPT-3 ( 7 ) doubled the window size to 2K tokens. More recent models such as GPT-4 ( 26 ) and Claude ( 3 ) have pushed the limits further, handling up to 128K or even 1M tokens. To provide the best user experience, LLM serving systems must respond to requests within stringent latency requirements defined by service-level objectives (SLOs) ( 56 ; 40 ; 1 ) . Upon receiving an input sequence of tokens, these systems generate output tokens autoregressively, one token at a time, through consecutive decode steps . Many LLM services, such as chat agents, creative writing tools, and real-time translators, stream tokens as they are generated so that users can read responses incrementally. To keep decode latency perceptually invisible, these services typically set per-token latency SLOs (e.g., 200–300 ms for chatbots) to match natural reading speeds ( 36 ; 56 ) . Frequent violations of these latency SLOs can impair user experience and reduce revenue ( 4 ) , regardless of response quality. As context windows expand and workloads become more decode-dominant, optimizing for per-token latency SLOs becomes ever more critical. However, serving long-context LLMs within strict time bounds is challenging because GPU memory demands vary widely during runtime. Modern serving systems accelerate token generation by performing decode steps based on cached states of all preceding input-output tokens, termed KV cache . Rapid decoding thus requires storing all KV caches in GPU memory, which was manageable with small context windows. Such caching has now become nontrivial for two reasons: (1) although initially small, the KV cache grows steadily during lengthy decoding, creating a high likelihood of exhausting GPU memory (i.e., token dimension ); and (2) LLM serving systems often dynamically batch multiple requests to process them concurrently on the GPU ( 28 ; 19 ) , further straining the available GPU memory (i.e., batch dimension ). There are thriving efforts to reduce KV cache size itself during the decode step. For instance, state restoration ( 45 ) recomputes parts of the KV cache at runtime , and token pruning ( 29 ; 47 ; 49 ; 54 ; 30 ) discards presumably less critical KV entries . While effective, these approaches carry inherent limitations, they either apply only to specific models with multi-head self-attention (MHA) ( 45 ) or risk degrading response quality. This paper tackles GPU memory challenges in long-context serving through KV offloading ( 50 ) , which moves parts of the KV cache to host memory and overlaps their retrieval with ongoing GPU computation. KV offloading does not compromise accuracy and, when scheduled concurrently, can achieve effective computation-communication overlap. The key insight behind this is that each layer uses only its own cached KV states and does not access caches from other layers. However, current common practices are largely SLO-unaware: they statically and monolithically offload and prefetch states for all batched requests layer by layer to save memory ( 51 ; 30 ) . Even with larger offload distances, i.e., evicting caches at regular but wider layer intervals within the GPU memory budget, static strategies fail to conceal escalating data-transfer overheads as the cache expands along both token and batch dimensions. This overhead markedly delays GPU computations and subsequent token decode steps for every request in a batch, making it difficult to consistently meet latency SLOs across requests. Our system, OrbitFlow , realizes effective KV offloading through adaptive , fine-grained offload plans. At its core, OrbitFlow employs a lightweight ILP-based solver that dynamically optimizes GPU memory usage across batched requests to minimize SLO violations. The solver iteratively refines optimal offload plans using real-time feedback on varying state-transfer and GPU computation costs per layer, allowing OrbitFlow to quickly adjust decisions under current GPU memory and compute resource constraints. For fine-grained KV offloading, OrbitFlow treats each request in a dynamic batch individually based on its current stage in the decoding phase. A key challenge is balancing GPU memory allocation among requests with distinct memory demands given limited GPU capacity. For instance, when serving a single request, OrbitFlow configures the largest possible offload distance to fully utilize available GPU memory and achieve maximum token generation speed. When another request arrives later, the two requests form a batch at different stages in decoding, with non-identical KV cache sizes. Within such batches, the memory demands of requests often conflict: a newly arrived request requires memory for rapid token generation, whereas an older request with substantial decoding progress needs to retain a large portion of its KV cache on GPU to effectively overlap computation and memory prefetching. To reconcile this conflict, our solver holistically examines various offload-reload plans within the GPU

Read the complete text