Tokencake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications

Zhuohang Bian, Feiyang Wu, Teng Ma, Youwei Zhuo · 2025

Read Tokencake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications by Zhuohang Bian, Feiyang Wu, Teng Ma, Youwei Zhuo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18586.txt.

Introduction

Zhuohang Bian and his four co-authors present a serving system called TokenCake. It serves apps where many language-model agents work together and call outside functions. Such loads strain the key-value cache, which stores each agent's working context. Agents stalled on slow calls leave cache idle in pricey graphics memory. Other agents get evicted and lose theirs.

TokenCake attacks both flaws with agent-aware scheduling. A time-based scheduler moves idle caches out during calls and reloads them just in time, hiding the transfer delay. A space-based scheduler reserves memory for key agents. Its priority score mixes workflow structure with live state. Tests on multi-agent benchmarks cut end-to-end delay by up to about forty-seven percent under tight memory. Useful graphics memory use rose by nearly seventeen points over a leading baseline.

Catalog record

CREATORZhuohang Bian, Feiyang Wu, Teng Ma, Youwei Zhuo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18586

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tokencake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications Zhuohang Bian, Feiyang Wu, Teng Ma, Youwei Zhuo Abstract . Large Language Models (LLMs) are increasingly deployed in complex multi-agent applications that use external function calls. This workload creates severe performance challenges for the KV Cache: space contention leads to the eviction of critical agents’ caches and time underutilization leaves the cache of agents stalled on long-running tool calls idling in GPU memory. We present Tokencake, a KV-Cache-centric serving framework that co-optimizes scheduling and memory management with an agent-aware design. Tokencake’s Space Scheduler uses dynamic memory partitioning to shield critical agents from contention, while its Time Scheduler employs a proactive offload and predictive upload mechanism to repurpose GPU memory during function call stalls. Our evaluation on representative multi-agent benchmarks shows that Tokencake can reduce end-to-end latency by over 47.06 47.06 %, improve effective GPU memory utilization by up to 16.9 16.9 % compared to vLLM. 1 1. Introduction Large Language Models (LLMs) are powerful reasoning engines, and applications built upon them are evolving from single-response generation to complex, multi-agent systems. This new approach has enabled powerful applications in domains like autonomous code generation ( Microsoft 2023 ) , complex financial analysis ( Xiao et al. 2025 ) , and realistic environment simulation ( Piao et al. 2025 ) . The defining characteristic of these applications is a dual-interaction model: frequent external, agent-tool interaction and complex internal, agent-agent collaboration. Externally, these agents use function calls to interact with tools, data sources, and APIs. Internally, these systems orchestrate multiple specialized agents that collaborate to solve a larger problem. Figure 1 illustrates this application model. For example, Code-Writer ( Zhang et al. 2024 ) and Deep-Research ( Team 2025 ) are composed of internal pipelines of agents (e.g., programmers, reviewers, searchers) that in turn make frequent external calls to tools like file systems and web APIs. The combination of these complex internal dependencies and frequent, long-running external interactions results in workload patterns distinct from traditional LLM inference, introducing unique and significant performance challenges for the underlying serving infrastructure. Figure 1 . Example LLM-based Multi-Agent Applications. Each colored box represents a specialized agent. Purple boxes denote function calls to external tools. (a) Multi-agent Coding (b) Deep Research These challenges are rooted in inefficient KV Cache management, which manifests as two critical problems: time underutilization and space contention . First, time underutilization stems from the frequent and long-running function calls inherent to agentic workloads. An agent’s execution follows an LLM Inference1 \Rightarrow Function Call \Rightarrow LLM Inference2 pattern, where its KV Cache sits unused during the function call. This forces a difficult trade-off: retain the cache and waste resources, or evict it and incur a costly recomputation. This inefficiency is substantial; as shown in Figure 3(a) , at peak moments, as much as 18.5 % of the GPU KV Cache pool can be wasted by stalled agents, directly reducing the system’s capacity for active requests, lowering the effective batch size, and degrading overall throughput. Systems like Teola ( Tan et al. 2025 ) have identified the latency challenge posed by this pattern and propose a workflow-level optimization. Teola intelligently pipelines the execution of LLM and non-LLM micro stages, aiming to overlap the tool-use-time of one agent with the computational work of another. While this approach effectively hides latency by improving the application’s end-to-end execution schedule, it is fundamentally compute-centric. Teola’s scheduler optimizes the flow of operations but remains blind to the state of the underlying GPU memory resources. Consequently, the KV Cache of the stalled agent continues to idly occupy valuable GPU memory, a problem that is exacerbated when multiple agents stall concurrently. Second, space contention arises as numerous agents compete for limited GPU memory. Agent-unaware memory allocation policies like FCFS often lead to performance bottlenecks, where a non-critical agent causes the eviction of a critical-path agent’s KV Cache, a problem we refer to as critical inversion. We do an empirical measurement on the application Code-Writer, and count the preemption number of critical inversion. As shown in Figure 2(a) , these harmful preemptions happen are frequent. This forces the evicted critical agent to undergo a costly context recomputation, stalling the entire application workflow. While systems like Parrot ( Lin et al. 2024 ) and Autellix ( Luo et al. 2025 ) are workflow-aware, their optimizations operate at the request level, agnostic to the fine-grained KV Cache contention between individual agents. It optimizes the order and batching of requests but does not manage the underlying memory allocation. Consequently, even with an optimal schedule, a high-throughput, non-critical task group identified by Parrot could still occupy GPU memory and inadvertently cause the eviction of a latency-sensitive, critical agent’s KV Cache. This exposes a fundamental limitation: scheduling optimization alone cannot solve memory contention. (a) Preemption events over time of the Code-Writer workload. (b) KV Cache blocks held by non-critical agents. Figure 2 . The Space Contention Problem. (a) Idle KV Cache blocks due to external function call. (b) The lifecycle of an agent’s KV Cache during a function call. Figure 3 . The Time Underutilization Problem. To address these challenges, we present Tokencake, a KV-Cache-centric serving framework that co-optimizes scheduling and memory management through an agent-aware design. Tokencake begins with a frontend API that allows developers to define an application’s internal agent collaborations and external tool interactions as a graph (Section 3.1 ). This graph enables two specialized schedulers to manage the KV Cache lifecycle with application-level context. To mitigate time underutilization, the Time Scheduler uses an event-driven, opportunistic policy to proactively offload the KV Cache of stalled agents during function calls and uses predictive uploading to hide data transfer latency (Section 4 ). To resolve space contention, the Space Scheduler employs a dynamic memory partitioning policy, guided by a hybrid priority metric, to reserve memory for critical-path agents (Section 5 ). We further introduce optimizations such as CPU block buffering and gradual GPU block reservation to minimize the overhead of these frequent offload operations. Our evaluation shows that Tokencake significantly reduces end-to-end latency by over 47.06 % compared to vLLM under high load and improves GPU memory utilization by over 16.9 % (Section 7 ). 2. Background and Motivation 2.1. LLM-based Multi-Agent Applications Agents. A common design pattern in modern agentic systems is the decomposition of complex problems into a series of sub-tasks handled by specialized agents ( Wu et al. 2023 ; Hong et al. 2024 ; Gao et al. 2024b ) . These collaborations are often modeled as a Directed Acyclic Graph (DAG), where nodes represent agents and edges represent explicit dependencies. In these workflows, the dependencies are critically important; for instance, a reviewer agent cannot begin its task until a programmer agent has completed its code generation. Furthermore, not all agents contribute equally to the application’s end-to-end latency. Some agents lie on the critical path, meaning any delay they experience directly increases the total time-to-result for the entire application. Function Calls. A key feature of modern agents is the use of function calls to interact with external data sources or execute actions. These calls connect the LLM to a vast array of tools, such as database clients, code interpreters, or third-party web APIs. To standardize these

Read the complete text