Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI

Tian Xia 1 , Hanchen Li 1 , Zhifei Li 2 , Xiaokun Chen 3 , Hao Kang 4 , Yifan Qiao 1 , Yi Xu 1 , Ion Stoica 1 1 University of California, Berkeley 2 Renmin University of China 3 Stanford University 4 Georgia Institute of Technology · 2026

Read Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI by Tian Xia 1 , Hanchen Li 1 , Zhifei Li 2 , Xiaokun Chen 3 , Hao Kang 4 , Yifan Qiao 1 , Yi Xu 1 , Ion Stoica 1 1 University of California, Berkeley 2 Renmin University of China 3 Stanford University 4 Georgia Institute of Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00866.txt.

Introduction

Xia and co-authors manage memory for AI agent workloads in this 2026 systems paper. Agent sessions call models dozens of times, with tool calls in between. Each step reuses a KV cache, which speeds the next call. Total cache size soon tops GPU memory, so caches spill to CPU memory. Moving them back and forth is costly.

Tool calls vary wildly in length, and agent work falls into busy and idle phases. Flat rules like least-recently-used miss that shape. Their system, MORI, treats idleness as a spectrum, not a switch. It ranks all live programs by idleness. The busiest sit in GPU memory and the idlest in CPU memory. The split line slides to match the true hardware ratio. Admission control guards each tier. On real coding-agent traces from Claude Code, the authors report, MORI uses both tiers well and cuts wasteful evictions.

Catalog record

CREATORTian Xia 1 , Hanchen Li 1 , Zhifei Li 2 , Xiaokun Chen 3 , Hao Kang 4 , Yifan Qiao 1 , Yi Xu 1 , Ion Stoica 1 1 University of California, Berkeley 2 Renmin University of China 3 Stanford University 4 Georgia Institute of Technology
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00866

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI Tian Xia 1 , Hanchen Li 1 , Zhifei Li 2 , Xiaokun Chen 3 , Hao Kang 4 , Yifan Qiao 1 , Yi Xu 1 , Ion Stoica 1 1 University of California, Berkeley 2 Renmin University of China 3 Stanford University 4 Georgia Institute of Technology Abstract . Modern LLM serving systems increasingly host agentic workloads, whose sessions issue tens of model invocations interleaved with tool calls, accumulating KV cache that can be reused across steps. As requests’ total KV cache size easily exceeds GPU HBM capacity, researchers offload them to CPU DRAM. However, tool-call durations span orders of magnitude, and the cost of transferring KV cache between tiers makes it impractical to re-place entries on every call. We observe that agentic programs exhibit a two-phase structure: busy phases of rapid short tool calls and idle phases dominated by long-running calls. Current eviction policies such as LRU fail to capture this property. A binary busy/idle label also falls short because the ratio of busy to idle programs may not match the hardware’s GPU-to-CPU capacity ratio. When it does not, one tier sits underutilized while the other is oversubscribed, wasting memory or forcing unnecessary evictions. We present MORI , an agent serving system that solves the above problem. Our key insight is that idleness is a continuous, relative spectrum. MORI ranks all active programs by idleness, assigns the busiest to GPU HBM and the most idle to CPU DRAM, dynamically shifts the partition boundary to match hardware capacity, and enforces admission control at each memory tier. Evaluated on real coding agent workloads collected from Claude Code across four GPU and model pairs, MORI delivers 20–71% higher throughput and 18–43% lower TTFT than the best baseline with offloading. thoncode [yamlcode]yaml baselinestretch=1.2 1. Introduction Modern large language model (LLM) serving systems increasingly execute long-running agentic workloads ( 6 ; 7 ; 36 ; 63 ; 55 ; 56 ) rather than isolated chat requests. In these workloads, a program is not a single prompt-response interaction but a stateful execution that follows a ReAct-style loop ( 65 ; 46 ) of model reasoning and tool execution: each model invocation produces an action, the action’s result is appended to the context, and the next invocation continues from the extended state. As a result, the unit of serving is no longer an individual LLM request, but an agent program whose context, tool state, and memory footprint evolve over time. This shift in agentic workload changes the way that serving systems interpret the gaps between model invocations. In conventional request-based serving, a completed invocation marks a natural boundary at which KV cache can be reclaimed or offloaded. In an agent program, however, the intervening tool-execution period is part of the same logical execution: the agent is waiting for feedback that will be embedded in the next prompt, often over the same accumulated context. While the inference engine itself sees only individual requests, a program-aware scheduler can recognize these requests as parts of the same program with prefix dependencies across requests and a persistent memory working set. As a result, program-aware scheduling has been increasingly adopted by recent systems ( 24 ; 27 ; 33 ; 13 ) , which pin a program’s KV cache in GPU HBM across tool calls to prevent eviction during short gaps that would otherwise cause thrashing. Figure 1 . Overview of MORI . (1) MORI categorizes programs into a continuous idleness spectrum reflecting how likely each is to remain GPU-resident. (2) An adaptive offloading scheduler places busy programs (low idleness) in GPU HBM and idle programs (high idleness) in CPU DRAM, with admission control for both tiers. (3) In multi-replica deployments, an affinity-aware load balancer routes requests to engines that already cache each program’s KV state while respecting load balance across replicas. As agent programs grow their memory footprint over time, the aggregate KV cache of concurrently running programs can quickly exceed the GPU KV cache capacity. However, pinning confines the working set to GPU-local HBM: the context of a program that do not fit must be evicted entirely and later recomputed from scratch, wasting GPU cycles and increasing latency. This has motivated memory spilling techniques, including KV cache offloading to CPU memory or other storage tiers ( 45 ; 60 ; 12 ; 31 ; 21 ; 41 ; 32 ; 64 ; 26 ; 50 ) , to increase the effective capacity of LLM serving systems. Such memory spilling introduces a two-tier hierarchy: GPU HBM and CPU DRAM. From the scheduler’s perspective, each program is placed in exactly one tier at any given time. A program in the GPU tier runs inference normally. However, when it calls a tool, its KV cache can be offloaded to CPU DRAM to free GPU memory for other programs. By contrast, a program with its context in the CPU tier remains paused even after its tool call completes, because inference cannot resume until that context is promoted back to GPU HBM. The scheduling objective is therefore to maximize GPU utilization by keeping the most active programs GPU-resident, while retaining as many other programs as possible in CPU DRAM to avoid costly recomputation when they resume. The key challenge is that tool-call durations can vary by orders of magnitude and are not known until the calls are completed. This unpredictability complicates offloading decisions. Ideally, the system would keep the program’s KV cache in GPU memory for short tool calls and offload it for long ones. Unfortunately, this would require one to accurately predict the duration of the tool call, which is not feasible in most cases. Existing eviction policies are ill-suited to this setting. Standard LRU, used in many serving systems ( 25 ; 68 ) , evicts the least recently active program without considering whether it will resume soon. Program-aware schedulers that choose eviction candidates based on static properties such as context length ( 24 ) , face the same problem: they may keep a program blocked on a long tool call in GPU memory while evicting one that is about to resume. Our trace analysis reveals a key pattern: despite per-call unpredictability, agentic programs typically remain in one of two phases for extended periods. In a busy phase, the program issues a sequence of short tool calls, each lasting hundreds of milliseconds, such as reading or writing multiple files. In an idle phase, the program blocks on a long tool call lasting seconds or more, such as waiting for human input or for a spawned subagent to return. Thus, keeping busy-phase programs in GPU HBM and moving idle-phase programs to CPU DRAM uses the fast tier for programs likely to resume soon, while freeing GPU memory from programs likely to remain blocked. However, a binary busy/idle classification is not sufficient for tier placement. Because programs continuously transition between phases and grow their contexts, the aggregate memory demand of busy programs relative to idle programs fluctuates over time. A fixed GPU-to-CPU capacity ratio cannot track this shifting demand: at some moments busy programs require more memory than GPU HBM can hold, forcing evictions of programs likely to

Read the complete text