MARS : Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06
Yifei Wang Hancheng Ye Yechen Xu Cong Guo Chiyue Wei Qinsi Wang Dongting Li Tingjun Chen Hai "Helen" Li Danyang Zhuo Yiran Chen Duke University · 2026
Read MARS : Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 by Yifei Wang Hancheng Ye Yechen Xu Cong Guo Chiyue Wei Qinsi Wang Dongting Li Tingjun Chen Hai "Helen" Li Danyang Zhuo Yiran Chen Duke University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26963.txt.
Introduction
Wang and ten colleagues design MARS, a co-scheduling system for agentic workloads that mix language-model inference with CPU tool execution. One information stream gives holistic visibility across GPU and CPU work, while a control plane separates admission from execution to prevent resource oversubscription. A scheduler shortens the critical path by favoring latency-sensitive continuations and keeping KV cache state only when warm resumption pays off. MARS cuts end-to-end latency by up to about six times while holding throughput near maximal.
Catalog record
| CREATOR | Yifei Wang Hancheng Ye Yechen Xu Cong Guo Chiyue Wei Qinsi Wang Dongting Li Tingjun Chen Hai "Helen" Li Danyang Zhuo Yiran Chen Duke University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57820 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26963
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MARS : Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Yifei Wang Hancheng Ye Yechen Xu Cong Guo Chiyue Wei Qinsi Wang Dongting Li Tingjun Chen Hai "Helen" Li Danyang Zhuo Yiran Chen Duke University Abstract . Large language models (LLMs) are increasingly deployed as the execution core of autonomous agents rather than as standalone text generators. Agentic workloads induce a temporal shift from single-turn inference to multi-turn LLM-tool loops, and a spatial shift from chat-scale, GPU-only execution to repository-scale, GPU-CPU co-located execution. Consequently, coordinating heterogeneous resource demands of agentic execution has emerged as a critical system challenge. We design and implement MARS , an efficient and adaptive co-scheduling system that globally coordinates heterogeneous agentic workloads under coupled GPU-CPU resource pressure . By establishing holistic visibility across GPU inference and CPU tool execution via a unified information stream, an external control plane in MARS decouples admission from execution to prevent heterogeneous resource oversubscription. An internal agent-centric scheduler further minimizes the end-to-end critical path by prioritizing latency-sensitive continuations and adaptively retaining KV cache state only when warm resumption yields a latency benefit. Our evaluations show that MARS reduces end-to-end latency by up to 5.94 × \times while maintaining nearly maximal system throughput. We further integrate MARS as the serving backend for the OpenHands coding agent framework, demonstrating its real-world effectiveness by accelerating end-to-end task completion time by up to 1.87 × \times . Our source code will be publicly available soon. 1. Introduction Large language models (LLMs) are increasingly deployed not as standalone text generators, but as the control core of autonomous agents. In these systems, the model plans, invokes tools, inspects intermediate outcomes, updates states, and iterates until the task completes. Such an iterative workflow has been well designed in existing orchestration frameworks such as MetaGPT ( 8 ) , AutoGen ( 32 ) and LangChain ( 18 ) , and in emerging agentic applications such as research assistants ( 24 ) , computer-use agents ( 33 ) , and software engineering agents that edit and test files over entire repositories ( 35 ; 31 ) . As a result, the serving problem is changing fundamentally since the serving object transitions from isolated model calls to a mixture of long-lived agentic workflows. Figure 1 . Agentic workloads induce both temporal and spatial shifts. Together, these shifts create dynamically misaligned GPU and CPU demand across concurrent requests. Existing modern LLM serving engines are mainly optimized for a different regime. Systems such as Orca ( 38 ) , TGI ( 9 ) , and vLLM ( 11 ) are designed around short-lived, largely independent, GPU-bound requests, targeting at maximizing token throughput through techniques such as continuous batching and KV cache optimization. Agent workloads break these paradigms in three concrete ways, as illustrated in Figure 1 . First , a session no longer ends after one generation step. It pauses for tool execution and later resumes, often repeatedly. The engine must therefore decide whether to keep its KV state resident for fast continuation or evict it and pay prefix recomputation on resumption. Second , request sizes become far more heterogeneous, ranging from chat-scale short prompts to repository-scale long contexts, e.g., accumulated histories, tool traces, and execution logs. Naively batching these requests together causes long prefills to block short continuations, hurting both latency and task completion time. Third , the critical path is no longer GPU-only. Tool phases consume host CPUs or external runtimes, so GPU and CPU execution become coupled: congestion on one side can strand capacity on the other. We summarize the root causes of these pathologies as two workload shifts. The first is a temporal shift : serving moves from single-turn inference to multi-turn execution. This shift lengthens session lifetimes and turns the KV cache into a persistent execution state. The second is a spatial shift : each session expands both its working set and its resource footprint. On the input side, workloads move from chat-scale prompts to repository-scale contexts and long execution traces, sharply increasing KV footprints ( 10 ; 15 ; 39 ) . On the execution side, serving moves from GPU-only inference to heterogeneous GPU-CPU coupled serving, as tool invocations often run on co-located host CPUs in practical deployments ( 31 ; 26 ) . Together, these shifts mean that maximizing GPU token throughput alone is no longer sufficient; the serving engine must instead optimize end-to-end workflow completion under coupled pressure across GPU compute, KV cache capacity, and host CPU resources. These shifts reveal a fundamental systems challenge : how to orchestrate the periodic bursts of heterogeneous resource demand arising from multi-turn agentic execution . Failing to holistically manage this coupled pressure across heterogeneous hardware inevitably degrades end-to-end performance, yet existing systems fall into three camps and solve only fragments of this problem. Traditional throughput-centric engines serve generation steps in simple arrival order, and they blindly admit work without respecting session dependencies or resource demands. Program-aware frameworks like Autellix ( 17 ) elevate scheduling to the application level but remain physically resource-agnostic. As they are unaware of how much GPU and CPU resources an agentic request consumes, they fail to protect GPUs from Head-of-Line (HoL) blocking caused by long prefill requests, while simultaneously ignoring host-side CPU overheads, which leads to severe GPU-CPU coupled queuing. Finally, tool-aware systems such as Infercept ( 1 ) and Continuum ( 12 ) rely on fragile heuristics defined at tool invocation time to statically manage KV cache residency during tool sessions. Under highly variable request sizes and unpredictable arrivals, such static one-off policies often misfire and degrade overall performance. At its core, the problem is that today’s infrastructure cannot make real-time dynamic scheduling and retention decisions at the granularity of agent sessions , because it lacks live, cross-plane telemetry spanning both GPU and CPU resource pressure. To address this gap, we present MARS , an efficient and adaptive serving architecture designed for heterogeneous demands of agentic workflows. MARS abandons the request-oblivious paradigm in favor of two core design principles: holistic visibility to explicitly monitor resource pressure across both GPU and CPU planes, and session-aware orchestration to jointly optimize admission, scheduling, and KV management around the multi-turn workflow lifecycle. To realize these principles, MARS introduces a Unified Information Stream that continuously exports execution boundaries and cross-device hardware pressure. Operating on this telemetry, an External Control Plane regulates admission to prevent heterogeneous thrashing, while an Internal Agent-Centric Scheduler dynamically prioritizes latency-sensitive continuations, resolves head-of-line blocking, and intelligently retains KV state only when warm resumption strictly shortens the end-to-end critical path. We implement MARS on top of vLLM ( 11 ) and further integrate it as the backend of the open-source coding-agent framework OpenHands ( 31 ) . In the OpenHands deployment on H200, MARS improves task completion time by up to 1.87 \times . On H100 and H200 platforms with Qwen3-Coder-30B ( 34 ) and GPT-OSS-120B ( 23 ) , driven by multi-round agentic workloads from SWE-bench ( 10 ) , GitTaskBench ( 22 ) , Terminal-Bench ( 20 ) , RepoBench ( 15 ) , and \infty Bench ( 39 ) ,