Scepsy : Serving Agentic Workflows Using Aggregate LLM Pipelines DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06

Marcel Wagenländer ∗† Otto White ∗† Britannio Jarrett ‡ Pedro Silvestre † Yanda Tao † Guo Li † Huanzhou Zhu † Llúis Vilanova † Peter Pietzuch † ∗ Equal contribution † Imperial College London ‡ Independent Researcher · 2026

Read Scepsy : Serving Agentic Workflows Using Aggregate LLM Pipelines DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 by Marcel Wagenländer ∗† Otto White ∗† Britannio Jarrett ‡ Pedro Silvestre † Yanda Tao † Guo Li † Huanzhou Zhu † Llúis Vilanova † Peter Pietzuch † ∗ Equal contribution † Imperial College London ‡ Independent Researcher (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15186.txt.

Introduction

White and colleagues serve multi-model agentic workflows with Scepsy, a scheduler for GPU clusters. Agentic workflows branch, fan out, and recurse unpredictably, and their models often outnumber available GPUs. Scepsy notes each model's share of run time stays fairly stable across requests, then profiles models under varied parallelism. Combining profiles with those shares yields an aggregate pipeline that predicts throughput and latency.

Catalog record

CREATORMarcel Wagenländer ∗† Otto White ∗† Britannio Jarrett ‡ Pedro Silvestre † Yanda Tao † Guo Li † Huanzhou Zhu † Llúis Vilanova † Peter Pietzuch † ∗ Equal contribution † Imperial College London ‡ Independent Researcher
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57926
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15186

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scepsy : Serving Agentic Workflows Using Aggregate LLM Pipelines DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Marcel Wagenländer ∗† Otto White ∗† Britannio Jarrett ‡ Pedro Silvestre † Yanda Tao † Guo Li † Huanzhou Zhu † Llúis Vilanova † Peter Pietzuch † ∗ Equal contribution † Imperial College London ‡ Independent Researcher Abstract . Agentic workflows carry out complex tasks by orchestrating multiple large language models (LLMs) and tools. Serving such workflows at a target throughput with low latency is challenging because they can be defined using arbitrary agentic frameworks and exhibit unpredictable execution times: execution may branch, fan-out, or recur in data-dependent ways. Since LLMs in workflows often outnumber available GPUs, their execution also leads to GPU oversubscription. We describe Scepsy , a new agentic serving system that efficiently schedules arbitrary multi-LLM agentic workflows onto a GPU cluster. Scepsy exploits the insight that, while agentic workflows have unpredictable end-to-end latencies, the shares of each LLM’s total execution times are comparatively stable across executions. Scepsy decides on GPU allocations based on these aggregate shares: first, it profiles the LLMs under different parallelism degrees. It then uses these statistics to construct an Aggregate LLM Pipeline, which is a lightweight latency/throughput predictor for allocations. To find a GPU allocation that minimizes latency while achieving a target throughput, Scepsy uses the Aggregate LLM Pipeline to explore a search space over fractional GPU shares, tensor parallelism degrees, and replica counts. It uses a hierarchical heuristic to place the best allocation onto the GPU cluster, minimizing fragmentation, while respecting network topology constraints. Our evaluation on realistic agentic workflows shows that Scepsy achieves up to 2.4 × \times higher throughput and 27 × \times lower latency compared to systems that optimize LLMs independently or rely on user-specified allocations. Figure 1 . Beam search as an agentic workflow (Beam search uses inference-time scaling, which uses LLMs to explore multiple reasoning paths as a search tree.) 1. Introduction Large language models (LLMs) can generate high-quality text, follow sophisticated instructions and solve problems based on only natural-language prompts ( 38 ; 2 ; 10 ) . Yet, due to their large size and autoregressive nature, LLMs are slow and expensive to execute, relying on parallelization over GPUs for acceleration. Furthermore, standalone LLMs are insufficient to carry out complex tasks: they are constrained by finite context windows ( 48 ) , lack access to external or domain-specific knowledge ( 22 ) , and struggle with hallucinations during extended reasoning ( 52 ) . As a result, developers increasingly build agentic workflows , which orchestrate multiple LLMs and external tools to enable novel capabilities and improve accuracy-cost trade-offs ( 5 ) . These workflows are diverse and often tailored to specific tasks; common features include using LLMs of different sizes for different roles, or leveraging inference-time scaling ( 43 ) techniques to enable structured reasoning (e.g., beam ( 42 ) and Monte Carlo tree search ( 65 ) ). They also augment LLMs with external tools ( 41 ) , which provide access to up-to-date information (e.g., web search and databases), or enable LLMs to offload exact, deterministic computation (e.g., to command-line utilities and language interpreters). Fig. 1 shows a typical beam-search workflow ( 42 ) : starting from a user prompt, the workflow repeatedly expands candidate reasoning steps (beams) with a generator LLM ( GEN ), scores steps with another verifier LLM ( VER ), and retains the most promising beams for the next iteration. This shows the key features of modern agentic workflows: they combine multiple LLMs of different sizes (a smaller generator and a larger verifier), intersperse LLM invocations with non-LLM code, and exhibit data-driven LLM parallelism with fan-out and fan-in invocation patterns. Hyperscalers have massive data centers and performance engineering teams to optimize agentic workflow serving ( 2 ; 10 ; 38 ) , but smaller institutions have a much harder time running self-managed deployments efficiently: managing small-to-medium scale GPU clusters means that small inefficiencies matter a lot more, and having smaller performance engineering teams means that they lack the knowledge necessary for the rapidly changing agentic landscape. Yet self-managed deployments are still desirable for privacy ( 14 ) and potential cost savings ( 31 ) . We therefore identify four challenges in managing self-managed agentic workload deployments: Arbitrary agentic programs. Developers use a variety of agentic frameworks, such as LangChain ( 20 ; 21 ) , AutoGen ( 55 ) or Camel ( 23 ) and other libraries (schema validation, numerical, databases) to define workflows ( 51 ) . Thus, a serving system cannot assume a single programming model, API or the definition of workflows. Unpredictable execution. The latency of an LLM execution is highly variable because generation proceeds token-by-token until the end of the sequence. Since LLMs may invoke tools and other LLMs, the workflow itself may branch, iterate, and recur in data-dependent ways. Together, these properties make predicting the throughput and latency of an agentic workflow based on a given GPU allocation difficult. Conflicting optimization objectives. Users want agentic workflows with high throughput and low latency. LLM throughput can be increased by exploiting data parallelism with parallel model replicas, but this reduces the GPU resources available for tensor parallelism , which lowers LLM latency. While serving a single LLM navigates one such trade-off ( 24 ; 18 ) , agentic workflows with multiple LLMs compose several objectives: adjusting the allocation of one LLM may affect another, or shift bottlenecks. The heterogeneity of GPU interconnects only worsens this problem ( 34 ) , as interconnect availability constrains tensor parallelism. Oversubscribed GPUs. GPUs are a scarce and expensive resource ( 31 ) , and even small inefficiencies bear a large cost impact in small-to-medium scale clusters. Compounding this, agentic workflows have a highly heterogeneous mix of LLMs (embedding, generator, reward, etc.), where coarse LLM-to-GPU allocations quickly lead to poor GPU utilization. Current solutions fail to address these challenges. Some serving approaches for agentic workflows ( 25 ; 3 ; 28 ) exploit the workflow structure to optimize execution, e.g., using graph-aware batching, KV-cache sharing, or reducing head-of-line blocking. However, several of these approaches ( 25 ; 3 ) are constrained by framework-specific programming models, and all are limited to single-LLM workflows. More fundamentally, they rely solely on temporal request scheduling to GPUs, and the complexity of managing the unpredictable execution of agentic workflows leaves the allocation of LLMs to GPUs to users. Other approaches ( 24 ; 8 ; 59 ; 18 ; 56 ) focus on multi-LLM serving through temporal or spatial multiplexing without assuming any agentic workflow structure; this makes them more flexible, but the lack of workflow awareness leads to suboptimal performance, since they schedule each LLM in isolation. We empirically observe that, while end-to-end execution times of a workflow vary greatly across requests, the fraction of a request’s total execution time that is spent on each LLM is substantially more stable. Intuitively, despite being unpredictable, workflows have underlying structure that creates stable steady-state behavior: e.g., in beam search, a generator execution is always followed by a verifier execution. Thus, our key insight is that, for

Read the complete text