Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective (Extended) CCS: Information systems Query optimization CCS: Computing methodologies Multi-agent systems

Noppanat Wadlom, Junyi Shen, Yao Lu · 2026

Read Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective (Extended) CCS: Information systems Query optimization CCS: Computing methodologies Multi-agent systems by Noppanat Wadlom, Junyi Shen, Yao Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16104.txt.

Introduction

Wadlom, Shen, and Lu serve agent workflows like database query plans. Agent apps chain many linked language model calls in sequence. Speculative branches repeat prompts and intermediate work across calls. Call-by-call servers miss the cross-call waste in that pattern. Their Helium frame models each workflow as a query plan. Model calls act as first-class operators inside the plan. Proactive caching and cache-aware scheduling spread reuse widely. Speedups reach 1.56 times over top agent servers. End-to-end workflow tuning proves vital at scale.

Catalog record

CREATORNoppanat Wadlom, Junyi Shen, Yao Lu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85615
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16104

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective (Extended) CCS: Information systems Query optimization CCS: Computing methodologies Multi-agent systems Noppanat Wadlom, Junyi Shen, Yao Lu Abstract . Agentic workflows are composed of sequences of interdependent Large Language Model (LLM) calls, and they have become a dominant workload in modern AI systems. These workflows exhibit extensive redundancy from overlapping prompts and intermediate results due to speculative and parallel exploration. Existing LLM serving systems, such as vLLM, focus on optimizing individual inference calls and overlook cross-call dependencies, leading to significant inefficiencies. This paper rethinks LLM and agent serving from a data systems perspective and introduces Helium , a workflow-aware serving framework that models agentic workloads as query plans and treats LLM invocations as first-class operators. Helium integrates proactive caching and cache-aware scheduling to maximize reuse across prompts, KV states, and workflows. Through these techniques, Helium bridges classic query optimization principles with LLM serving, achieving up to 1.56× speedup over state-of-the-art agent serving systems on various workloads. Our results demonstrate that end-to-end optimization across workflows is essential for scalable and efficient LLM-based agents. 1. Introduction AI agents are autonomous LLM-based programs that act on a user’s behalf ( 70 ; 74 ; 46 ) . They operate through agentic workflows ( 66 ; 85 ; 82 ; 59 ) that are goal-driven sequences of steps involving multiple LLM invocations (often using different prompts or tools), orchestrated to solve complex tasks. They have become a dominant workload in modern AI systems ( 5 ; 73 ; 45 ) . Figure 1. Three disparities between traditional SQL pipelines and agentic workflows with LLM as operators. An emerging scenario of such workflows is that agents may explore many strategies or subtasks (e.g., trying alternative analyses) in parallel or in sequence to achieve their goal. This agentic speculation ( 45 ) is a high-throughput exploration to blend LLM serving with batch-style query processing. In other words, an agentic workflow resembles a batch analytics pipeline where LLM calls function as operators. While powerful, these workflows can issue a large volume of LLM queries, often with overlapping or repeated sub-queries, leading to significant redundancy and inefficiency. Recent advances in LLM serving systems have largely focused on optimizing individual LLM inference tasks. State-of-the-art serving engines such as vLLM ( 31 ) employ techniques like continuous batching ( 83 ) , which dynamically group incoming requests to better utilize GPUs. These innovations, along with optimized GPU kernels and memory paging strategies ( 9 ; 81 ; 29 ; 52 ) , have dramatically increased throughput and reduced latency for standalone LLM queries. However, existing LLM inference engines operate at the granularity of individual LLM calls, lacking visibility into the broader workflow structure. These solutions are not designed for batch agentic workflows that chain up multiple LLM calls (often with different prompts). As a result, in complex multi-LLM pipelines, e.g., an agent that spawns several helper agents, each querying an LLM, these per-call optimizations fail to capture cross-call commonalities (e.g., shared sub-prompts or intermediate results). Recent work, such as AgentScope ( 17 ; 51 ) , supports basic multi-agent pipelines, but they again treat each LLM call or agent as an individual unit, optimizing locally rather than globally. This gap leaves performance optimizations unexplored for batch agentic workflows. How can we optimize agentic LLM workflows end-to-end? Many of the challenges mirror classic problems in query processing and optimization. An agent workflow can be represented as a directed acyclic graph (DAG) of operations: nodes perform data retrieval or invoke an LLM (i.e., LLM-as-operators ), and edges represent data or prompt flow between operators. This is analogous to a complex relational query plan or a workflow in a data system. Prior work suggests that decades of database query optimization principles can be applied by modeling an agentic workflow as a query-plan DAG and applying cost-based optimization. The approach of logical and physical optimization, i.e., rewriting plans, choosing operator implementations, and scheduling execution to minimize cost, contributes to the “agent-first” query processing techniques to handle the scale and complexity of LLM-driven workloads ( 12 ) . In essence, this is a familiar data systems problem framed in the context of LLM serving to eliminate redundant work by globally optimizing the workflow, much like a SQL optimizer would do for a complex relational query. However, at the same time, important new factors differentiate LLM workflows from traditional SQL pipelines. We identify and summarize the disparities of LLM-as-operators as follows and as illustrated in Figure 1 : • Operator abstraction : The operators in an agentic workflow wrap expensive LLM inference processes. The continuous batching technique in LLM serving systems becomes dominant to enable processing of multiple input queries in a streaming fashion without being blocked by other queries in the same batch. Within a single LLM call, the model maintains internal state (e.g., a KV cache of the prompt) and generates output token by token in an iterative fashion. In essence, the operators’ batching mechanisms are no longer abstracted by classic relational functions like select or filters. • Inter-operator sharing : As LLMs are inherently stateful, standard data-processing optimizations must be rethought. Across multiple calls, an agent may carry forward conversational context or reuse an earlier answer in a later prompt, along with the KV cache and model’s internal state, creating stateful dependencies between operations. • Inter-workflow sharing : For speculative and batch agentic workflows, prefix sharing among the queries from an input batch becomes an important technique to reuse partial or even entire agentic sub-chains across queries. This is also the case for workflows across different input batches to query common data sources (e.g., agents that query daily weather or news). Unfortunately, current data systems are not tailored for these shifts and foresee significant challenges in efficiently processing agentic workflows. LLM operators often are wrapped in user-defined functions (UDFs) by current frameworks (e.g., Spark, Dask ( 84 ; 58 ) ), while an LLM call is treated as a black-box UDF, which prevents introspection or special-case handling for performance. This lack of visibility precludes many optimizations: the system cannot automatically reuse common partial work (like shared prompt prefixes), pipeline LLM calls, or reorder operations based on LLM-specific cost factors. Importantly, continuous batching does not naturally happen beyond each individual UDF, causing blockage and significant performance degradation. Nevertheless, cost-based Volcano-style query optimization ( 20 ) principles still apply, as operator costs and even semantics are often readily available, but LLM-as-operators introduces a new form of batching abstraction, statefulness, and cost that traditional optimizers were never designed to handle. To mitigate the paradigm shifts in agentic workflows and close the gap between the continuous batching abstract in LLM serving systems and the classic batching abstract in data systems, we consider a key missing jigsaw to be a new workflow-aware LLM serving that applies continuous batching while efficiently manages KV and prompt caches from a holistic, workflow perspective, instead of within the scope of individual UDFs. Specifically, it must simultaneously support: (i) inter-operator sharing, by enabling KV state communication across LLMs; (ii) inter-query and inter-batch sharing, by organizing caches according to prompt prefixes across multiple workflows, as well as (iii) an optimizer to maximize the chance

Read the complete text