Orla: A Library for Serving LLM-Based Multi-Agent Systems

Rana Shahout, Hayder Tirmazi, Minlan Yu, Michael Mitzenmacher · 2026

Read Orla: A Library for Serving LLM-Based Multi-Agent Systems by Rana Shahout, Hayder Tirmazi, Minlan Yu, Michael Mitzenmacher (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13605.txt.

Introduction

Rana Shahout, Hayder Tirmazi, Minlan Yu, and Michael Mitzenmacher introduce Orla, a library for serving LLM-based multi-agent systems. Agentic applications combine many LLM inference steps, tool calls, and heterogeneous infrastructure. Developers now compose orchestration code, serving engines, and tool logic by hand. Orla separates request execution from workflow-level policy as a serving layer above existing inference engines. A stage mapper assigns stages to models and backends, an orchestrator schedules them, and a memory manager handles KV cache state. A customer support workflow demonstrates the system.

Catalog record

CREATORRana Shahout, Hayder Tirmazi, Minlan Yu, Michael Mitzenmacher
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31023
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13605

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Orla: A Library for Serving LLM-Based Multi-Agent Systems Rana Shahout, Hayder Tirmazi, Minlan Yu, Michael Mitzenmacher Abstract . We introduce Orla, a library for constructing and running LLM-based agentic systems. Modern agentic applications consist of workflows that combine multiple LLM inference steps, tool calls, and heterogeneous infrastructure. Today, developers typically build these systems by manually composing orchestration code with LLM serving engines and tool execution logic. Orla provides a general abstraction that separates request execution from workflow-level policy. It acts as a serving layer above existing LLM inference engines: developers define workflows composed of stages, while Orla manages how those stages are mapped, executed, and coordinated across models and backends. It provides agent-level control through three mechanisms: a stage mapper , which assigns each stage to an appropriate model and backend; a workflow orchestrator , which schedules stages and manages their resources and context; and a memory manager , which manages inference state such as the KV cache across workflow boundaries. We demonstrate Orla with a customer support workflow that exercises many of its capabilities. We evaluate Orla on two datasets, showing that stage mapping improves latency and cost compared to a single-model vLLM baseline, while workflow-level cache management reduces time-to-first-token. 1. Introduction Agentic systems built on large language models (LLMs) have demonstrated strong performance across diverse applications, including code generation ( Jimenez et al. 2023 ; Xia et al. 2025 ) , finance ( Wu et al. 2023 ; Bloomberg 2026 ) , and scientific discovery ( Gottweis et al. 2025 ) . They are also widely viewed as a promising path toward artificial general intelligence (AGI) ( Gao et al. 2025 ) . These systems solve complex tasks by executing workflows : iterative cycles of reasoning, planning, and acting, where the system consumes context, generates intermediate outputs, invokes external tools, and incorporates tool results into subsequent inference steps. We refer to each step in this workflow as a stage . As tasks grow in complexity, the field has increasingly turned to multi-agent systems, where specialized agents, potentially backed by different models, collaborate by exchanging context and intermediate results across workflow stages. Despite rapid adoption, the current infrastructure for serving agentic systems remains poorly suited to their demands. Today’s agentic inference stacks are loosely assembled from isolated components, pairing application-level orchestration code with LLM inference engines ( Shahout et al. 2024b ; Shahout et al. 2024a ; Mitzenmacher and Shahout 2025 ) such as vLLM ( Kwon et al. 2023 ) or SGLang ( Kang et al. 2026 ) . These serving systems optimize request-level execution through prefill and decode scheduling ( Patel et al. 2024 ) , continuous batching ( Yu et al. 2022 ) , and KV cache management to improve throughput and reduce latency. These optimizations are necessary, but they address only a subset of the system challenges that arise in agentic workloads. Serving agentic systems must address two closely related challenges: heterogeneity across workflow stages and coordination and decision-making across the components that execute them. Heterogeneity appears along several dimensions, including the models used, the infrastructure on which stages execute, and the characteristics of the workloads themselves. Within a single workflow, different stages often have different computational requirements. For example, a routing or summarization step may benefit from a small, fast model, whereas a synthesis or code-generation step requires a larger, more capable one. At the same time, stages may execute on different infrastructure. Modern deployments frequently span multiple serving backends and hardware environments, such as SGLang, vLLM, or Ollama. Finally, agentic workloads vary in task complexity, input characteristics, and performance objectives. Stage decomposition further leads to the need for coordination and decision-making. Agentic systems must decide which models to use, where to execute each stage, and how to schedule and manage resources across the workflow. While existing request-level serving systems can optimize the execution of individual inference calls, they lack mechanisms to coordinate decisions across stages, models, and infrastructure within a workflow. Motivated by the need for a new serving system that operates at the workflow level to coordinate decisions across stages, we introduce Orla, a serving system for agentic workflows that sits above existing LLM serving backends. Our key insight is to separate request execution from agent-level policy through three components: Stage mapper. The stage mapper aims to assign each stage to the most appropriate model and backend based on its computational requirements, performance, and available infrastructure. Orla supports both explicit developer-specified mappings and dynamic routing through pluggable mappers. Workflow orchestrator. The workflow orchestrator coordinates stage execution across workflows. It operates at two levels: scheduling and resource and context management. The scheduler determines the execution order of stages across workflows and enforces stage dependencies within each workflow. Resource and context management dispatch ready stages concurrently, keep backends utilized during tool-call waits, and reroute stages to alternative backends under load. As workflows progress, the orchestrator also manages evolving context across stages; iterative calls often share large context prefixes, while tool outputs and stage transitions may introduce new information that must be incorporated into subsequent stages. Memory manager. KV-cache decisions in agentic workloads should not be made at the level of individual inference calls. Consecutive stages in a workflow may reuse the same context, while multiple workflows must share limited GPU memory. This creates a tradeoff between preserving KV cache for reuse, which avoids redundant prefill and reduces latency, and reclaiming memory, which frees GPU capacity for other workflows. The memory manager coordinates the life-cycle of inference state across workflow boundaries by deciding when to preserve KV cache for reuse across stages and when to reclaim memory under pressure. These decisions rely on workflow-level signals, such as stage dependencies, predicted reuse, backend transitions, and workflow completion, that are invisible to a request-level backend. Together, these components form a lightweight library for constructing agentic systems. Developers define workflows using simple stage abstractions, while Orla handles the underlying coordination of models, infrastructure, and resources. Our evaluation shows that stage mapping improves latency and cost, while workflow-level cache management reduces first-token latency. Orla as a library and research testbed. A key design decision in Orla is to treat all backends as plug-and-play modules accessed through a uniform interface. Orla communicates with backends via an OpenAI-compatible HTTP API, enabling it to interact uniformly with systems such as SGLang, Ollama, or lightweight simulators. Switching between backends requires only changing the endpoint URL in the stage configuration. No modifications are required to the workflow definition, scheduling policies, or memory manager. This makes Orla suitable both for production serving and as a research testbed. Researchers studying agentic scheduling, KV cache management, or multi-agent coordination can run experiments on commodity hardware without GPU infrastructure. A simulated backend that models prefill and decode latency parametrically isolates Orla’s orchestration behavior from the variance of real LLM inference, enabling controlled and reproducible experiments. The live demo artifact exploits this property: on machines without a GPU, it runs the full customer support workflow using lightweight Ollama models on a CPU. 2. Orla Design Figure 1 . Orla design. 2.1. Core Abstractions Stage. A stage is the basic execution unit in a workflow. It performs LLM inference on a given context and may iteratively invoke tools whose results

Read the complete text