When Should We Orchestrate Multiple Agents?

Umang Bhatt, Sanyam Kapoor, Mihir Upadhyay, Ilia Sucholutsky, Francesco Quinzan, Katherine M. Collins, Adrian Weller, Andrew Gordon Wilson, Muhammad Bilal Zafar · 2025

Read When Should We Orchestrate Multiple Agents? by Umang Bhatt, Sanyam Kapoor, Mihir Upadhyay, Ilia Sucholutsky, Francesco Quinzan, Katherine M. Collins, Adrian Weller, Andrew Gordon Wilson, Muhammad Bilal Zafar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13577.txt.

Introduction

Umang Bhatt and a team of co-authors ask when it pays to coordinate several agents in a 2025 paper. Teams of people and AI systems can be arranged in many ways, but coordination itself costs time and computing, and naive setups overstate gains. The authors design a framework that plans coordination under real limits such as cost and availability.

They prove that coordination helps only when agents differ in skill or in cost. Tests then show the framework picking agents in a simulated world, choosing a learning strategy for a classic social science puzzle, and handing off tasks in a question answering study with users.

Catalog record

CREATORUmang Bhatt, Sanyam Kapoor, Mihir Upadhyay, Ilia Sucholutsky, Francesco Quinzan, Katherine M. Collins, Adrian Weller, Andrew Gordon Wilson, Muhammad Bilal Zafar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13577

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Should We Orchestrate Multiple Agents? Umang Bhatt, Sanyam Kapoor, Mihir Upadhyay, Ilia Sucholutsky, Francesco Quinzan, Katherine M. Collins, Adrian Weller, Andrew Gordon Wilson, Muhammad Bilal Zafar Abstract Strategies for orchestrating the interactions between multiple agents, both human and artificial, can wildly overestimate performance and underestimate the cost of orchestration. We design a framework to orchestrate agents under realistic conditions, such as inference costs or availability constraints. We show theoretically that orchestration is only effective if there are performance or cost differentials between agents. We then empirically demonstrate how orchestration between multiple agents can be helpful for selecting agents in a simulated environment, picking a learning strategy in the infamous Rogers’ Paradox from social science, and outsourcing tasks to other agents during a question-answer task in a user study. † † affiliationnotice: Equal contribution 1 Introduction The world is full of agents, making decisions and taking actions based on those decisions. Historically, “agents” have been restricted to humans: human decision makers, humans assisting other human decision makers, and organizations or institutions taking collective action on behalf of other humans. Increasingly, however, the purview of “agents” is expanding to include artificial systems, e.g., large language models (LLMs) or artificial intelligence (AI) agents, including research assistants for structured output generation and computational thought partners for plan execution ( Collins et al., 2024b ) . For a given task, each agent comes with constraints, expertise, and preferences. As AI agents are endowed with exceptional capabilities ( Jaech et al., 2024 ) , invoking the right agent at the right time is crucial to realizing the potential of AI: the safe and secure use of AI will be under increased scrutiny in the coming years. “Orchestration” has become a common way to route tasks between a set of agents ( Fügener et al., 2022 ; Holstein & Olsen, 2023 ; Rasal & Hauer, 2024 ) . In practice, this amounts to selecting an agent with the highest expected utility or choosing the most accurate agent based on offline data. Existing research around agent orchestration seldom accounts for real-world costs and constraints formally ( Keswani et al., 2021 ; Lai et al., 2022 ) . Realistically, some agents may be unsuitable (e.g., lacking expertise,), unavailable (e.g., no security clearance, misaligned with regulation, preoccupied with another task), or costly (e.g., astronomical inference-time compute costs, high billable rates, large carbon emissions). Failing to consider agent capabilities alongside costs and constraints can overestimate the effectiveness of orchestration. Orchestration is not only about the impact on a single agent or a set of agents’ ability to achieve a goal. How we orchestrate agents with one another (humans and AI systems) can have ripple effects across networks of interacting agents. For instance, Collins et al. (2025) advocate for AI researchers to consider engaging with the classic social science challenge, Rogers’ Paradox ( Rogers, 1988 ) , to better understand – at a coarse, abstract network level – the impact of human-AI interaction on a population of agents. In Rogers’ Paradox ( Rogers, 1988 ) , agents must adapt either via individual learning or via learning from other agents. Each agent needs to select a learning strategy at every time step: orchestrating when to pick each learning strategy is key. In this paper, we consider how to orchestrate between agents when the utility, constraints, and costs of each agent are known. We discuss when non-random orchestration of a given set of agents is worthwhile. We take practical steps to understand when orchestrating agents is helpful and to incorporate real world considerations into orchestration mechanisms. Our contributions are as follows: 1. In Section 2 , we lay out a framework that allows for learning agent capabilities online and orchestrates between multiple agents under real-world considerations (e.g. cost, constraints, capability requirements). 2. We describe the conditions under which orchestration of agents is sensible by proposing a metric, the appropriateness of orchestration ( Section 3 ). We show how differences in agent capabilities can affect the effectiveness of orchestration. 3. We use our orchestration framework to resolve a variant of Rogers’ Paradox in Section 4 by helping agents narrow down the learning strategies worth prioritizing. 4. We close with a human subject experiment in Section 5 and find that users are poor decision-makers when allowed to select from a set of assisting agents. We show that orchestration improves user performance. 2 A Framework for Orchestration We start by describing a framework for orchestrating between a set of agents. While many have attempted to approximate agent capability over the entire data distribution ( Gao et al., 2021 ; Charusaie et al., 2022 ; Hemmer et al., 2022 ) , we consider modeling an agent’s performance over pre-specified regions that partition the data distribution. Regions and Agents. We denote with \mathcal{D} the input dataset. We posit that the input data distribution can be partitioned into mutually-exclusive regions \mathcal{R}_{1},\dots,\mathcal{R}_{M} . For instance, in CIFAR-10 ( Krizhevsky et al., 2009 ) regions correspond to data points with different labels, whereas in MMLU ( Hendrycks et al., 2020 ) they correspond to different subjects, e.g. mathematics, philosophy, astronomy. We use the notation \mathbb{P}(\mathcal{R}_{m}) to denote the probability of observing a data sample from region \mathcal{R}_{m} . In this framework, the dataset \mathcal{D} consists of triplets \mathcal{D}=\{x_{t},y_{t},r_{t}\}_{t=1}^{N} , where x_{t}\in\mathcal{X} is the input, y_{t}\in\mathcal{Y} is the corresponding input label, and r_{t}\in\{1,\dots,M\} is the region label, indicating if x_{t} belongs to a region \mathcal{R}_{r_{t}} . Let \mathcal{A}=\{A_{1},\dots,A_{K}\} be a predefined set of agents, which we desire to orchestrate. Every agent is a function {A_{k}:\mathcal{X}\to\mathcal{Y}} , that maps the inputs x\in\mathcal{X} to outputs y\in\mathcal{Y} . For example, in the context of Human + Large Language Model (LLM) interaction, a possible set of agents is {\mathcal{A}=\{\texttt{HUMAN},\texttt{AI},\texttt{HUMAN+AI}\}} , where HUMAN denotes the decision of an unaided human, AI that of a LLM, and HUMAN+AI denotes that of a human with LLM assistance. For each agent, we denote by \mathbb{P}(A_{k}\mid\mathcal{R}_{m}) the probability of correctness of an agent A_{k} in region \mathcal{R}_{m} , i.e., the fraction of input data such that {A_{k}(x_{t})=y_{t}} . Orchestrating with Agents. Given inputs from dataset \mathcal{D} streamed one at a time, orchestration consists of selecting the optimal agent A_{k}\in\mathcal{A} to maximize correctness over the complete dataset. Notably, orchestration is different from the batch setting, where all data points are observed at once. We define the long-running correctness of an agent A_{k} as: \displaystyle\begin{split}\mathsf{C}(A_{k})&=\sum_{m=1}^{M}\mathbb{P}(\mathcal{R}_{m})\mathbb{P}(A_{k}\mid\mathcal{R}_{m}).\end{split} Next, we define the onwards correctness of an agent as at each step t\in\{1,\dots,N\} of the data stream, which we denote with \mathsf{C}_{\geq t}(A_{k}) . This quantity can be conveniently decomposed into the correctness at time step t \mathsf{C}_{>t}(A_{k}) and the future long-running correctness \mathsf{C}_{>t}(A_{k}) . Given an observed data point \{x_{t},y_{t},r_{t}\} at a fixed time step t , the correctness at time step t can be computed as \mathsf{C}_{t}(A_{k})=\mathbb{P}(A_{k}\mid\mathcal{R}_{r_{t}}) . On the other hand, the future long-running correctness \mathsf{C}_{>t}(A_{k}) can be computed as {\mathsf{C}_{>t}(A_{k})=\mathsf{C}(A_{k})} , since the future is independent of the current time-step. Then, the onwards correctness of an agent is defined as \displaystyle\begin{split}\mathsf{C}_{\geq t}(A_{k})&=\mathsf{C}_{t}(A_{k})\cdot\mathsf{C}_{>t}(A_{k}),\\ &=\mathbb{P}(A_{k}\mid\mathcal{R}_{r_{t}})\cdot\sum_{m=1}^{M}\mathbb{P}(\mathcal{R}_{m})\mathbb{P}(A_{k}\mid\mathcal{R}_{m}).\end{split} (1) Intuitively, \mathsf{C}_{\geq t}(A_{k}) provides us the utility of an selecting an agent A_{k} , which acts as a lookahead for the future probability of correctness \mathsf{C}_{>t}(A_{k}) weighed by the

Read the complete text