One Supervisor, Many Modalities: Adaptive Tool Orchestration for Autonomous Queries
Mayank Saini † Arit Kumar Bishwas † PwC US PwC US [email protected] [email protected] · 2026
Read One Supervisor, Many Modalities: Adaptive Tool Orchestration for Autonomous Queries by Mayank Saini † Arit Kumar Bishwas † PwC US PwC US [email protected] [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11545.txt.
Introduction
Saini and Bishwas present an adaptive framework that answers multimodal queries by orchestrating tools under one supervisor. The supervisor splits each query and sends subtasks to tools fit for each modality, then merges results. Text-only queries use learned routing via RouteLLM; other paths lean on small-model decomposition. On 2,847 queries across 15 task types, time to a correct answer falls 72 percent. Rework drops 85 percent and cost 67 percent against a matched baseline, with accuracy held even.
Catalog record
| CREATOR | Mayank Saini † Arit Kumar Bishwas † PwC US PwC US [email protected] [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73388 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11545
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
One Supervisor, Many Modalities: Adaptive Tool Orchestration for Autonomous Queries Mayank Saini † Arit Kumar Bishwas † PwC US PwC US [email protected] [email protected] Abstract We present an agentic AI framework for autonomous multimodal query processing that coordinates specialized tools across text, image, audio, video, and document modalities. A central Supervisor dynamically decomposes user queries, delegates subtasks to modality-appropriate tools (e.g., object detection, OCR, speech transcription), and synthesizes results through adaptive routing strategies rather than predetermined decision trees. For text-only queries, the framework uses learned routing via RouteLLM, while non-text paths use SLM-assisted modality decomposition. Evaluated on 2,847 queries across 15 task categories, our framework achieves 72% reduction in time-to-accurate-answer, 85% reduction in conversational rework, and 67% cost reduction compared to the matched hierarchical baseline while maintaining accuracy parity. These results demonstrate that intelligent centralized orchestration fundamentally improves multimodal AI deployment economics. Keywords: Agentic AI AI Agents Multimodal AI Multimodal Reasoning AI Orchestration Tool Orchestration Tool-Augmented LLMs Cost-Aware Inference LLM Routing Multi-Agent Systems Autonomous Query Processing 1 Introduction Modern AI deployment confronts a critical challenge in reconciling conflicting requirements that users demand from production systems: the ability to autonomously process any query type ranging from simple text instructions to complex multimodal requests involving images, audio, video, and structured documents, while simultaneously maintaining operational cost-efficiency at scale and delivering real-time responsiveness suitable for interactive applications. Current solutions prove inadequate along multiple dimensions. Monolithic large language model deployments that route all queries to a single powerful model such as GPT-4 [ 15 ] or Gemini Ultra [ 5 ] incur prohibitive operational costs when applied uniformly across heterogeneous workloads, particularly given that the vast majority of real-world queries do not require the full reasoning capacity of frontier models [ 4 , 2 ] . Conversely, hierarchical routing systems that attempt to direct queries through predetermined decision trees based on explicit classification rules exhibit catastrophic brittleness when queries deviate from anticipated patterns—requiring complete pipeline restarts that waste computational resources already invested in partial execution, creating unacceptable latencies for time-sensitive applications, and producing frustrating user experiences that undermine system trust and adoption. Our earlier work on resource-efficient multimodal intelligence [ 20 ] established the fundamental viability of coordinated processing across diverse modalities through conditional orchestration strategies that route queries based on detected input characteristics. However, that mechanistic approach relied on manually specified routing logic that required explicit enumeration of all anticipated query types and their associated processing paths, creating substantial maintenance burdens as new capabilities were added and producing brittle failure modes when real-world queries fell outside the anticipated design space, as illustrated in Figure 1 . When users submitted queries that did not match any predetermined pattern—whether due to novel phrasing, unexpected modality combinations, or edge cases not considered during system design—the conditional orchestration framework lacked any mechanism for graceful degradation or adaptive response, instead failing entirely and forcing users to manually reformulate their requests. This fundamental limitation arises directly from the architectural decision to embed all routing intelligence within predetermined decision trees rather than enabling the system to reason about and adapt to query characteristics autonomously based on learned patterns and contextual understanding. We introduce a centralized orchestration framework that reimagines multimodal query processing through intelligent coordination of specialized tools. The framework employs a central Supervisor that reads tool specifications expressed as typed interfaces with preconditions, postconditions, and latency priors, makes contextual routing decisions based on query characteristics and historical memory state, and dynamically decomposes tasks for delegation to appropriate tools. This approach addresses the limitations of both monolithic systems (which process all queries uniformly at high cost) and hierarchical routing (which relies on predetermined decision trees). For perceptual workloads, the Supervisor routes to domain-optimized tools (e.g., detection, embedding, OCR, transcription) and then contextualizes their outputs for downstream synthesis. We validate our framework through comprehensive evaluation across 2,847 queries spanning 15 distinct task categories drawn from standard benchmarks. Compared to a matched hierarchical baseline, our approach achieves 72% median reduction in time-to-accurate-answer with interquartile range of 65–77%, 85% reduction in conversational rework requiring user clarification or correction, 67% reduction in expensive model invocations, and 20% improvement in concurrent throughput (54 vs 45 q/s) under realistic load conditions, all while maintaining accuracy parity within statistical variance. Perceptual tasks benefit from routing to specialized models (e.g., object detection in 180 ms/frame) instead of end-to-end LLM vision approaches (e.g., 2.4 s/frame), reducing latency and cost while preserving accuracy. These results establish that intelligent orchestration through composable supervision can fundamentally reshape the economics and scalability of multimodal AI deployment without sacrificing response quality or system reliability. Figure 1: Architectural paradigm comparison showing the transition from scripted routing with brittle global restarts to Supervisor-driven adaptive orchestration with tool and model pools. 2 Related Work Our work intersects with four active research areas: agentic workflows for tool coordination, multi-agent orchestration frameworks, cost-aware model routing strategies, and multimodal intelligence systems. Recent advances demonstrate large language models’ growing capability to interact with external tools and environments through learned action selection. ReAct [ 28 ] establishes a foundation by interleaving reasoning traces with environment actions to reduce hallucination in complex decision-making tasks. Toolformer [ 21 ] extends this capability by learning when to invoke external APIs through self-supervised training on annotated demonstrations. MRKL [ 8 ] provides theoretical foundations for composing neural and symbolic modules within unified architectures, demonstrating that hybrid systems can leverage complementary strengths of learned and engineered components. HuggingGPT [ 22 ] positions large language models as meta-controllers that decompose user requests into subtasks, dispatch each subtask to appropriate specialist models from repositories like Hugging Face, and synthesize partial results into coherent final responses. Concurrent model-centric multimodal systems, including GPT-4, Gemini, and open vision-language models such as LLaVA, substantially improve end-to-end capability but do not directly address orchestration-level cost and repair policies [ 15 , 5 , 11 ] . While these approaches demonstrate effective tool utilization, they maintain fixed orchestration architectures where the coordination logic remains embedded within the system. Our framework builds on these concepts by implementing a flexible centralized orchestrator that coordinates specialized tools through context-conditioned routing strategies rather than predetermined decision trees. Multi-agent orchestration frameworks provide infrastructure for coordinating multiple specialized components within complex workflows. AutoGen [ 26 ] introduces conversational patterns where agents negotiate through structured dialogue to resolve ambiguous requests and dynamically adjust execution strategies based on intermediate results. LangGraph [ 9 ] extends this coordination through graph-based workflow specifications that support branching, conditional execution, and cyclic dependencies. MegaAgent [ 25 ] demonstrates large-scale coordination across hundreds of specialized agents without requiring predefined standard operating procedures, instead relying on learned negotiation protocols. Recent surveys [ 13 , 27 ] highlight rapid progress in agent reasoning, planning capabilities, and the emergence of increasingly sophisticated coordination protocols. ConAgents [ 23 ] introduces cooperative tool learning where agents jointly discover optimal tool combinations through multi-agent reinforcement learning. However, existing frameworks maintain hierarchical architectures where a central controller makes coordination decisions. Our centralized orchestration approach implements an intelligent orchestrator that dynamically decomposes tasks, delegates to specialized tools, and synthesizes results—enabling parallel processing, adaptive tool selection, and local failure recovery through context-conditioned routing rather than predetermined workflows. Cost-aware routing and model selection strategies address the economic challenges of