OrchVis: Hierarchical Multi-Agent Orchestration for Human Oversight

Jieyu Zhou · 2025

Read OrchVis: Hierarchical Multi-Agent Orchestration for Human Oversight by Jieyu Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24937.txt.

Introduction

Jieyu Zhou, the sole author, built OrchVis, a frame to let humans watch and steer teams of AI agents as they chase a goal. Such teams can split work, but a human lead can lose sight of who does what. OrchVis lays out goals in a stack, gives out tasks, and sorts clashes as they come.

User aims are parsed into set goals, and an auto check step tracks each run. A live plan pane shows how agent tasks hang off each other. When a clash comes, the user can scan the system's own fix picks and replan just the parts they pick. The aim is close watch with no micro rule of each step.

Catalog record

CREATORJieyu Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE13 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24937

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OrchVis: Hierarchical Multi-Agent Orchestration for Human Oversight Jieyu Zhou Abstract . We introduce OrchVis , a multi-agent orchestration framework that visualizes, verifies, and coordinates goal-driven collaboration among LLM-based agents. Through hierarchical goal alignment, task assignment, and conflict resolution, OrchVis enables humans to supervise complex multi-agent workflows without micromanaging each step. The system parses user intent into structured goals, monitors execution via automated verification, and exposes inter-agent dependencies through an interactive planning panel. When conflicts arise, users can explore system-proposed alternatives and selectively replan. OrchVis advances human-centered design for multi-agent systems by combining transparent visualization with adaptive autonomy. Figure 1. System overview of our multi-agent, goal-aware visualization. (1) In the planning phase, the user specifies the task; the orchestration/supervisor agent interprets the intent and decomposes it into structured subgoals. (2) During execution, the UI tracks and displays per-goal progress. (3) When inter-goal conflicts arise, they are flagged in situ and can be expanded into the Planning Panel. (4) The Planning Panel reveals how the orchestrator routes work to sub-agents and pinpoints which sub-agents are in conflict. (5) A summary pane presents high-level rationales, proposed resolutions, and predicted future states to help the user make informed decisions and set the desired level of autonomy. 1. Introduction Most commercial AI agents today execute tasks end-to-end with little to no user oversight ( OpenAI 2025 ; Anthropic 2024 ; Manus 2024 ) . In single-agent settings, HCI work has introduced a broad toolkit for human involvement in planning and execution ( Shao et al. 2024 ; Das Antar et al. 2024 ; Zhao et al. 2024 ; Suh et al. 2024 ; Feng et al. 2024 ; Lawley and MacLellan 2024 ; Epperson et al. 2025 ; Kazemitabaar et al. 2024 ) , such as co-planning and co-execution of research tasks ( Feng et al. 2024 ) , pausing and editing agent behaviors in programming ( Epperson et al. 2025 ) , and decomposing complex data analysis workflows for verification By contrast, user oversight in multi-agent scenarios remains underexplored: most prior work emphasizes technical orchestration frameworks ( Wu et al. 2024 ; Fourney et al. 2024 ) or quantifies how human involvement scales with agent collectives ( Kolling et al. 2016 ; Clarke et al. 2024 ) , with comparatively few system-level designs that make coordination legible and controllable to end users. One notable exception is CommunityBot ( Jiang et al. 2023 ) , which focuses on socially oriented chat; however, it does not address the challenges of supervising multi-agent collaboration on complex task execution. In this paper, we present OrchVis, a visualization and orchestration framework that enables users to oversee, understand, and correct multi-agent collaborations through hierarchical goal alignment, layered visualization, and conflict resolution. Building on insights from prior HCI work on mixed-initiative interaction, OrchVis rethinks how users can strategically guide multiple agents without micromanaging them. We first derive three design goals for enabling effective human oversight in multi-agent systems, then introduce the system design and technical implementation that realize these goals. 2. Design Goals Based on prior research, we conclude three design goals for the multi-agent system. 2.1. From Direct Manipulation to Orchestration. There has been a long-standing debate in HCI: should agents operate under direct human supervision and control, or should they function in a mostly autonomous fashion ( Horvitz 1999 ; Shneiderman and Maes 1997 ; Lieberman 1997 ; Maes 1995 ) ? However, for complex tasks involving multiple agents, direct manipulation quickly becomes infeasible. Kolling et al. 2016 introduced the concept of human cognitive complexity, showing that directly coordinating k sub-group agents scales at least as O(k) in user effort, which grows rapidly as the number of agents increases. In contrast, an orchestration agent that supervise sub agents can substantially reduce this complexity . User studies further confirm that users consistently prefer orchestration over direct manipulation of multiple agents ( Clarke et al. 2024 ) . Some research ( Moran et al. 1997 ) has conceptualized the human as simply another agent within the system. However, we argue that the human should take the lead, acting at a higher level of abstraction, supervising with autonomy. Design Goal 1. Introduce an orchestration agent that supervises sub-agents and consolidates low-level information into high-level summaries for human oversight. The human, in turn, should operate at a strategic level rather than managing procedural details. 2.2. Separate Goals from Tasks and Align Early. We distinguish two forms of planning organization following prior work: goals and tasks (e.g., hierarchical task networks ( Nau et al. 2003 ) , and hierarchical goal networks ( Shivashankar et al. 2013 ) ). A goal is a desired target state—a set of constraints that should hold in the world (the what). A task is an executable procedure—a sequence of actions that can make those constraints true (the how). For example, in flight purchasing, “have a confirmed ATL→SFO ticket under $400” is a goal; searching flights, applying coupon, and checkouting are subtasks that may achieve it. Misalignment between the user’s intended goal and the agent’s inferred goal is common ( Bansal et al. 2024 ; Coscia et al. 2025 ) , caused by vague user inputs and model interpretation drift ( Li et al. 2020 ) . Detecting such misalignment early prevents costly derailments during execution ( Zhou et al. 2025 ) . In practice, users focus on outcomes first (goal satisfaction) and only examine process details (tasks) when results appear incorrect ( Zhou et al. 2025 ) . To reach the first design goal of showing right amount of information to users, unnecessary information-such as action sequences when the goal has benn achieved-should remain hidden. Design Goal 2. Before executing the task, ensure both users and agents agree on the goal. During execution, emphasize goal achievement as the primary view, while task-level details are revealed only when users need to inspect or correct errors. 2.3. Visualizing Inter-Agent Relationship The relationships between agents are often complex, essentially reflecting the intricate dependencies among subtasks within a larger task. Prior research has characterized task relationships as sequential, parallel, nested, or hierarchical ( Wu et al. 2024 ) . Agents may also engage in debate, which has been shown to improve overall task success rates ( Du et al. 2023 ) . Meanwhile, prior work has found that text-heavy interfaces reduce users’ flexibility to revise or provide feedback, motivating the design of GUI-first interfaces for single-agent systems ( Feng et al. 2024 ; Arawjo et al. 2024 ) . For multi-agent systems, where dependencies and interactions among agents are even more intricate, GUI-first visualization becomes especially crucial for helping users understand and oversee agent relationships. Design Goal 3. Visualize inter-agent dependencies and conflicts through a GUI to support user oversight. 3. OrchVis: System and Technical Design OrchVis visualizes and coordinates multi-agent planning through a layered interface that balances automation with user oversight. Using a travel-planning scenario (Fig. 2 ), the system proceeds through four tightly coupled stages—goal alignment, task assignment, progress tracking, and conflict resolution—each supported by specific technical components (goal parser, orchestrator, verifier, and re-planner). Together, these stages enable users to monitor, diagnose, and revise multi-agent workflows while maintaining transparency and control across hierarchical goals and actions. 3.1. Goal Alignment and Parsing

Read the complete text