Surgical Agent Orchestration Platform for Voice-directed Patient Data Interaction

Hyeryun Park, Byung Mo Gu, Jun Hee Lee, Byeong Hyeon Choi, Sekeun Kim, Hyun Koo Kim, Kyungsang Kim · 2025

Read Surgical Agent Orchestration Platform for Voice-directed Patient Data Interaction by Hyeryun Park, Byung Mo Gu, Jun Hee Lee, Byeong Hyeon Choi, Sekeun Kim, Hyun Koo Kim, Kyungsang Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07392.txt.

Introduction

Hyeryun Park and six co-authors give surgeons a voice-controlled agent for patient data in the operating room. During robotic surgery a surgeon's hands and eyes are fully committed, so pulling up clinical information or manipulating CT scans normally means breaking workflow. Their system, VISA, is a hierarchy of language-model agents: an orchestration agent coordinates three task experts that plan, refine, validate. Reason over voice commands to retrieve records, adjust scans, or navigate 3D anatomical models in the surgical video.

The team built a dataset of two hundred forty commands in hierarchical categories and a two-level evaluation metric scoring both single commands and categories. VISA achieved high stage-level accuracy and workflow success, and proved robust by correcting transcription errors, resolving ambiguity, and handling free-form phrasing from different speakers.

Catalog record

CREATORHyeryun Park, Byung Mo Gu, Jun Hee Lee, Byeong Hyeon Choi, Sekeun Kim, Hyun Koo Kim, Kyungsang Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07392

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Surgical Agent Orchestration Platform for Voice-directed Patient Data Interaction Hyeryun Park, Byung Mo Gu, Jun Hee Lee, Byeong Hyeon Choi, Sekeun Kim, Hyun Koo Kim, Kyungsang Kim Abstract In da Vinci robotic surgery, surgeons’ hands and eyes are fully engaged in the procedure, making it difficult to access and manipulate multimodal patient data without interruption. We propose a voice-directed Surgical Agent Orchestrator Platform (SAOP) built on a hierarchical multi-agent framework, consisting of an orchestration agent and three task-specific agents driven by Large Language Models (LLMs). These LLM-based agents autonomously plan, refine, validate, and reason to map voice commands into specific tasks such as retrieving clinical information, manipulating CT scans, or navigating 3D anatomical models on the surgical video. We also introduce a Multi-level Orchestration Evaluation Metric (MOEM) to comprehensively assess the performance and robustness from command-level and category-level perspectives. The SAOP achieves high accuracy and success rates across 240 voice commands, while LLM-based agents improve robustness against speech recognition errors and diverse or ambiguous free-form commands, demonstrating strong potential to support minimally invasive da Vinci robotic surgery. 1 Introduction In robot-assisted surgery using the da Vinci surgical system, the primary surgeon must concentrate with their hands and eyes fully engaged in the surgical procedure. However, accessing and manipulating patient data during the procedure requires the surgeon to shift attention away from the surgeon console and use another monitor or auxiliary interfaces such as TilePro (Intuitive Surgical). These interruptions during surgery become more significant when surgeons need to check various types of multimodal data, including clinical information, CT images, MRI images, and 3D models. Such demands underscore the need for a dynamic interaction system to support minimally invasive robotic surgery. Early interaction studies on voice-controlled robotic surgery mainly focused on camera and equipment manipulation using systems such as AESOP allaf1998laparoscopic ; mettler1998one ; reichenspurner1999use ; nathan2006voice , the HERMES operating room control center salama2005utility , and PARAMIS vaida2010development . These systems were shown to reduce operation time and minimize the need for a second assistant nathan2006voice , while improving surgical efficiency and reducing communication burden among the operating room staff salama2005utility . Subsequent research extended the scope to a broader range of tasks, including image retrieval and 3D distance measurement forte2022design , real-time camera tracking and zoom control elazzazi2022natural ; kim2025speech , and tissue manipulation through robotic assistant arms davila2024voice . Recent studies have progressed toward interpreting natural language commands utilizing Large Language Models (LLMs). A ChatGPT-integrated system interpreted free-form user speech, and matched it to the most relevant predefined command pandya2023chatgpt . In addition, LLM-integrated mixed reality systems enable natural language–driven robot operation zhang2025llm and real-time surgery assessment and procedure guidance gavric2025surgery . LLMs exhibit advanced capabilities for understanding and generating natural language and such models include GPT-4 Touvron2023 , PaLM Chowdhery2023 , LLaMA Touvron:2023:LLaMA ; Touvron:2023:LLaMA2 ; Dubey:2024:LLaMA3 ; Meta:2025:LLaMA4 , and Gemma Gemma:2024 ; Gemma2:2024 ; Gemma3:2025 . These models can interpret natural instructions and perform multi-step problem solving through advanced prompting and reasoning techniques, including Chain-of-Thought (CoT) Wei2022 , Self-Consistency CoT wang2022scCoT , Tree-of-Thought yao2023ToT , and Graph-of-Thought besta2024GoT These advancements in reasoning have transformed LLMs from language generators into autonomous agents that can perceive their environment, plan to achieve specific goals, and act accordingly franklin1996agent ; russell1995modern ; huang2025introduction . Various frameworks such as ReAct yao2022react , AutoGPT, BabyAGI, AutoGen wu2024autogen , and CAMEL li2023camel have been proposed to integrate reasoning and action. Recent research further applies reinforcement learning to help LLMs learn from past reasoning failures and optimize their decision-making zhang2025survey . These diverse reasoning and interaction methods have become the driving force of LLM-based autonomous agents capable of planning, tool use, and collaborative problem solving. The LLM-based agent typically consists of five key modules: a profile module that defines objectives and constraints, a memory module that stores past states and actions, a reasoning and planning module that formulates strategies for goal achievement, an action module that executes tasks via models, tools, or APIs, and a feedback module that evaluates outcomes and refines future behavior wang2024survey . These agents can operate as a single-agent system using one LLM or as a multi-agent system (MAS) that utilizes specialized agents for distinct tasks, offering advantages in scalability, efficiency, and coordinated workflows li2024survey ; zhao2025llm . The coordination in MAS is enabled by LLM-based orchestration, where an LLM functions as the cognitive core of the system to manage interactions among agents, planning execution sequences, interpreting intermediate results, and controlling the overall workflow tran2025multi ; rasal2024navigating ; du2025multi ; dang2025multi ; zhang2025osc . Such MAS systems can be classified as centralized, decentralized, or hierarchical, exhibiting cooperative, competitive, or negotiated behaviors zhao2025llm ; tran2025multi . To implement such orchestration in practice, various agent frameworks have been developed, including LangGraph LangGraph2024 , AutoGen wu2024autogen , LlamaIndex Liu_LlamaIndex_2022 , CrewAI, and AgentKit. Figure 1: Overall framework of the voice-directed Surgical Agent Orchestrator Platform (SAOP). a Workflow Orchestrator Agent that autonomously plans the order of function execution. b Workflow functions that capture, transcribe, correct, and interpret voice commands. c Task-specific agent functions that overlay and manipulate patient data on the target video clip. d Memory state with local memory states for each video clip and a global memory state for shared contextual understanding across frames. In this work, we highlight the importance of effective interaction between surgeons and surgical tools during procedures, with voice interaction offering a promising approach. Based on this motivation, we modularize key tool functionalities into specialized agents and coordinate them through an orchestrator. Specifically, we introduce a voice-directed Surgical Agent Orchestrator Platform (SAOP) that interprets free-form voice commands to enable context-aware, multimodal data interaction directly on surgical videos. To our knowledge, this is the first hierarchical multi-agent framework, consisting of workflow orchestrator agent and multiple task-specific agents as demonstrated in Fig. 1 . In contrast to previous research that relied on predefined commands or a single LLM for direct mapping between voice commands and functions, our SAOP autonomously plans and executes functions through intermediate steps, providing more flexible and robust mapping for complex voice commands. The workflow orchestrator agent uses an LLM to perform probabilistic decision-making across workflow functions and task-specific agent functions, predicting which function to execute next. Within workflow functions, real-time audio function obtains voice input, Speech-to-Text (STT) function transcribes it, correction and validation function uses LLM to revise the command and determine its validity, and the command reasoning function leverages LLM to determine the appropriate task agent. The Information Retrieval (IR) agent retrieves relevant clinical information, the Image Viewer (IV) agent slides through axial, coronal, and sagittal planes of CT scans with zoom operations, and the Anatomy Rendering (AR) agent manages 3D anatomical models through structure selection and view selection, rotation, and zoom control. Each task-specific agent includes an action determination function that leverages an LLM to infer the appropriate action and its parameters. The SAOP also utilizes a memory state that stores previous commands, agents, and various parameters for continuous task execution and adaptive responses to ambiguous commands. To evaluate the SAOP performance, we construct a command dataset consisting of 240 user commands, each annotated with three hierarchical categories: command structure (single or composite), command type (explicit, implicit, or natural language question), and command expression

Read the complete text