SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Surgical Intelligence
Chang Han Low 1, Ziyue Wang 1, Tianyi Zhang 1,2, Zhitao Zeng 1, Zhuo Zhu 1, Evangelos B. Mazomenos 3, Yueming Jin 1 🖂 · 2025
Read SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Surgical Intelligence by Chang Han Low 1, Ziyue Wang 1, Tianyi Zhang 1,2, Zhitao Zeng 1, Zhuo Zhu 1, Evangelos B. Mazomenos 3, Yueming Jin 1 🖂 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10265.txt.
Introduction
Chang Han Low and co-authors present SurgRAW, a team-of-agents system that reasons over robotic surgery videos, in a 2025 paper. Surgical AI tools tend to handle one task at a time, giving fragmented answers that clinicians cannot easily check. Big vision-language models reason more broadly but invent details and miss surgical specifics.
The team first built a benchmark of more than fourteen thousand question-answer pairs across five surgical tasks, each with step-by-step reasoning. Their system then splits scene reading among specialist agents, adds retrieved surgical knowledge, and has higher agents weigh how tasks connect. In tests it beat leading models and even a trained specialist model, while showing its reasoning for review.
Catalog record
| CREATOR | Chang Han Low 1, Ziyue Wang 1, Tianyi Zhang 1,2, Zhitao Zeng 1, Zhuo Zhu 1, Evangelos B. Mazomenos 3, Yueming Jin 1 🖂 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10265
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Surgical Intelligence Chang Han Low 1, Ziyue Wang 1, Tianyi Zhang 1,2, Zhitao Zeng 1, Zhuo Zhu 1, Evangelos B. Mazomenos 3, Yueming Jin 1 🖂 Abstract Integration of Vision-Language Models (VLMs) in surgical intelligence is hindered by hallucinations, domain knowledge gaps, and limited understanding of task interdependencies within surgical scenes, undermining clinical reliability. While recent VLMs demonstrate strong general reasoning and thinking capabilities, they still lack the domain expertise and task-awareness required for precise surgical scene interpretation. Although Chain-of-Thought (CoT) can structure reasoning more effectively, current approaches rely on self-generated CoT steps, which often exacerbate inherent domain gaps and hallucinations. To overcome this, we present SurgRAW, a CoT-driven multi-agent framework that delivers transparent, interpretable insights for most tasks in robotic-assisted surgery. By employing specialized CoT prompts across five tasks: instrument recognition, action recognition, action prediction, patient data extraction, and outcome assessment, SurgRAW mitigates hallucinations through structured, domain-aware reasoning. Retrieval-Augmented Generation (RAG) is also integrated to external medical knowledge to bridge domain gaps and improve response reliability. Most importantly, a hierarchical agentic system ensures that CoT-embedded VLM agents collaborate effectively while understanding task interdependencies, with a panel discussion mechanism promoting logical consistency. To evaluate our method, we introduce SurgCoTBench, the first reasoning-based dataset with structured frame-level annotations. With comprehensive experiments, we demonstrate the effectiveness of proposed SurgRAW with 29.32% accuracy improvement over baseline VLMs on 12 robotic procedures, achieving SOTA performance and advancing explainable, trustworthy, and autonomous surgical assistance. Dataset and code is available at https://github.com/jinlab-imvr/SurgRAW. 1 Introduction Surgical scene understanding is fundamental to modern surgical intelligence. It facilitates real-time monitoring of procedures, enhances intra-operative support, and ultimately leads to improved delivery and patient outcomes. Due to the intrinsic complexity of surgical videos, a comprehensive analysis requires observations from multiple perspectives, encompassing actions, instruments, and patient-related factors. In the operating room, surgical teams are assigned with specific roles, systematically reasoning through each procedural step and collaboratively contributing to a detailed, overall analysis [ 21 ] . This process is labor-intensive and time-consuming, automation of intraoperative decision via computer intelligence presents considerable potential for improving efficiency [ 28 ] . Multiple AI-driven approaches [ 8 , 9 , 29 ] have been developed to address these issues. These models tackle specific tasks within the surgical workflow analysis, including workflow recognition, instrument recognition, and error detection [ 25 , 2 , 3 , 22 , 8 , 26 , 6 , 31 ] . However, these methods employ end-to-end networks to generate results without providing further explanations, limiting interpretability. Additionally, model architectures for different tasks vary significantly, necessitating careful design and task-specific data collection for training. These challenges hinder the development of a unified and explainable system for surgical intelligence. Recently, the advancement of general Large Language Models (LLMs) has garnered attention due to their strong reasoning capabilities [ 1 ] . LLMs can perform various tasks based on human prompts while providing reasoning processes and can further enhance response quality when additional information is included in the prompts. Building on LLMs, VLMs expand their capabilities to process visual inputs and have demonstrated remarkable performance across diverse applications, highlighting their potential in surgical scene understanding. However, the direct application of VLMs to the surgical domain remains challenging : i) VLMs are highly prone to hallucinations, with responses varying significantly depending on prompt content, even for the same question, compromising their reliability. ii) Primarily trained on natural scenarios, VLMs lack surgical knowledge, often leading to inaccurate response and further exacerbating the hallucinations [ 31 ] . iii) VLMs handle different surgical tasks independently, ignoring their interconnections within the overall workflow. As a result, VLMs perform poorly when the surgical tasks are given as simple multiple-choice questions (MCQs). Some approaches [ 15 , 13 ] attempt to fine-tune VLMs for specific domains; however, the surgical domain lacks the massive high-quality data required for effective fine-tuning. To mitigate these limitations, approaches like LLaVA-COT [ 30 ] employ LLMs to generate detailed CoT as prompts to enhance responses with zero-shot VLMs. Nonetheless, without domain-specific knowledge, these auto-generated CoTs remain unreliable in surgical contexts, still requiring further research to integrate VLMs into surgical intelligence. Figure 1: The overall structure of SurgRAW processes surgical queries through hierarchical orchestrators and CoT-embedded expert agents, with RAG and panel discussions enhancing accuracy and domain reliability. To address these issues, we propose a R easoning multi- A gent W orkflow for Surg ical intelligence, termed SurgRAW. Mirroring real-world surgical teamwork, SurgRAW assigns different tasks in surgical scene understanding to specialized VLM agents through hierarchical orchestrators and integrates their responses to refine the final answer. Specifically, rather than fine-tuning VLMs or relying on LLM-generated CoTs, we design task-specific CoT prompts tailored to each task. These prompts help mitigate hallucinations by ensuring robust and interpretable reasoning. By guiding step-by-step logical deductions, they enhance both response accuracy and explainability. Additionally, we integrate RAG tools to bridge the domain gap, ensuring that decisions are grounded in professional, domain-specific surgical resources, further improving the system’s reliability. Furthermore, SurgRAW introduces a novel panel discussion mechanism which initiates multi-agent collaboration and debate. By verifying the consistency between responses and intermediate steps across different agents, SurgRAW facilitates rich interactions among agents, enabling a more comprehensive analysis of surgical videos. Overall, our contribution can be summarized as the following aspects: • We introduce SurgRAW, the first framework capable of addressing all major questions across the entire surgical workflow while enhancing explainability. • Mirroring the cognitive process of surgeons, we enhance VLM agents with task-specific CoT prompts, enabling transparent and step-by-step analysis. • A panel discussion mechanism and a RAG module are further introduced, enabling agents to share insights and generate reliable responses. • We introduce a new benchmark to evaluate our framework, which is the first benchmark to encompass all tasks within the same surgical scenario. 2 Methodology 2.1 Surgical Multi-Agent Workflow Surgical scene understanding requires both visual-semantic perception and cognitive-inference reasoning, each with distinct processing needs. Tasks within the surgical workflow fall into two types: visual-semantic tasks, which involve perceptual scene analysis for fine-grained action and instrument recognition, and cognitive-inference tasks, which rely on procedural reasoning and deeper contextual understanding to support decision-making. To effectively address these tasks, we propose SurgRAW to handle each task efficiently, as illustrated in Fig. 1 . SurgRAW employs a hierarchical orchestration system, where the first orchestrator, the Department Coordinator, first classifies each query as visual-semantic or cognitive-inference. A secondary orchestrator, either the Visual-Semantic Department Head or the Cognitive-Inference Department Head, directs the task to the granular task-level agent. This work investigates five crucial tasks in surgical workflow, containing five corresponding agents: Action Interpreter, Action Predictor, Instrument Specialist, Outcome Analyst, and Patient Advocate. To further enhance reasoning transparency and reliability, we integrate explicit reasoning capabilities into the core agents, transforming them into CoT-embedded VLM agents that generate transparent, evidence-based responses. Additionally, an evaluator agent is designed for panel discussion, which verifies surgical actions and instrument usage, facilitating complex scene interpretation in visual-semantic tasks. Meanwhile, SurgRAW enhances cognitive-inference tasks by incorporating RAG to provide more context-aware recommendations an refine decision-making. Detailed designs are introduced in the sections below. 2.2 Chain-of-Thought Prompt Generation Surgeons approach surgical scene analysis