SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark
Gui Wang, YongSong Zhou, Kaijun Deng, Wooi Ping Cheah, Rong Qu, Jianfeng Ren, Linlin Shen · 2026
Read SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark by Gui Wang, YongSong Zhou, Kaijun Deng, Wooi Ping Cheah, Rong Qu, Jianfeng Ren, Linlin Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20319.txt.
Introduction
Wang and six colleagues present SurgCoT, a test bed for chain of thought reasoning in surgery videos across 7 specialties and 35 procedures. Five test angles cover causal action order, cue action links, affordance maps, micro shift spots, and anomaly start tracking, scored through a question option knowledge clue answer scheme. Knowledge gives background context and clues give hard space-time evidence. Tests on 10 leading models show shop models ahead of open source and medical rivals, with wide gaps in surgery reasoning and room to sharpen step-by-step reasoning.
Catalog record
| CREATOR | Gui Wang, YongSong Zhou, Kaijun Deng, Wooi Ping Cheah, Rong Qu, Jianfeng Ren, Linlin Shen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36176 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.20319
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark Gui Wang, YongSong Zhou, Kaijun Deng, Wooi Ping Cheah, Rong Qu, Jianfeng Ren, Linlin Shen Abstract Fine-grained spatiotemporal reasoning on surgical videos is critical, yet the capabilities of Multi-modal Large Language Models (MLLMs) in this domain remain largely unexplored. To bridge this gap, we introduce SurgCoT , a unified benchmark for evaluating chain-of-thought (CoT) reasoning in MLLMs across 7 surgical specialties and 35 diverse procedures . SurgCoT assesses five core reasoning dimensions: Causal Action Ordering, Cue–Action Alignment, Affordance Mapping, Micro‑Transition Localization, and Anomaly Onset Tracking, through a structured CoT framework with an intensive annotation protocol ( Question→Option→Knowledge→Clue→Answer ), where the Knowledge field provides essential background context and Clue provides definitive spatiotemporal evidence. Evaluation of 10 leading MLLMs shows: 1) commercial models outperform open-source and medical-specialized variants; 2) significant gaps exist in surgical CoT reasoning; 3) SurgCoT enables effective evaluation and enhances progressive spatiotemporal reasoning. SurgCoT provides a reproducible testbed to narrow the gap between MLLM capabilities and clinical reasoning demands. Code: https://github.com/CVI-SZU/SurgCoT. Figure 1 : SurgCoT comprises 2,841 surgical videos across 7 specialties and 35 procedures , with 19,345 main questions and 59,177 sub-questions . SurgCoT advances beyond frame-level tasks ( e.g. , phase/tool recognition) by introducing a three-stage, five-tuple annotation protocol ( Question→Option→Knowledge→Clue→Answer ) to scaffold chain-of-thought reasoning. The framework’s efficacy stems from its multi-stage reasoning structure and the synergistic interaction between the Knowledge field, which supplies contextual background, and the Clue field, which provides targeted spatiotemporal evidence, jointly enabling hierarchical reasoning. † † footnotetext: *Corresponding author 1 Introduction Surgical videos are a cornerstone of perioperative care and education, rich with dynamic anatomical and procedural data 49 ; 31 . The growing volume and complexity of this data make fine-grained spatiotemporal understanding critical for clinical safety and efficiency 20 . In recent years, Multi-modal Large Language Models (MLLMs) have been adapted for surgical settings, enabling interactive analysis for tasks like surgical phase recognition 35 ; 55 ; 42 , instrument recognition 18 ; 9 , tissue detection 18 ; 9 , and surgery understanding 10 ; 8 ; 13 . This rapid progress, however, creates a parallel need for robust evaluation frameworks to assess their true clinical utility. Surgical benchmarks fall into two categories: General-purpose benchmarks 51 ; 35 , covering broad scenarios but emphasize generic question answering ( e.g. , phases, instruments) while specialized benchmarks 23 ; 52 focus on narrow procedural domains ( e.g. , ophthalmic, endoscopic) but rely on frame/clip-level VQAs that treat videos as discrete segments, ignoring cross-temporal dependencies and failing to evaluate spatiotemporal or causal reasoning 50 ; 56 . In contrast, surgeons must track subtle, rapid spatiotemporal changes to support fine-grained inference and decision-making 29 ; 38 . A critical question remains: Can MLLMs achieve expert-level progressive spatiotemporal reasoning and causal judgment across diverse surgical contexts? Table 1 : Comparison of surgical benchmarks. Our SurgCoT uniquely spans 7 surgical specialties with multi-level annotations (video/clip/frame), supporting hierarchical spatiotemporal reasoning with localization supervision and clinician-derived reference standards. Benchmark Domain #Spe. #Pro. Scale Unit ST Pro. Loc. Clin. Surgical-VQA 36 {}_{\text{(MICCAI'21)}} Endoscope \sim 3 11.8K Frame ✗ ✗ ✓ ✓ EndoVis-VQLA 3 {}_{\text{(ICRA'23)}} Endoscope \sim 1 9.5K Frame ✗ ✗ ✓ ✓ Endo-FM 46 {}_{\text{(MICCAI'23)}} Endoscope \sim 2 33K+50K Clip/Frame ✗ ✗ ✓ ✓ CoPESD 43 {}_{\text{(arXiv'24)}} Endoscope \sim 1 17.7K Frame ✗ ✓ ✓ ✓ SurVLP 52 {}_{\text{(MedIA'25)}} Endoscope \sim 11 25.5K Frame ✗ ✗ ✗ ✗ Surg-3M 5 {}_{\text{(arXiv'25)}} Endoscope \sim 35 30K Frame ✗ ✗ ✗ ✗ Surg-396K 42 {}_{\text{(MIA'26)}} Endoscope 3 3 396K Frame ✗ ✗ ✓ ✓ Cholec80-VQA 36 {}_{\text{(MICCAI'22)}} Laparoscopic 1 1 43K Frame ✗ ✗ ✗ ✓ GSViT 34 {}_{\text{(arXiv'24)}} Laparoscopic 1 28 7000K Frame ✗ ✗ ✗ ✗ GenSurg+ 15 {}_{\text{(arXiv'24)}} Laparoscopic 2 28 17K Clip ✗ ✗ ✗ ✗ MedVidQA 12 {}_{\text{(BioNLP@ACL'24)}} Instructional \sim \sim 1.2K Clip ✗ ✗ ✗ ✓ Surg-QA 23 {}_{\text{(arXiv'24)}} Instructional \sim \sim 10.2K Video ✗ ✗ ✗ ✗ M 3 -Med 28 {}_{\text{(arXiv'25)}} Instructional \sim \sim 3.7K Clip ✗ ✗ ✗ ✓ OphNet 16 {}_{\text{(ECCV'24)}} Ophthalmic 1 \sim 14K Video ✗ ✓ ✓ ✓ OphVL 17 {}_{\text{(arXiv'24)}} Ophthalmic 1 \sim 375K Clip ✗ ✗ ✗ ✓ EyePCR 44 {}_{\text{((arXiv'25)}} Ophthalmic 1 \sim 210K Clip/Frame ✗ ✓ ✗ ✓ PSI-AVA-VQA 35 {}_{\text{(MICCAI'23)}} Generalist 1 1 10.3K Frame ✗ ✗ ✓ ✓ SSG-VQA 51 {}_{\text{(IJCARS'24)}} Generalist 1 1 960K Frame ✗ ✗ ✓ ✓ SurgVLM-Bench 53 {}_{\text{(arXiv'25)}} Generalist 4 16 7798.4K Frame ✗ ✓ ✓ ✗ MedFrameQA 50 {}_{\text{(arXiv'25)}} Generalist 9 \sim 3.4K Multi-Frame ✓ ✗ ✗ ✓ SurgBench 47 {}_{\text{(arXiv'25)}} Generalist 11 22 23K+5300K Clip/Frame ✗ ✗ ✓ ✓ SurgCoT {}_{\text{(Ours)}} Generalist 7 35 2.8K+19K+19K Video/Clip/Frame ✓ ✓ ✓ ✓ #Spe. : #Specialties; #Pro. :#Procedures; ST : Spatiotemporal modeling; Pro. : Progressive reasoning protocol; Loc. : Localization supervision (caption/bbox/mask/timestamps/tracks.); Clin. : Clinicians as a reference standard. ✓: yes, ✗: no, \sim : unknown. To tackle this, we introduce SurgCoT (Fig. 1 ), a unified surgical video benchmark for evaluating MLLMs under a chain-of-thought (CoT) reasoning framework. SurgCoT advances the field in two key aspects: 1) Comprehensive Task Coverage : Unlike prior benchmarks focused on narrow tasks 44 ; 23 ; 52 or generic queries 51 ; 35 ; 50 , SurgCoT spans 35 procedures across 7 diverse specialties, enabling broad and realistic assessment of expert-level cognitive reasoning. 2) In-depth Reasoning Dimensions : SurgCoT introduces 5 clinical-relevant reasoning dimensions ( Causal Action Ordering , Cue-Action Alignment , Affordance Mapping , Micro-Transition Localization , and Anomaly Onset & Tracking ), reflecting critical cognitive processes in real world surgery. Collectively, these rich reasoning dimensions on diverse procedures enable a fine-grained evaluation of spatiotemporal understanding and clinical decision-making, closely mirroring actual surgical workflows. To enable CoT reasoning in surgical video understanding, as shown in Fig. 1 , we introduce a three-stage progressive reasoning framework ( Q1→Q2→Q3 ) with a comprehensive five-tuple annotation protocol ( Question→Option→Knowledge→Clue→Answer ). The novelty is two-fold: 1) It establishes a hierarchical reasoning pathway progressing from video-level comprehension through clip-level analysis to frame-level localization. As shown in Fig. 1 , all five annotation tuples evolve through these stages, with each stage’s Answer serving as contextual evidence for the subsequent stage, forming a continuous reasoning chain. 2) It employs synergistic annotation strategies where the Knowledge field provides clinically relevant background context based on the Question and Option , while the Clue supplies targeted spatiotemporal evidence extracted directly from the surgical video content. This integrated design progressively narrows both temporal and spatial scope, effectively guiding MLLMs through structured hierarchical reasoning. To validate its effectiveness as a unified spatiotemporal reasoning benchmark, we systematically evaluate 12 state-of-the-art MLLMs across diverse surgical procedures within a CoT framework, including open-source 4 ; 39 ; 30 ; 22 , medically specialized 37 ; 48 ; 6 , and commercial variants 32 ; 2 ; 11 . Results show that while commercial models lead their counterparts, all models exhibit significant limitations in fine-grained spatiotemporal understanding. Notably, by utilizing the three-stage progressive reasoning framework with five-tuple annotation protocol, considerable accuracy improvements could be achieved, indicating a viable path toward enhanced reasoning capability. SurgCoT offers a reproducible benchmark for method development and fair evaluation in whole-body surgical video analysis, highlighting both recent progress