Surg Σ \Sigma : A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence
Zhitao Zeng, Mengya Xu Jian Jiang Pengfei Guo Yunqiu Xu Zhu Zhuo, Chang Han Low Yufan He Dong Yang Chenxi Lin Yiming Gu Jiaxin Guo, Yutong Ban Daguang Xu Qi Dou Yueming Jin · 2026
Read Surg Σ \Sigma : A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence by Zhitao Zeng, Mengya Xu Jian Jiang Pengfei Guo Yunqiu Xu Zhu Zhuo, Chang Han Low Yufan He Dong Yang Chenxi Lin Yiming Gu Jiaxin Guo, Yutong Ban Daguang Xu Qi Dou Yueming Jin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16822.txt.
Introduction
Zeng and co-authors introduce SurgΣ, a large data and model foundation for surgical AI. Most surgical systems are built for one task and transfer poorly across procedures and hospitals. SurgΣ-DB unifies open datasets, in-house clinical data, and web data into one schema. It spans six specialties and 18 tasks, from understanding and reasoning to planning and generation. The data hold over 5.98 million conversations, plus layered reasoning notes. The authors also report tests of foundation models trained on it. They find gains in cross-task generalization and interpretability.
Catalog record
| CREATOR | Zhitao Zeng, Mengya Xu Jian Jiang Pengfei Guo Yunqiu Xu Zhu Zhuo, Chang Han Low Yufan He Dong Yang Chenxi Lin Yiming Gu Jiaxin Guo, Yutong Ban Daguang Xu Qi Dou Yueming Jin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57731 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16822
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Surg Σ \Sigma : A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence Zhitao Zeng, Mengya Xu Jian Jiang Pengfei Guo Yunqiu Xu Zhu Zhuo, Chang Han Low Yufan He Dong Yang Chenxi Lin Yiming Gu Jiaxin Guo, Yutong Ban Daguang Xu Qi Dou Yueming Jin Abstract Surgical intelligence has the potential to improve the safety and consistency of surgical care, yet most existing surgical AI frameworks remain task-specific and struggle to generalize across procedures and institutions. Although multimodal foundation models, particularly multimodal large language models, have demonstrated strong cross-task capabilities across various medical domains, their advancement in surgery remains constrained by the lack of large-scale, systematically curated multimodal data. To address this challenge, we introduce Surg Σ \Sigma , a spectrum of large-scale multimodal data and foundation models for surgical intelligence. At the core of this framework lies Surg Σ \Sigma -DB, a large-scale multimodal data foundation designed to support diverse surgical tasks. Surg Σ \Sigma -DB consolidates heterogeneous surgical data sources (including open-source datasets, curated in-house clinical collections and web-source data) into a unified schema, aiming to improve label consistency and data standardization across heterogeneous datasets. Surg Σ \Sigma -DB spans 6 clinical specialties and diverse surgical types, providing rich image- and video-level annotations across 18 practical surgical tasks covering understanding, reasoning, planning, and generation, at an unprecedented scale (over 5.98M conversations). Beyond conventional multimodal conversations, Surg Σ \Sigma -DB incorporates hierarchical reasoning annotations, providing richer semantic cues to support deeper contextual understanding in complex surgical scenarios. We further provide empirical evidence through recently developed surgical foundation models built upon Surg Σ \Sigma -DB, illustrating the practical benefits of large-scale multimodal annotations, unified semantic design, and structured reasoning annotations for improving cross-task generalization and interpretability. 1 Introduction According to estimates from the Lancet Commission, more than 300 million surgical procedures are performed worldwide each year 49 , underscoring the urgent demand for safer and more accessible surgical care. Despite advances in minimally invasive 26 ; 50 and robotic techniques 25 ; 32 , surgery remains inherently complex, requiring continuous interpretation of dynamic anatomy and high-stakes decision-making under uncertainty. Surgical AI is therefore emerging as a transformative paradigm, acting as an intelligent collaborator that enhances perception, understanding, and reasoning. By leveraging multimodal intraoperative signals ( e.g. , visual streams, textual instructions, robotic kinematics, and preoperative imaging), AI systems promise to improve safety, reduce variability, and broaden access to high-quality surgical expertise. However, most prior surgical AI systems remain narrowly designed for isolated tasks, including phase recognition 71 ; 40 , tool or tissue segmentation 6 ; 5 , and action classification 57 ; 7 , often within a tailored task or a single surgical type. This task-specific paradigm limits knowledge transfer and leads to brittle generalization, where models degrade under distribution shifts caused by differences in imaging systems, anatomy, or surgical styles. Foundation models, particularly multimodal large language models 70 ; 68 ; 9 ; 8 , have recently emerged as a promising paradigm for unified visual perception, language understanding and multimodal reasoning, enabling models to jointly interpret visual content and reason with natural language. In principle, such models offer a unified framework capable of supporting a broad spectrum of surgical tasks, ranging from describing anatomical structures and instrument states to answering intraoperative queries, summarizing procedural context, and generating interpretable decision-support rationales. While foundation models have achieved remarkable success across domains such as radiology 84 ; 69 ; 12 , pathology 17 ; 23 ; 73 , and molecular biology 38 ; 1 , their application to the surgical domain remains comparatively underexplored. Surgery poses fundamentally distinct challenges for training multimodal foundation models. Intraoperative scenes are not only visually complex ( e.g. , severe occlusion, tissue deformation, and rapid camera motion), but also exhibit strong spatiotemporal structure and causal interdependence, where subtle instrument–tissue interactions can induce irreversible anatomical changes. Clinically relevant cues are often fine-grained, transient, and context-dependent, demanding long-horizon temporal reasoning and precise spatial grounding beyond static image understanding. Furthermore, variability across institutions, surgeons, devices, and patient anatomies introduces substantial distribution shifts that hinder generalization. We observe that a fundamental obstacle to advancing surgical multimodal foundation models lies in the lack of large-scale, high-quality, and systematically curated multimodal data. Conventional surgical datasets 72 ; 7 ; 81 are predominantly vision-centric and designed under a closed-set paradigm, providing only predefined categorical annotations ( e.g. , surgical phase or instrument tags) while lacking practical natural language instruction–visual pairs that better reflect real-world clinical usage and flexible interaction. In addition, these datasets are typically limited in scale and surgical-type diversity, as they are often confined to a small number of procedures or institutions. Consequently, they remain fragmented across tasks and modalities, with inconsistent annotation standards and heterogeneous label spaces that hinder cross-dataset integration and large-scale training . Although some recent works 20 ; 56 ; 58 have introduced datasets for surgical foundation models, they still exhibit notable limitations in scale, diversity, and task coverage, as summarized in Table 1 . On the other hand, annotation quality and granularity remain insufficient: the absence of a unified label space can mislead training and weaken generalization, while existing datasets largely lack high-quality multi-step reasoning traces. Figure 1 : Surg \Sigma -DB is a large-scale multimodal data foundation for surgical intelligence. To facilitate research, we present Surg \Sigma , a spectrum of large-scale multimodal data and foundation models for surgical intelligence. At its core, Surg \Sigma -DB serves as a unified multimodal data foundation designed to enable large-scale training of surgical foundation models. Rather than releasing isolated task-specific datasets, our Surg \Sigma -DB systematically consolidates diverse surgical data sources into a unified and well-structured foundation. We curate data spanning multiple surgical specialties and procedures, covering diverse clinical departments and operative types to ensure broad anatomical and procedural variability. Surg \Sigma -DB combines open-source resources with web-collected surgical videos, and employs a semi-automated annotation pipeline integrating expert human labeling and controlled synthesis to ensure both real-world representativeness and scalable clinical fidelity. Within the same surgical scenes, Surg \Sigma -DB provides rich annotations with hierarchical reasoning traces , enabling multi-grained spatial understanding as well as temporal modeling. To the best of our knowledge, Surg \Sigma -DB provides one of the most comprehensive task coverages in surgical intelligence , spanning understanding, reasoning, planning, and generative capabilities through richly structured and multi-level annotations, and is constructed at an unprecedented scale ( i.e. , \sim 5.98M) across diverse surgical types from 6 clinical specialties. Importantly, all data are organized under a unified format, facilitating interoperable training, cross-task integration, and future extensibility, while promoting more consistent label spaces across heterogeneous datasets. Building upon Surg \Sigma -DB, a family of surgical foundation models ( i.e. , BSA 87 , SurgVLM 95 , Surg-R1 33 , and Cosmos-H-Surgical 31 ) are developed, which empirically demonstrate the effectiveness of the proposed unified data spectrum. BSA 87 demonstrates that fundamental surgical actions exhibit consistent, recognizable patterns across anatomically diverse procedures, enabling cross-specialty generalization without domain-specific adaptation and supporting clinically meaningful downstream applications including skill assessment and procedural planning. SurgVLM 95 demonstrates that large-scale multimodal instruction-tuning data can enhance cross-task generalization, enabling a single model to effectively handle diverse surgical understanding tasks through