ReSW-VL: Representation Learning for Surgical Workflow Analysis Using Vision-Language Model

Satoshi Kondo · 2025

Read ReSW-VL: Representation Learning for Surgical Workflow Analysis Using Vision-Language Model by Satoshi Kondo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13746.txt.

Introduction

Satoshi Kondo put forth a training method in 2025 to help software read the phase of a surgery from video. Phase reads can guide live help, staff plans, trainee scores, and safety checks in the room. Past top tools split the job: one net reads each frame, and a second reads the run of frames. How the first net is trained got little study.

Kondo tunes the image half of a vision-language model with prompt learning, so its frame reads fit surgery terms. The tuned reads then feed the phase call. Tests on three surgery video sets showed gains over past tools. The author frames the gain as proof that how frame traits are learned counts as much as the time model on top.

Catalog record

CREATORSatoshi Kondo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13746

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ReSW-VL: Representation Learning for Surgical Workflow Analysis Using Vision-Language Model Satoshi Kondo Abstract Surgical phase recognition from video is a technology that automatically classifies the progress of a surgical procedure and has a wide range of potential applications, including real-time surgical support, optimization of medical resources, training and skill assessment, and safety improvement. Recent advances in surgical phase recognition technology have focused primarily on Transform-based methods, although methods that extract spatial features from individual frames using a CNN and video features from the resulting time series of spatial features using time series modeling have shown high performance. However, there remains a paucity of research on training methods for CNNs employed for feature extraction or representation learning in surgical phase recognition. In this study, we propose a method for representation learning in surgical workflow analysis using a vision-language model (ReSW-VL). Our proposed method involves fine-tuning the image encoder of a CLIP (Convolutional Language Image Model) vision-language model using prompt learning for surgical phase recognition. The experimental results on three surgical phase recognition datasets demonstrate the effectiveness of the proposed method in comparison to conventional methods. 1 Introduction The implementation of surgical phase recognition technology entails the analysis of surgical videos, thereby facilitating the automatic classification of surgical progress. This technology confers numerous advantages, encompassing real-time surgical assistance, optimized resource utilization, educational prospects, skill evaluation, and enhanced safety measures. For instance, real-time progress monitoring during surgical procedures enables medical personnel to expeditiously and precisely ascertain the subsequent step, thereby enhancing surgical efficiency and safety. Furthermore, the analysis of surgical records facilitates the effective allocation of medical resources, as well as the training and assessment of medical students and residents, which is expected to improve the quality of surgery and reduce the burden on medical personnel. Recent advances in surgical phase recognition using deep learning have been reported [ 3 , 14 ] . Surgical phase recognition is a process performed on videos; therefore, the key point is how to capture the spatio-temporal features necessary for phase recognition from image sequences. Surgical phase recognition technology generally extracts spatial features for each frame in the video, and then extracts the features of the video by time-series modeling from the time-series of the obtained spatial features. Convolutional Neural Networks (CNNs), exemplified by ResNet [ 9 ] and Vision Transformer (ViT) [ 5 ] , are frequently employed for the extraction of spatial features. Initially, Recurrent Neural Networks (RNNs) [ 18 ] and Long Short-Term Memory (LSTM) networks [ 10 ] were utilized for time series modeling, followed by Temporal Convolutional Networks (TCNs) [ 12 ] and Transformers [ 23 ] . In recent years, studies have emerged that propose the application of Transformers for both spatial feature extraction and time series modeling. However, it is still reported that the combination of CNNs and time series modeling yields superior performance [ 20 ] . The combination of CNN and time series modeling is generally trained in two stages. In the first stage, the CNN is trained as a spatial feature extractor. In the second stage, the time series modeling is trained. In the second stage, the network parameters of the CNN remain fixed. The primary function of the CNNs in the first stage is to classify phases, and while the enhancement of phase recognition performance through the acquisition of effective spatial features with CNNs is paramount, there is a paucity of studies that have thoroughly examined the training methodologies of CNNs in phase recognition. In this paper, we propose a novel representation learning method for extracting spatial features from images, with the objective of facilitating surgical phase recognition. The proposed method draws inspiration from extant literature [ 13 ] . We utilize CLIP [ 19 ] , a visual-language model, to perform prompt learning and image encoder fine tuning. The resulting image encoder functions as a feature extractor, and in conjunction with time series modeling, enables surgical phase recognition. The contributions of this study are as follows: • This is one of the few studies on representation learning for surgical phase recognition, and it is the first method to use the vision-language model and prompt learning. • We show that the proposed method outperforms conventional methods on three widely used surgical phase recognition datasets. The code will be made publicly available if the paper is accepted. 2 Related Works Table 1 shows representative prior arts on deep learning-based surgical phase recognition methods. As illustrated in Table 1 , conventional surgical phase recognition methodologies predominantly entail the extraction of spatial features from individual frames within a video sequence. These spatial features are then utilized to derive video features through the application of time series modeling techniques. As illustrated in Table 1 , CNN (ResNet-50) and Vision Transformer (ViT-B/16) are frequently employed for spatial feature extraction, while LSTM, TCN, and Transformer are often utilized for time series modeling. Table 1: Representative prior arts on deep learning-based surgical phase recognition methods. Reference Year Spatial feature extractor Temporal modeling Jin et al. [ 11 ] (SV-RCNet) 2018 ResNet-50 LSTM Czempiel et al. [ 1 ] (TeCNO) 2020 ResNet-50 TCN Gao et al. [ 8 ] (Trans-SVNet) 2021 ResNet-50 TCN, Transformer Czempiel et al. [ 2 ] (Opera) 2021 ResNet-50 Transformer Zou et al. [ 25 ] (ARST) 2022 ResNet-50 TCN, Transformer Liu et al. [ 16 ] (SKiT) 2023 ViT-B/16 Transformer Liu et al. [ 15 ] (LoViT) 2025 ViT-B/16 Transformer In the context of training feature extractors, CNNs, and Vision Transformers for surgical phase recognition, a two-stage approach has been employed, wherein the feature extractors are fine-tuned with parameters initially trained on the ImageNet dataset [ 4 ] . This two-stage training method has been shown to enhance the predictive capabilities of surgical phases. The training of the feature extractor in the first stage can be regarded as representation learning from images. Consequently, the acquisition of suitable spatial features for surgical phase recognition is crucial for enhancing the performance of surgical phase recognition. However, there is a paucity of studies that have thoroughly examined the training methodologies of feature extractors in the context of surgical phase recognition. In the subsequent section, we propose a methodology to address this knowledge gap. 3 Propose Method In this paper, we propose a novel representation learning methodology for the extraction of spatial features from images, with the objective of facilitating surgical phase recognition. The proposed methodology employs CLIP, a visual-language model, to facilitate prompt learning and image encoder fine-tuning. We have designated our proposed methodology as ReSW-VL, an acronym for “ Re presentation learning in S urgical W orkflow analysis using a V ision- L anguage model.” First, we describe the representation of surgical phases. In some surgical phase recognition datasets, the surgical phases are represented as strings. For example, in the Cholec80 dataset [ 22 ] , the seven phases are defined as “preparation”, “calot triangle dissection”, “clipping and cutting”, “gallbladder dissection”, “gallbladder packing”, “cleaning and coagulation”, and “gallbladder retraction”. These are converted to numbers defined by the dataset. If the number of surgical phases is P , then the number of surgical phases p takes the value 1\leq p\leq P . As illustrated in Figure 1 , the proposed method involves

Read the complete text