Analysis of Transferability Estimation Metrics for Surgical Phase Recognition
Prabhant Singh, Yiping Li, Yasmina Al Khalil · 2025
Read Analysis of Transferability Estimation Metrics for Surgical Phase Recognition by Prabhant Singh, Yiping Li, Yasmina Al Khalil (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16730.txt.
Introduction
Prabhant Singh and co-authors graded shortcuts for picking surgical AI models in two thousand twenty-five. Fine-tuning a pretrained model suits surgical video well. Expert labels are scarce and costly there. Transferability scores promise to rank models cheaply. They use only outputs, never full retraining. Three leading scores were benchmarked head to head.
Two surgical datasets supplied the test beds. One covered robotic esophagus surgery phases. The other covered laparoscopic procedures. LogME, scored per subset, tracked fine-tuning best. Other scores aligned less reliably with real accuracy. Hospitals can shortlist models with more confidence now. Annotation budgets can focus on the winning model.
Catalog record
| CREATOR | Prabhant Singh, Yiping Li, Yasmina Al Khalil |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-16730
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Analysis of Transferability Estimation Metrics for Surgical Phase Recognition Prabhant Singh, Yiping Li, Yasmina Al Khalil Abstract Fine-tuning pre-trained models has become a cornerstone of modern machine learning, allowing practitioners to achieve high performance with limited labeled data. In surgical video analysis, where expert annotations are especially time-consuming and costly, identifying the most suitable pre-trained model for a downstream task is both critical and challenging. Source-independent transferability estimation (SITE) offers a solution by predicting how well a model will fine-tune on target data using only its embeddings or outputs, without requiring full retraining. In this work, we formalize SITE for surgical phase recognition and provide the first comprehensive benchmark of three representative metrics, LogME, H-Score, and TransRate, on two diverse datasets (RAMIE and AutoLaparo). Our results show that LogME, particularly when aggregated by the minimum per-subset score, aligns most closely with fine-tuning accuracy; H-Score yields only weak predictive power; and TransRate often inverses true model rankings. Ablation studies show that when candidate models have similar performances, transferability estimates lose discriminative power, emphasizing the importance of maintaining model diversity or using additional validation. We conclude with practical guidelines for model selection and outline future directions toward domain-specific metrics, theoretical foundations, and interactive benchmarking tools. 1 1 1 Accepted at 3rd Workshop in Data Engineering in Medical Imaging (DEMI), MICCAI-2025 Workshop 1 Introduction In recent years, deep learning (DL) has played a central role in advancements in surgical artificial intelligence (AI), particularly in the domain of surgical video understanding . Among these, surgical phase recognition, the task of classifying each frame of a surgical video into predefined procedural stages, has received significant attention due to its potential to support intra-operative guidance, post-operative analytics, and autonomous robotic assistance. However, despite promising results, the widespread adoption of DL models in surgical AI remains hindered by a persistent challenge: the lack of annotated surgical video data. Acquiring annotations for surgical videos is a highly resource-intensive process, requiring domain expertise, time, and effort. Moreover, the surgical environment presents substantial variability and complexity: frequent occlusions, motion blur, illumination changes, and heterogeneous surgical styles across institutions and surgeons. These factors, coupled with the high temporal and contextual dependencies inherent in surgical workflows, make it difficult to train generalizable and reliable DL models without large-scale, high-quality labeled datasets. To address data scarcity, transfer learning has become a widely adopted strategy. Models pre-trained on large-scale datasets such as ImageNet [ 6 ] are often fine-tuned on specific medical imaging tasks, including surgical phase recognition. This approach can significantly boost performance and reduce training time, especially in low-data regimes. Recent works have demonstrated that large-scale pre-training improves downstream performance in surgical tasks [ 10 , 2 , 16 ] . However, the effectiveness of transfer learning in surgical applications remains uncertain. The domain gap between natural images and surgical data often limits the utility of off-the-shelf pre-trained models, and the performance gain is heavily dependent on factors such as model architecture, source dataset characteristics, and fine-tuning strategies. Given the rapidly growing landscape of available pre-trained models, selecting the most suitable model for a specific surgical application has become a non-trivial problem. Exhaustive fine-tuning of each candidate model is computationally expensive and impractical, particularly in time- and resource-constrained clinical settings. As such, there is a critical need for transferability estimation methods that can predict how well a pre-trained model is likely to perform on a surgical downstream task without extensive retraining. This challenge, known as source-independent transferability estimation (SITE), seeks to evaluate model suitability without relying on fine-tuning or access to source training data. While SITE has been explored in natural image classification and general domain adaptation contexts, its application to surgical video understanding remains underexplored. Effective SITE methods could drastically reduce the cost of model selection and adaptation in surgical AI, enabling more efficient use of scarce annotated data and computational resources. In this work, we frame Surgical Phase Recognition as a testbed for investigating SITE in the context of surgical AI. We highlight the limitations of current pre-training paradigms for surgical tasks and motivate the need for principled transferability assessment approaches tailored to the unique challenges of the surgical domain. Our code for the experiments can be found on https://github.com/prabhant/surgical_phase_transferability_estimation 1.1 Related Work In the context of Source-Independent Transferability Estimation, numerous metrics have been proposed, each grounded in different theoretical perspectives and methodological assumptions. LogME [ 24 ] formalizes the transferability estimation as the maximum label marginalized likelihood and adopts a directed graphical model to solve it. SFDA [ 17 ] proposes a self-challenging mechanism, it first maps the features and then calculates the sum of log-likelihood as the metric. NCTI [ 21 ] treats it as a nearest centroid classifier problem and measures how close the geometry of the target features is to their hypothetical state in the terminal stage of the fine-tuned model. LEEP [ 15 ] is the average log-likelihood of the log-expected empirical predictor, which is a non-parameter classifier based on the joint distribution of the source and target distribution, N-LEEP [ 12 ] is a further improvement on LEEP by substituting the output layer with a Gaussian mixture model. TransRate [ 9 ] treats SITE from an information theory point of view by measuring the transferability as the mutual information between features of target examples extracted by a pre-trained model and their labels. In the context of SITE in medical imaging, only a limited number of studies have explored transferability estimation. Notably, Juodelyte et al. [ 11 ] introduced a dynamic nearest component analysis-based metric for medical image classification in a comprehensive evaluation setup, while CC-FV [ 23 ] was proposed for SITE in the domain of medical image segmentation. 2 Methodology In this section, we (i) formulate the general transferability estimation problem ( 2.1 ), (ii) tailor the framework to the surgical phase recognition task ( 2.2 ), and (iii) outline our experimental setup for assessing the effectiveness of different transferability metrics ( 2.3 ). 2.1 Problem Statement We assume that we are given a target dataset \mathcal{D}=\{(\mathbf{x}_{n},y_{n})\}_{n=1}^{N} of N labeled points and M pre-trained models \{\Phi_{m}=(\phi_{m},\psi_{m})\}_{m=1}^{M} . Each model \Phi_{m} consists of a feature extractor that returns a d -dimensional embedding \phi_{m}(x)\in\mathbb{R}^{d} and the final layer or head \psi_{m} that outputs the label prediction for the given input x . Figure 1 : Estimating transferability via frame-wise estimation. The task of estimating transferability is to generate a score for each pre-trained model so that the best model can be identified via a ranking list. For each pre-trained model \Phi_{m} a transferability metric outputs a scalar score T_{m} that should be coherent in its ranking with the performance of the fine-tuned classifier \hat{\Phi}_{m} . That is, the goal is to obtain scores T_{m} as: T_{m}\geq T_{n}\Leftrightarrow\frac{1}{N}\sum_{n=1}^{N}p(y_{n}|\mathbf{x}_{n};\hat{\Phi}_{m})\geq\frac{1}{N}\sum_{n=1}^{N}p(y_{n}|\mathbf{x}_{n};\hat{\Phi}_{n}), (1) where p(y_{n}|x_{n};\hat{\Phi}_{m}) indicates the probability that the fine-tuned model \hat{\Phi}_{m} predicts label y_{n} for input \mathbf{x}_{n} . A larger T_{m} indicates better performance model on target data \mathcal{D} . 2.2 Application to Surgical Phase Recognition In the specific context of surgical phase recognition , the target dataset \mathcal{D} is composed of \mathbf{z} sampled static frames taken from surgical videos, with each frame annotated by a phase label. These frames may be grouped into Z disjoint subsets (e.g.,