ProtoFlow: interpretable and robust surgical workflow modeling with learned dynamic scene graph prototypes Authors: Felix Holm, Ghazal Ghazaei, Nassir Navab Published: 2026 ABSTRACT PurposeDetailed surgical recognition is critical for advancing AI-assisted surgery, yet progress is hampered by high annotation costs, data scarcity, and a lack of interpretable models. While scene graphs offer a structured abstraction of surgical events, their full potential remains untapped. In this work, we introduce ProtoFlow, a novel framework that learns dynamic scene graph prototypes to model complex surgical workflows in an interpretable and robust manner.MethodsProtoFlow leverages a graph neural network (GNN) encoder–decoder architecture that combines self-supervised pretraining for rich representation learning with a prototype-based fine-tuning stage. This process discovers and refines core prototypes that encapsulate recurring, clinically meaningful patterns of surgical interaction, forming an explainable foundation for workflow analysis.ResultsWe evaluate our approach on the fine-grained CAT-SG dataset. ProtoFlow not only outperforms standard GNN baselines in overall accuracy but also demonstrates exceptional robustness in limited-data, few-shot scenarios, maintaining strong performance when trained on as few as one surgical video. Our qualitative analyses further show that the learned prototypes successfully identify distinct surgical sub-techniques and provide clear, interpretable insights into workflow deviations and rare complications.ConclusionBy uniting robust representation learning with inherent explainability, ProtoFlow represents a significant step toward developing more transparent, reliable, and data-efficient AI systems, accelerating their potential for clinical adoption in surgical training, real-time decision support, and workflow optimization.Supplementary InformationThe online version contains supplementary material available at https://doi.org/10.1007/s11548-026-03697-x. Introduction The development of AI-driven surgical automation and decision support systems relies heavily on accurate recognition and understanding of complex surgical workflows. However, surgical video analysis presents unique challenges, including significant variability across procedures, real-time constraints, and the necessity for structured yet interpretable representations of surgical activities. Traditional deep learning-based methods primarily focus on frame-wise classification tasks such as phase recognition [1–3], tool detection [4, 5], or semantic scene segmentation [6]. While effective for isolated recognition tasks, these methods struggle to capture the intricate and evolving interdependent relationships between surgical elements, limiting both model generalizability and interpretability, two key factors for deploying AI-driven decision support in clinical environments. A promising alternative lies in scene graphs, structured graphical representations that explicitly model objects and their interactions within a scene [7, 8]. Dynamic scene graphs (DSGs) extend this concept by capturing the temporal evolution of relationships in complex domains such as surgical workflow [9]. In surgical workflow modeling, recent works have leveraged scene graphs for surgical workflow recognition in laparoscopic cholecystectomy [10], simulated knee replacement surgery [11] and cataract surgery [9, 12] using positional and temporal relations between the surgical elements. Despite these advances, existing approaches still face important hurdles, primarily due to three key limitations: (1) Data scarcity. Acquiring large-scale, expert-labeled surgical datasets is costly and time-consuming, limiting the applicability of supervised learning approaches. (2) Generalization challenges. Surgical workflows exhibit high variability across patients, techniques, and clinical settings, making it difficult for models to generalize effectively. Edge cases and anomalous events, in particular, are the hardest to model yet often the most critical to handle accurately. (3) Interpretability and Transparency. Furthermore, there is a pressing need for improved interpretability, so that medical teams can understand and trust the decisions made by AI-based systems. To address these challenges, we introduce ProtoFlow, a novel prototype learning framework for surgical workflow modeling, inspired by recent advances in prototype learning for graph-based representations [13]. Instead of treating each surgical scene as an independent instance, ProtoFlow identifies and learns recurring structured representations of surgical interactions as workflow prototypes. These prototypes encapsulate fundamental patterns in surgical workflows, enabling a more interpretable and generalizable representation of procedure dynamics. By leveraging graph autoencoder architectures, our approach learns abstract prototype embeddings, which serve as structured class representations that enhance both accuracy and interpretability in surgical workflow modeling. A key advantage of ProtoFlow is its ability to perform few-shot learning, significantly reducing the reliance on large labeled datasets. By integrating self-supervised and supervised learning, we enhance prototype robustness, enabling effective generalization across different surgical procedures, even in low-data scenarios. We evaluate our approach on cataract surgery, utilizing a newly introduced CAT-SG scene graph dataset [14] featuring positional, geometrical, temporal and semantic relationships among surgical elements. Cataract surgery is a highly standardized yet delicate procedure, with significant intraoperative variability influenced by factors such as cataract grade, patient-specific conditions, and surgical technique adaptations. Beyond surgical workflow recognition, ProtoFlow supports qualitative deviation analysis, providing interpretable insights into deviations from standard surgical procedure. While prior work has explored deviation detection in surgical workflows, such as in laparoscopic rectopexy [15], these approaches lack explicit interpretability and structured workflow representations. By integrating prototype learning and scene graphs, our method enhances explainability, making it potentially useful for surgical training, workflow optimization, and AI-driven decision support. Our contributions are summarized as follows: Architecture: We propose ProtoFlow, a novel prototype learning framework for surgical workflow modeling using dynamic scene graphs, integrating supervised and self-supervised learning. Comprehensive Evaluation: We conduct ablation studies and benchmarking on the CAT-SG dataset [14], demonstrating that ProtoFlow outperforms existing baselines. Our few-shot learning experiments highlight the framework’s ability to generalize more effectively on as few as one surgical video. Enhanced Interpretability: We explore qualitative deviation analysis and explainability, leveraging prototype-driven representations to provide interpretable insights into surgical workflow variations and model decisions. Critically, we find ProtoFlow handles low-frequency phases and anomalous events with greater reliability. Methodology Fig. 1Overview of the four-stage pipeline (pretraining, prototype initialization, fine-tuning, and inference) for prototype-based surgical workflow modeling (ProtoFlow) Overview of the four-stage pipeline (pretraining, prototype initialization, fine-tuning, and inference) for prototype-based surgical workflow modeling (ProtoFlow) Our goal is to learn interpretable, prototype-based representations of dynamic scene graphs for surgical workflow modeling. Our pipeline, therefore, predicts the surgical phase directly from dynamic scene graphs. The pipeline proceeds in four stages: (1) pretraining a graph autoencoder model for graph reconstruction, (2) prototype initialization, (3) fine-tuning with prototype-based supervision, and (4) inference. An overview is shown in Fig. 1. GNN Architecture and Pretraining Let each DSG at time t be denoted by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$G_{t} = (A_{t}, X_{t})$$\end{document}Gt=(At,Xt), where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_{t} \in \mathbb {R}^{N \times N}$$\end{document}At∈RN×N is the adjacency matrix (encoding spatial, temporal, and semantic relationships among the N nodes), and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{t} \in \mathbb {R}^{N \times d}$$\end{document}Xt∈RN×d is the node feature matrix with d features per node. Our model uses a GNN encoder \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{\theta }$$\end{document}fθ that transforms each graph into node embeddings \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ H_{t}\in \mathbb {R}^{N \times d_h}$$\end{document}Ht∈RN×dh with embedding dimension \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_h$$\end{document}dh. A global DSG-level embedding \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{t} \in \mathbb {R}^{d_h}$$\end{document}zt∈Rdh is then obtained by pooling \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{t}$$\end{document}Ht (e.g., via mean pooling). To stabilize the latent space and promote discriminative embeddings, we use a decoder \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g_{\phi }$$\end{document}gϕ to reconstruct the original graph. Given \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{t}$$\end{document}Ht, the decoder outputs reconstructed node features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{X}_{t}$$\end{document}X^t and (optionally) adjacency \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{A}_{t}$$\end{document}A^t. The reconstruction loss is defined as a mean-squared error (MSE) loss. Additionally, a classification head \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h_{\psi }$$\end{document}hψ takes \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{t}$$\end{document}zt and predicts a surgical workflow phase \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_{t}$$\end{document}yt. We adopt a cross-entropy classification loss for phase prediction. The overall pretraining objective is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \mathcal {L}_{\text {pretrain}} \;=\; \mathcal {L}_{\text {rec}} \;+\; \,\mathcal {L}_{\text {cls}} $$\end{document}Lpretrain=Lrec+Lcls. Prototype Initialization After pretraining, we use the trained encoder \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{\theta }$$\end{document}fθ to map all training DSGs to embeddings \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{t}$$\end{document}zt. For each class \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c \in \{1,\dots ,C\}$$\end{document}c∈{1,⋯,C}, we cluster the embeddings that belong to class c into K groups (using k-nearest neighbors). Each cluster’s centroid is taken as an initial prototype. These prototypes serve as reference points in the embedding space for each surgical phase.1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {P} \;=\; \{\textbf{p}_{c,k} \mid c=1,\ldots ,C;\; k=1,\ldots ,K\}. \end{aligned}$$\end{document}P={pc,k∣c=1,…,C;k=1,…,K}.Fine-tuning with Prototype Regularization In fine-tuning, both the GNN encoder \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{\theta }$$\end{document}fθ and the prototypes \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}_{c,k}$$\end{document}pc,k are updated to optimize classification based on distance to each prototype. Given an input DSG embedding \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{t}$$\end{document}zt, we compute the distance to each prototype, softmax the distance vector for normalization, and then average the softmaxed distances per class to convert to prediction \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{y}_{t,c}$$\end{document}y^t,c:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} & d_{c,k}(z_{t}) = \Vert z_{t} - \textbf{p}_{c,k}\Vert ,\nonumber \\ & \quad \hat{y}_{t,c} \;=\; \frac{1}{K} \sum _{k=1}^{K} \frac{\exp \bigl (-d_{c,k}(z_t)\bigr )}{\displaystyle \sum _{c{\prime }=1}^{C}\sum _{k{\prime }=1}^{K}\exp \bigl (-d_{c{\prime },k{\prime }}(z_t)\bigr )}. \end{aligned}$$\end{document}dc,k(zt)=‖zt-pc,k‖,y^t,c=1K∑k=1Kexp(-dc,k(zt))∑c′=1C∑k′=1Kexp(-dc′,k′(zt)).We then apply a cross-entropy loss on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{y}_{t}$$\end{document}y^t, denoted \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {L}_{\text {cls}}$$\end{document}Lcls. To preserve the cluster structure that emerged in pretraining, we regularize prototype movement with a MSE term:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {L}_{\text {reg}} = \sum _{c,k} \Vert \textbf{p}_{c,k}^{\,(0)} - \textbf{p}_{c,k}\Vert ^{2}, \end{aligned}$$\end{document}Lreg=∑c,k‖pc,k(0)-pc,k‖2,where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}_{c,k}^{\,(0)}$$\end{document}pc,k(0) are the initial prototypes from clustering. We also maintain the graph reconstruction loss \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {L}_{\text {rec}}$$\end{document}Lrec. Altogether, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \mathcal {L}_{\text {finetune}} \;=\; \mathcal {L}_{\text {cls}} \;+\; \mathcal {L}_{\text {reg}} \;+\; \mathcal {L}_{\text {rec}} $$\end{document}Lfinetune=Lcls+Lreg+Lrec. Inference and Explainability At inference, we encode a new DSG, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$G_{*}$$\end{document}G∗ to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{*}$$\end{document}z∗, measure its Euclidean distances to each prototype, and produce a softmax-based prediction. Because each prototype \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}_{c,k}$$\end{document}pc,k is tied to representative substructures of a surgical phase, these distance values provide intuitive explanations, indicating which prototypical scenario most closely matches the new graph. Further interpretability arises from comparing node-level or edge-level alignments between the input graph and the closest prototypes. Experiments We conduct a series of experiments to evaluate the performance of our proposed ProtoFlow approach for surgical workflow recognition on the CAT-SG dataset. Our experiments are designed to (1) compare ProtoFlow against existing baselines, (2) analyze the impact of different temporal windows within the dynamic scene graph prototypes, (3) highlight generalizability via few-shot learning experiments, and (4) showcase the interpretability of our learned prototypes. CAT-SG Dataset We use the recently introduced CAT-SG dataset, which extends the original CATARACTS dataset [16] with semantic scene graphs. Nodes in each scene graph represent surgical objects (e.g., instruments, anatomical structures), while edges capture a variety of relationships such as spatial proximity and semantic interactions. In total, the dataset comprises 50 cataract surgery videos, each ranging from 6 to 40 min in length. The videos in the dataset are segmented into 19 different surgical steps, describing the workflow of the cataract surgery in high detail. The dataset is split into training, validation, and test sets at the video level to ensure no overlap of procedures across folds. We follow the proposed split of CATARACTS, yielding 25 training, 5 validation, and 20 test videos [16]. To leverage the graphs of the CAT-SG temporally, we construct dynamic scene graphs, similar to [9]. Scene graphs for multiple frames are aggregated by connecting nodes of the same class from temporally adjacent frames with temporal edges, resulting in one large graph.Table 1Surgical workflow recognition resultsTemporalSemanticMethodWindowRelationsAccuracyF1Holm et al. [9]1 frame✗ CATARACTS65.5652.24GATv2 [14]1 frame✓ CAT-SG70.8156.02Llama 3.2 3B [14]1 frame✓ CAT-SG69.1353.87Holm et al. [9]30 frames (90 s)✗ CATARACTS73.7764.93GATv2 [14]30 frames (90 s)✓ CAT-SG78.6370.15ProtoFlow1 frame✓ CAT-SG69.05 ± 1.0760.06 ± 1.55ProtoFlow30 frames (90 s)✓ CAT-SG80.07 ± 0.1963.21 ± 0.72ProtoFlow60 frames (180 s)✓ CAT-SG80.14 ± 3.1568.66 ± 3.06Bold indicates the best result and underlined indicates the second best result (per metric) Table 1Surgical workflow recognition resultsTemporalSemanticMethodWindowRelationsAccuracyF1Holm et al. [9]1 frame✗ CATARACTS65.5652.24GATv2 [14]1 frame✓ CAT-SG70.8156.02Llama 3.2 3B [14]1 frame✓ CAT-SG69.1353.87Holm et al. [9]30 frames (90 s)✗ CATARACTS73.7764.93GATv2 [14]30 frames (90 s)✓ CAT-SG78.6370.15ProtoFlow1 frame✓ CAT-SG69.05 ± 1.0760.06 ± 1.55ProtoFlow30 frames (90 s)✓ CAT-SG80.07 ± 0.1963.21 ± 0.72ProtoFlow60 frames (180 s)✓ CAT-SG80.14 ± 3.1568.66 ± 3.06Bold indicates the best result and underlined indicates the second best result (per metric) Surgical workflow recognition results Bold indicates the best result and underlined indicates the second best result (per metric) Experimental Setup: Backbone GNN We implement the encoder as a multilayer GNN using 3 GATv2 [17] layers, and the decoder as a 3-layer MLP. The hidden dimension for both models is 1024, and the size of the graph encoding \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{t}$$\end{document}zt is set to 512. We use mean pooling for graph-level readout and reconstruct only node features (not adjacency). Edge types are encoded as one-hot vectors of length 9 (1 geometric, 7 semantic, 1 temporal) that the GATv2 layers attend to. Prototype Module. We initialize 19 (#classes) × × 3 prototypes (#prototypes per class) from cluster centroids in the latent space after pretraining. Optimization. We train all methods for a maximum of 50 (pretraining) + 50 (fine-tuning) epochs with an initial learning rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$3 \times 10^{-4}$$\end{document}3×10-4 and a stepped learning rate schedule using the Adam optimizer with a batch size of 64. Early stopping based on validation set performance is used to prevent overfitting. We use PyTorch [18] and PyTorch geometric [19] as our framework and train our models on NVIDIA A40 GPUs with 40GB VRAM, taking approximately 24 h per run. Temporal Window Setup. We explore different temporal window sizes (1 frame to 60 frames, corresponding to up to 180 s) to measure the effect of temporal context on accuracy and F1 scores (see Table 1 and Fig. 2). Evaluation Metrics. We report accuracy and F1 score as standard metrics for workflow recognition. These metrics are averaged over the videos, to give each video equal weight in the final result. Since our model is initialized from scratch, we always show the average results from three runs for our results and indicate the standard deviation.Table 2Few-shot surgical workflow recognition with varying number of annotated videosMethod# Annotated VideosAccuracyF1GATv2115.43 ± 0.129.49 ± 0.07ProtoFlow39.49 ± 2.2728.89 ± 2.57GATv2240.65 ± 1.0330.17 ± 2.43ProtoFlow58.63 ± 2.1541.11 ± 2.49GATv2543.56 ± 2.8738.14 ± 3.27ProtoFlow62.66 ± 4.8244.03 ± 6.1Bold indicates the best result per category (1, 2, and 5 training videos) Table 2Few-shot surgical workflow recognition with varying number of annotated videosMethod# Annotated VideosAccuracyF1GATv2115.43 ± 0.129.49 ± 0.07ProtoFlow39.49 ± 2.2728.89 ± 2.57GATv2240.65 ± 1.0330.17 ± 2.43ProtoFlow58.63 ± 2.1541.11 ± 2.49GATv2543.56 ± 2.8738.14 ± 3.27ProtoFlow62.66 ± 4.8244.03 ± 6.1Bold indicates the best result per category (1, 2, and 5 training videos) Few-shot surgical workflow recognition with varying number of annotated videos Bold indicates the best result per category (1, 2, and 5 training videos) Results and discussion Comparison with Baselines for Surgical Workflow Recognition Table 1 presents the workflow recognition results of our method alongside baselines. We compare ProtoFlow to baselines previously reported in [14], including GNN- and LLM-based approaches, all predicting surgical phase from (dynamic) scene graphs. When employing only one frame (i.e., no temporal aggregation), ProtoFlow already achieves comparable performance. However, as we expand the temporal window from 1 frame to 30–60 frames (see Fig. 2), ProtoFlow’s performance markedly improves, surpassing 80% in accuracy and reaching 68.66% in F1 score at 60 frames. Our results confirm that capturing the evolving dependencies and temporal context strengthens the latent representations for surgical workflow recognition. Notably, using more than 20 frames (60 s) yields good results but strongly diminishing returns on the increased graph complexity, suggesting that modeling roughly one minute of context is generally sufficient. While ProtoFlow achieves the highest accuracy, we observe a modest trade-off in F1 score compared to GATv2. We attribute this to phases exhibiting high intra-class diversity, where a fixed number of prototypes may not fully capture all valid surgical execution styles. Importantly, this does not reflect systematic underperformance on minority classes—as demonstrated by strong performance on the rare Vitrectomy phase (Fig. 5). We view this trade-off as inherent to prototype-based classification, justified by the substantial gains in interpretability and few-shot robustness.Fig. 2Impact of temporal window size on ProtoFlow’s performance (accuracy in blue, F1 in red). The model’s performance improves as the temporal context increases, but the gains start to plateau after 20-30 frames (60-90 s). This indicates diminishing returns and suggests that a context of roughly one minute is sufficient for effective workflow recognition Impact of temporal window size on ProtoFlow’s performance (accuracy in blue, F1 in red). The model’s performance improves as the temporal context increases, but the gains start to plateau after 20-30 frames (60-90 s). This indicates diminishing returns and suggests that a context of roughly one minute is sufficient for effective workflow recognition Robustness in Limited-Data Scenarios Gathering surgery data remains expensive and time-consuming. We thus assessed ProtoFlow in few-shot settings, where only a handful of fully labeled surgical videos (1, 2, or 5) were available for training. As shown in Table 2, our prototype-driven approach significantly outperforms the GATv2 baseline, with a margin of 19 up to 24 percentage points (p.p.) in accuracy. Even when only a single procedure was used for training, ProtoFlow achieved over 39% accuracy (versus 15% for GATv2), highlighting how self-supervised prototype learning confers stronger generalization than purely supervised GNN methods. These findings underscore the potential value of our method in real-world clinical scenarios, where annotated data for specific procedure variations can be scarce. Learned Prototype Groups A key advantage of ProtoFlow lies in its self-explainable structure, derived from prototype embeddings that capture recurring scene graph patterns. Figure 3 illustrates three learned prototypes for the “Capsulorhexis” phase, each corresponding to a distinct surgical variation: Some surgeons employ only a cystotome, others prefer forceps, and a third group uses both instruments in tandem. Notably, our pipeline discovers these variations entirely via clustering on the latent embeddings, without requiring explicit labeling of which instrument technique was used. Such interpretability aids surgeons and data scientists in verifying whether the model’s “understanding” of a phase aligns with known, clinically meaningful sub-techniques. It could also promote establishing more standardized workflows or verify adherence to predefined ones. The t-SNE visualization of the embedding space in Fig. 4 further illustrates this automatic discovery. The samples for the “Capsulorhexis” phase, located in the bottom right of the plot, visibly separate into roughly three distinct sub-clusters. These groupings correspond directly to the different surgical sub-techniques captured by the prototypes in Fig. 3. This provides strong visual confirmation that our method automatically identifies and separates these clinically meaningful variations within a single surgical phase, without any explicit supervision. The prototypes (starred points) are spread throughout the clusters of each class, showing that they successfully represent distinct variations in action patterns for each phase of the workflow.Fig. 3Learned prototypes (temporal window = 5 frames) for Capsulorhexis, each capturing a distinct instrument usage pattern discovered by the model (use of cystotome, forceps, or both)Fig. 4t-SNE visualization of the learned graph embeddings (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_t$$\end{document}zt) using a temporal window of T=30 frames, with each color representing a different surgical phase. The black stars indicate the positions of the prototype embeddings. The prototypes are situated within their corresponding class clusters, effectively capturing distinct variations in surgical action patterns. This is particularly evident for the “Capsulorhexis” phase (bottom right), where the embeddings visibly separate into sub-groups that align with the different instrument-use patterns discovered by the model (as shown in Fig. 3), highlighting its ability to automatically uncover meaningful workflow variations Learned prototypes (temporal window = 5 frames) for Capsulorhexis, each capturing a distinct instrument usage pattern discovered by the model (use of cystotome, forceps, or both) t-SNE visualization of the learned graph embeddings (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_t$$\end{document}zt) using a temporal window of T=30 frames, with each color representing a different surgical phase. The black stars indicate the positions of the prototype embeddings. The prototypes are situated within their corresponding class clusters, effectively capturing distinct variations in surgical action patterns. This is particularly evident for the “Capsulorhexis” phase (bottom right), where the embeddings visibly separate into sub-groups that align with the different instrument-use patterns discovered by the model (as shown in Fig. 3), highlighting its ability to automatically uncover meaningful workflow variations Handling Workflow Deviations Beyond recognizing standard workflows, ProtoFlow demonstrates resilience in handling low-frequency or emergent surgical phases. Figure 5 highlights this capability using a challenging sequence from a test video where a rare Vitrectomy phase occurs—an event seen only once during training. This situation is further complicated by the presence of a prolapsed iris, a rare surgical complication (Fig. 5A). As the surgeon transitions from the Phacoemulsification phase to the unexpected Vitrectomy, the model immediately registers a significant deviation. As shown in Fig. 5B, the distance from the live scene graph to all learned Phacoemulsification prototypes sharply increases. This spike acts as an automatic flag, signaling that the current workflow is no longer consistent with the established patterns of the preceding phase. Despite the event’s rarity, ProtoFlow correctly classifies the phase as Vitrectomy, demonstrating robust performance where baseline models fail (Fig. 5C). The framework’s interpretability, however, extends beyond just flagging a phase change. By matching node encodings between the input DSG and the learned prototypes, we can generate a more granular explanation. The analysis reveals that the iris node in the input graph is a significant outlier when compared to the corresponding nodes in the standard Vitrectomy prototypes (Fig. 5D). This highlights an anomalous state in the iris’s representation, providing a clear, quantitative signal that directly corresponds to the observed clinical complication (Fig. 5E). This level of interpretability is crucial when encountering unexpected or emergent events; the system not only flags the anomaly but also provides evidence allowing an expert to trace the signal back to the specific components of the scene that triggered its detection. This capability to provide transparent, multi-level explanations for its reasoning in complex situations is essential for building trust and facilitating the clinical adoption of AI systems.Fig. 5ProtoFlow’s interpretability in handling a rare phase (Vitrectomy) with a complication (prolapsed iris). (A) Frames from the test video show the unexpected event. (B) As the procedure transitions from Phacoemulsification to Vitrectomy ( 800 s), the distance to the Phacoemulsification prototypes spikes, flagging a workflow deviation. (C) Quantitative results show ProtoFlow robustly identifies the rare phase. (D, E) The model’s interpretability allows for a granular explanation: The iris node in the input graph is a major outlier compared to the learned Vitrectomy prototypes, providing a quantitative signal directly corresponding to the visible complication. A video of this sequence is attached in the Supplementary ProtoFlow’s interpretability in handling a rare phase (Vitrectomy) with a complication (prolapsed iris). (A) Frames from the test video show the unexpected event. (B) As the procedure transitions from Phacoemulsification to Vitrectomy ( 800 s), the distance to the Phacoemulsification prototypes spikes, flagging a workflow deviation. (C) Quantitative results show ProtoFlow robustly identifies the rare phase. (D, E) The model’s interpretability allows for a granular explanation: The iris node in the input graph is a major outlier compared to the learned Vitrectomy prototypes, providing a quantitative signal directly corresponding to the visible complication. A video of this sequence is attached in the Supplementary End-to-End Results While our main experiments use ground truth (GT) dynamic scene graphs, we also validate our method’s practical performance in an end-to-end setting. We trained and tested our model using graphs generated by CatSGG+ [14], the current SOTA scene graph generation model for CAT-SG [14]. The results in Table 3 indicate a minimal performance gap between using generated graphs and GT, demonstrating our method’s robustness to prediction noise. This robustness is promising, as it paves the way for using self- or weakly supervised scene graph generators to reduce the significant effort of manual annotation.Table 3End-to-end surgical workflow recognition resultsMethodAccuracyF1GT + ProtoFlow80.07 ± 0.1963.21 ± 0.72CatSGG+[14] + ProtoFlow78.66 ± 1.2760.43 ± 0.87Bold indicates the best result per category (1, 2, and 5 training videos) Table 3End-to-end surgical workflow recognition resultsMethodAccuracyF1GT + ProtoFlow80.07 ± 0.1963.21 ± 0.72CatSGG+[14] + ProtoFlow78.66 ± 1.2760.43 ± 0.87Bold indicates the best result per category (1, 2, and 5 training videos) End-to-end surgical workflow recognition results Bold indicates the best result per category (1, 2, and 5 training videos) Conclusion and future work This work establishes that learning dynamic scene graph prototypes is a highly effective strategy for surgical workflow modeling, directly addressing the critical barriers of data scarcity and model opacity. We demonstrated that ProtoFlow achieves competitive accuracy while delivering exceptional robustness in limited-data, few-shot scenarios, significantly lowering obstacles for development in data-scarce clinical settings. Beyond data efficiency, ProtoFlow’s interpretability provides a tangible path toward the transparent AI systems required for clinical adoption. The model automatically discovers clinically meaningful workflow variations, like Capsulorhexis sub-techniques, offering a verifiable foundation for analysis. This capability extends to complex, rare events: The model robustly identifies an unexpected Vitrectomy phase and provides granular, node-level explanations for complications like an iris prolapse. Such detailed, multi-level explanations are essential for building clinical trust. By uniting prototype-based interpretability with robust representation learning, ProtoFlow takes an essential step toward more explainable, reliable, and efficient AI solutions for next-generation surgical data science. Supplementary information We attach a video of the analysis shown in Fig. 5. Supplementary Information Below is the link to the electronic supplementary material.Supplementary file 1 (mp4 65117 KB) Supplementary file 1 (mp4 65117 KB)