End to end AI system for surgical gesture sequence recognition and clinical outcome prediction Authors: Xi Li, Nicholas Matsumoto, Ujjwal Pasupulety, Atharva Deo, Cherine Yang, Jay Moran, Miguel E. Hernandez, Peter Wager, Jasmine Lin, Jeanine Kim, Alvin C. Goh, Christian Wagner, Geoffrey A. Sonn, Andrew J. Hung Published: 2026 ABSTRACT Fine-grained analysis of intraoperative behavior and its impact on patient outcomes remains a longstanding challenge. We present Frame-to-Outcome (F2O), an end-to-end system that translates tissue dissection videos into gesture sequences and uncovers patterns associated with postoperative outcomes. Leveraging transformer-based spatial and temporal modeling and frame-wise classification, F2O robustly detects consecutive short (˜2 s) gestures in the nerve-sparing step of robot-assisted radical prostatectomy (AUC: 0.80 frame-level; 0.81 video-level). F2O-derived features—gesture frequency, duration, and transitions—predicted postoperative outcomes with accuracy comparable to human annotations (0.79 vs. 0.75; overlapping 95% CI). Across 25 shared features, effect size directions were concordant with small differences (∆davg ≈ 0.07), and strong correlation (r = 0.96, p < 1 × 10−14). F2O also captured key patterns linked to erectile function recovery, including prolonged tissue peeling and reduced energy use. By enabling automatic interpretable assessment, F2O establishes a foundation for data-driven surgical feedback and prospective clinical decision support. Introduction The automated quantification of intraoperative surgical activity and analysis of its relationship to clinical outcomes remain a fundamental challenge in surgical data science1–3. While high-resolution robotic and endoscopic video recordings have become more common in modern operating rooms4, the majority of procedures still lack standardized, automated tools for capturing and interpreting intraoperative activities. This gap presents a significant barrier to advancing evidence-based surgery, where novel data-driven insights into performance are critical5. Several longstanding challenges have limited progress in this area. One is the inconsistent and non-standardized surgical terminology6, which has led researchers to develop alternative dictionaries and taxonomies tailored to specific procedures or datasets7–9. Another major bottleneck is the labor-intensive nature of manual video annotation. It requires domain expertise, is highly time-consuming, and is prone to inter-rater variability and subjective interpretation2,6,10. In addition, the high variability in technique across patients, surgeons, procedures, and institutions further complicates model development. Factors such as anatomical differences, the presence of blood, camera motion, and variations in surgical style introduce considerable heterogeneity into video data6. Furthermore, the technical difficulty of modeling fine-grained surgical actions poses significant obstacles to automated recognition6. Collectively, these barriers limit access to granular standardized intraoperative surgical data, constrain assessments of surgical technique, and restrict investigation into relationships between surgical performance and patient outcomes. Surgical workflow analysis spans multiple levels of granularity, from procedure to phase, step, activity, and action triplet recognition, progressing from coarse to fine resolution11. Deep learning approaches, including variants of Convolutional Neural Networks (CNNs) and Transformer-based architectures, have substantially advanced spatiotemporal representation learning in surgical video12–17. At the level of fine-grained activity recognition, modeling rapid transitions and subtle motion patterns remains challenging. Although multimodal approaches, e.g., the inclusion of instrument kinematic data, have been shown to improve performance18–21, such signals are not universally available in routine clinical practice. Moreover, much of the literature on gesture recognition focuses on suturing tasks, particularly those defined in the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) dataset7, a public dry-lab dataset that does not account for dissection gestures, which are more prevalent during surgeries. Our prior work introduced a novel, procedure-agnostic taxonomy for dissection gestures in robot-assisted surgeries8. Gestures here, defined as the smallest meaningful unit of surgical instrument and human tissue interaction, capture the intentional and discrete actions performed by a surgeon during a procedure. Each gesture reflects a specific movement shaped by anatomical context and surgical intent, such as “spreading tissue” or “cutting tissue”. In robot-assisted radical prostatectomy (RARP), for example, the nerve-sparing (NS) step consists of approximately 260 such gestures within a 10-min segment, each lasting an average of 2 s (Fig. 1). Our prior studies show that surgical cases annotated using this gesture classification system revealed strong correlations between skilled surgeons and their propensity to use certain gestures with higher efficacy compared to novice surgeons22–24. Through follow-up studies, these gestures were also found to be predictive of long-term patient outcomes25,26. Thus, accurate identification and classification of these gestures provide a critical foundation for large-scale, data-driven surgical analytics. However, recognizing surgical gestures within their natural sequence poses unique challenges, given the fine temporal resolution, rapid transitions, and high inter-case variability. Prior efforts achieved clip-level gesture classification using human-annotated start and end times, followed by fixed time-interval inference and post hoc aggregation to construct gesture sequences27. While effective for segment-level recognition, such methods require extensive manual annotations and do not model temporal continuity across the full procedural context.Fig. 1Frame-to-Outcome (F2O) is an end-to-end AI system for surgical gesture sequence recognition and clinical outcome analysis.a Surgical videos, such as those from the nerve-sparing (NS) step of robot-assisted radical prostatectomy (RARP), are annotated by trained human raters who identify over ten dominant gesture classes and label the start and end times of each gesture. Each video typically contains ˜260 gestures over a 10-min duration, with an average gesture lasting 2 s. F2O automates the recognition of these fine-grained gestures and enables downstream clinical outcome analysis. b The system processes untrimmed tissue dissection videos and outputs a sequence of standardized surgical gesture probabilities by combining spatial and temporal modeling with frame-wise classification. Specifically, it processes sequences of 16 frames, leveraging spatial and temporal neighbors (red and green) to compute self-attention for each target patch (blue). These context-aware embeddings are then passed through a frame-wise classifier, which produces gesture probability distributions for each frame based on the aggregated representations. c Sequence-based feature engineering is then applied to identify relationships with clinical outcomes, and results are evaluated through concordance analysis, including both feature-level and model-level concordance. Frame-to-Outcome (F2O) is an end-to-end AI system for surgical gesture sequence recognition and clinical outcome analysis. a Surgical videos, such as those from the nerve-sparing (NS) step of robot-assisted radical prostatectomy (RARP), are annotated by trained human raters who identify over ten dominant gesture classes and label the start and end times of each gesture. Each video typically contains ˜260 gestures over a 10-min duration, with an average gesture lasting 2 s. F2O automates the recognition of these fine-grained gestures and enables downstream clinical outcome analysis. b The system processes untrimmed tissue dissection videos and outputs a sequence of standardized surgical gesture probabilities by combining spatial and temporal modeling with frame-wise classification. Specifically, it processes sequences of 16 frames, leveraging spatial and temporal neighbors (red and green) to compute self-attention for each target patch (blue). These context-aware embeddings are then passed through a frame-wise classifier, which produces gesture probability distributions for each frame based on the aggregated representations. c Sequence-based feature engineering is then applied to identify relationships with clinical outcomes, and results are evaluated through concordance analysis, including both feature-level and model-level concordance. To address these challenges, we developed Frame-to-Outcome (F2O), an end-to-end system that automatically recognizes fine-grained gestures in surgical video and supports downstream clinical outcome analysis. While our prior work demonstrated the value of surgical gestures to predict patient outcomes after surgery, the present work allows for the scalable decoding of surgical videos into gesture sequences and further prediction of a patient-reported outcome. Results F2O achieves accurate frame-level classification across surgical gesture classes The core of the F2O pipeline is built on precise frame-level gesture classification. To model the fine temporal granularity of surgical gestures (Fig. 1), the system integrates a Transformer-based spatial and temporal modeling28 with a frame-wise classifier29 (Fig. 1). This architecture is enhanced by stratified and adaptive overlapping sequence sampling, which prioritizes underrepresented gestures based on their prevalence, and by targeted data augmentation strategies. Using 294 annotated RARP NS videos from 4 international centers, spanning 23 surgeons (Figs. 1 and 2, Supplementary Fig 1), the model achieved accurate frame-level classification.Fig. 2F2O accurately classifies frames across classes and videos.a Gesture distribution at occurrence level. Annotations above bars report the percentage of total gesture occurrences and the corresponding count. Gray denotes the remaining 3% excluded from this study: ineffective motion (w), idle time (i), two-hand spread (t), pedicalize (d), coagulate then cut (o), burn (b). b Frame-level performance across gesture classes of cold cut (c), hook (h), clip (k), camera move (m), peel (p), retraction (r), spread (s), assistant (a), coagulation (g), and energy cut (e), evaluated over five randomized data splits. c Temporal alignment between model-predicted gesture probabilities and ground truth when applied frame-by-frame to the entire video. The Highest-probability class is highlighted for visualization. d Video-level performance across 29 full-length videos in the test set, showing the distribution of AUC values. F2O accurately classifies frames across classes and videos. a Gesture distribution at occurrence level. Annotations above bars report the percentage of total gesture occurrences and the corresponding count. Gray denotes the remaining 3% excluded from this study: ineffective motion (w), idle time (i), two-hand spread (t), pedicalize (d), coagulate then cut (o), burn (b). b Frame-level performance across gesture classes of cold cut (c), hook (h), clip (k), camera move (m), peel (p), retraction (r), spread (s), assistant (a), coagulation (g), and energy cut (e), evaluated over five randomized data splits. c Temporal alignment between model-predicted gesture probabilities and ground truth when applied frame-by-frame to the entire video. The Highest-probability class is highlighted for visualization. d Video-level performance across 29 full-length videos in the test set, showing the distribution of AUC values. To evaluate robustness, we conducted five random data splits and measured the area under the receiver operating characteristic curve (AUC). The model achieved 0.80 (95% CI: 0.78–0.81) on the test set. Per-class ROC curves from the 5 test splits are shown (Fig. 2). The model demonstrated consistent classification performance across ten dominant gesture classes, which cover approximately 97% of gestures in a video. Notably, the gesture clip (k) achieved the highest performance (AUC = 0.92), followed by hook (h) (AUC = 0.89), and energy cut (e) (AUC = 0.86) assistant (a) and retraction (r) were among the lower-performing classes (AUC = 0. 71, 0.74), while the remaining gesture classes yielded AUCs ranging from 0.75 to 0.80. These findings demonstrate the capability of F2O to learn temporally precise features that are necessary for fine-grained gestures across various class types. F2O enables automatic gesture sequence recognition across videos To simulate real-world clinical settings, we applied the trained F2O system to 29 full-length videos in the test dataset, each capturing continuous tissue dissection activity during the NS section of RARP. The model predicted gestures continuously across entire recordings. Despite the pronounced class imbalance characteristic of surgical procedures, where some gestures are frequent and others are rare, the model maintained consistently strong performance across all test videos. To illustrate the temporal coherence of gesture predictions, Fig. 2 presents stacked class probability plots for a representative video. The model’s predictions exhibit smooth and interpretable transitions over time, with pronounced shifts in the predicted class distribution closely aligned with ground-truth gesture annotations (indicated in the top bar). For instance, the probability for the clip gesture (Class k) remains close to zero for most of the video, except for three distinct peaks that correspond precisely to the clip events in the ground truth. This alignment underscores the model’s capacity for fine-grained temporal localization, even for short-duration gestures within complex procedures. Figure 2, shows the distribution of video-level AUCs across the 29 videos, ranging mostly from 0.77 to 0.83, with a median AUC near 0.81. The distribution was moderately symmetrical, with only a few outlier videos exhibiting below 0.70. These results demonstrate the model’s robustness in handling continuous, heterogeneous surgical recordings and its capacity to generalize effectively beyond the training examples. Collectively, these results demonstrate that the F2O pipeline not only achieves accurate frame-level classification but also enables automatic recognition of complete gesture sequences in surgical videos. This capability offers strong potential for real-time surgical analysis, retrospective review, and scalable annotation of large-scale surgical datasets. Predicting clinical outcomes using engineered features from surgical gesture sequences To assess the clinical utility of automatically predicted surgical gestures, we evaluated whether sequence-based features engineered from F2O-predicted gesture sequences could accurately predict postoperative erectile function (EF) recovery at 12 months. We compared these predictions to those generated using features derived from manually annotated ground-truth gesture sequences. All predictions were performed using a tabular transformer model trained to classify binary EF outcome based on engineered gesture features (see “Methods”). Engineered features included gesture frequency, transition entropy, gesture duration statistics, and temporal sequence patterns Fig. 3. The same feature engineering pipeline was applied to both ground truth and F2O-derived gesture sequences. Models were trained using stratified 5-fold cross-validation and evaluated by accuracy with 95% confidence intervals computed across splits.Fig. 3Sequence-based features.Included: Gesture frequency, Duration, Sequence structure, Transition probability, Run length, N-grams, Weighted Decay, Dwell time. Sequence-based features. Included: Gesture frequency, Duration, Sequence structure, Transition probability, Run length, N-grams, Weighted Decay, Dwell time. The tabular transformer trained on F2O-derived features to determine a binary classification of EF outcome achieved a mean accuracy of 0.7902 (95% CI: [0.6696, 0.9109]). In comparison, the model trained on human-annotated ground-truth gesture features achieved a mean accuracy of 0.7506 (95% CI: [0.6397, 0.8616]). Performance across both input sources was closely aligned, with overlapping confidence intervals and similar classification metrics, suggesting strong concordance between predicted and manually annotated gesture-derived representations. Feature attributions of two representative cases are shown in Supplementary Fig 2. The correct positive prediction case showed strong positive attributions from features indicating controlled, sustained technique: spread duration, peel-to-peel transitions, and continuous gesture patterns, while the correct negative prediction case exhibited negative attributions from features suggesting technical variability: repeated energy cuts, frequent camera-coagulation transitions, irregular inter-gesture intervals, and extended duration. Interestingly, the mean accuracy of the model trained on F2O-derived features exceeded that of the ground-truth-based model, despite being fully automated. This result suggests that F2O may not only replicate the clinically relevant information embedded in manual annotations, but in some cases improve upon it by producing more temporally consistent or less noisy representations. These findings reinforce the utility of F2O as a scalable and potentially more robust alternative to manual gesture annotation in surgical outcome modeling, where human annotation may introduce bias and inconsistency. These results indicate that F2O-predicted gesture sequences preserve clinically relevant structure and support downstream clinical outcome analysis, as its comparable model performance underscores the feasibility of using F2O for large-scale, automated surgical outcome studies without requiring exhaustive manual labeling. Concordance of clinically informative gesture features To further assess the fidelity of the gesture sequences predicted by F2O, we compared the clinical significance of engineered features derived from F2O and manually annotated sequences. Specifically, we evaluated whether both sources identify similar gesture metrics as being associated with erectile function recovery at 12 months. We conducted two-tailed Student’s t tests (N = 138) for each of 2484 candidate gesture-derived features, using the binary clinical outcome as the target. Features were then ranked by their p-values in both the F2O-derived and ground-truth sets. Among the top 50 features from each source, 25 overlapped (50%), indicating that F20 captures half of the most clinically relevant features detectable through manual annotation. This overlap suggests strong agreement in outcome-relevant signal, despite the absence of human intervention in the F2O pipeline. As the output from the frame level classification is not in the format of the human annotated data, where it is a sequence of gestures with timestamps, the frame classifications are reformatted using a change-point detection (see Methods). Using the optimized change-point detection parameters and gesture-specific weights, F2O produced a sequence of gestures that subsequently go through feature engineering from which we can identify the most strongly correlated with clinical outcome. Of the 50 features identified per set, 25 overlapped across comparisons between good and poor EF outcome groups (Fig. 4), with high concordance based on effect size analysis. A positive effect size indicates that the feature has a higher value in the group with better EF outcomes. Supplementary Table 1 and Fig. 4 categorize these overlapping features as follows:Peel-related metrics: Total peel duration, frequency of peel gestures, and maximum run time of peel gestures are all global peel metrics that are positively correlated with favorable EF outcomes. Two-gram features, which count any sequence of two consecutive gestures, indicate that transitions between peel and camera move in either direction are positively associated with good outcomes. Transition features, representing probabilities of moving from one gesture to another, show that peel to peel and camera move to peel transitions are positively correlated. In contrast, transitions from peel to coagulation are negatively correlated, likely reflecting that peel gestures requiring coagulation, possibly due to bleeding, are associated with worse EF outcomes.Spread-related metrics: Average duration of continuous spread gestures, maximum duration of continuous spread gestures, total frequency of spread gestures, and variability in the time between repeated spread gestures all show positive correlation with favorable EF outcomes. Transition features such as spread to spread, peel to spread, and cold cut to spread are also positively correlated.Clip-related metrics: Greater variability in clip duration and longer average clip durations are observed in patients with favorable EF outcomes. This may indicate more deliberate and careful clipping.Energy-related metrics: Higher frequency of coagulation gestures, more repeated energy cuts, and transitions from hook to coagulation are all negatively correlated with EF outcome. This suggests that increased use of energy-based tools may be associated with poorer outcomes.Other global metrics: Longer total surgical duration and greater overall camera move are positively correlated with favorable EF outcomes.Fig. 4Effect size comparisons between the overlapping significant features of F2O and the ground truth gesture-derived features.Positive effect size shows that the feature is correlated with better EF outcome. All features show the same direction and representation of populations. Peel-related metrics: Total peel duration, frequency of peel gestures, and maximum run time of peel gestures are all global peel metrics that are positively correlated with favorable EF outcomes. Two-gram features, which count any sequence of two consecutive gestures, indicate that transitions between peel and camera move in either direction are positively associated with good outcomes. Transition features, representing probabilities of moving from one gesture to another, show that peel to peel and camera move to peel transitions are positively correlated. In contrast, transitions from peel to coagulation are negatively correlated, likely reflecting that peel gestures requiring coagulation, possibly due to bleeding, are associated with worse EF outcomes. Peel-related metrics: Total peel duration, frequency of peel gestures, and maximum run time of peel gestures are all global peel metrics that are positively correlated with favorable EF outcomes. Two-gram features, which count any sequence of two consecutive gestures, indicate that transitions between peel and camera move in either direction are positively associated with good outcomes. Transition features, representing probabilities of moving from one gesture to another, show that peel to peel and camera move to peel transitions are positively correlated. In contrast, transitions from peel to coagulation are negatively correlated, likely reflecting that peel gestures requiring coagulation, possibly due to bleeding, are associated with worse EF outcomes. Spread-related metrics: Average duration of continuous spread gestures, maximum duration of continuous spread gestures, total frequency of spread gestures, and variability in the time between repeated spread gestures all show positive correlation with favorable EF outcomes. Transition features such as spread to spread, peel to spread, and cold cut to spread are also positively correlated. Spread-related metrics: Average duration of continuous spread gestures, maximum duration of continuous spread gestures, total frequency of spread gestures, and variability in the time between repeated spread gestures all show positive correlation with favorable EF outcomes. Transition features such as spread to spread, peel to spread, and cold cut to spread are also positively correlated. Clip-related metrics: Greater variability in clip duration and longer average clip durations are observed in patients with favorable EF outcomes. This may indicate more deliberate and careful clipping. Clip-related metrics: Greater variability in clip duration and longer average clip durations are observed in patients with favorable EF outcomes. This may indicate more deliberate and careful clipping. Energy-related metrics: Higher frequency of coagulation gestures, more repeated energy cuts, and transitions from hook to coagulation are all negatively correlated with EF outcome. This suggests that increased use of energy-based tools may be associated with poorer outcomes. Energy-related metrics: Higher frequency of coagulation gestures, more repeated energy cuts, and transitions from hook to coagulation are all negatively correlated with EF outcome. This suggests that increased use of energy-based tools may be associated with poorer outcomes. Other global metrics: Longer total surgical duration and greater overall camera move are positively correlated with favorable EF outcomes. Other global metrics: Longer total surgical duration and greater overall camera move are positively correlated with favorable EF outcomes. Effect size comparisons between the overlapping significant features of F2O and the ground truth gesture-derived features. Positive effect size shows that the feature is correlated with better EF outcome. All features show the same direction and representation of populations. An analysis of the gesture types represented within these top 50 significant feature sets further underscored the systems’ alignment. For the F2O features, those related to peel gestures (14 features) and spread (9 features) were most prevalent. Similarly, for the ground-truth features, peel gestures (9 features) and spread (7 features) were also the most frequently represented among the top significant features. This suggests that both systems converged on the importance of these particular gestures in relation to the clinical outcome. To further evaluate the concordance between F2O and ground truth-derived features, we computed Cohen’s d effect size for each of the 25 overlapping features, quantifying the magnitude of difference between outcome classes. These features represent the most clinically significant variables shared across both feature sets and are expected to convey a consistent narrative. A positive d value indicates a higher mean in the better outcome group compared to the poor outcome group. All effect directions (signs of d) were concordant between systems. The mean absolute difference in effect size magnitude was minimal (∆davg ≈ 0.07). Cohen’s d values from Frame-to-Outcome and ground-truth annotations were highly correlated (r = 0.96, n = 25, p < 1 × 10−14), confirming that the AI-derived features approximate manual annotations not only in statistical significance but also in clinical effect size (See Fig. 4). These 25 overlapping features reveal that deliberate, tissue-sparing gestures (longer peel, spread, and clip actions; varied instrument sequences) favor erectile-function recovery, whereas frequent, repetitive energy use (coagulation and energy cuts) and hurried maneuvers increase the risk of postoperative dysfunction. F2O thus not only replicates manual scoring but pinpoints specific surgical behaviors with clear mechanistic ties to patient outcomes. This high degree of alignment further supports that F2O not only captures statistically significant features but also approximates their clinical relevance as quantified by effect size. These findings demonstrate that F2O reliably extracts clinically meaningful gesture features that closely match expert manual annotation, supporting its potential utility in automated clinical outcome prediction. Generalizability across Transformer backbones To support long-term adaptability and computational efficiency, the F2O system was designed with a backbone-agnostic architecture capable of integrating emerging transformer models without retraining from scratch. Frame-level performance across three state-of-the-art backbones, VideoMAE V230, TimeSformer28, and ViViT31, is shown in Fig. 5. The system achieved AUCs of 0.79, 0.78, and 0.73, respectively. These results confirm the generalizability of the pipeline and its compatibility with evolving transformer architectures. This flexibility is particularly important in academic and clinical research environments, where access to large-scale computational infrastructure is often limited. For instance, pretraining a ViT-g model with VideoMAE V2 requires around two weeks on 64 GPUs. The ability to interchange backbones without modifying the rest of the system ensures long-term scalability and alignment with advances in video representation learning.Fig. 5Experiments across backbones and data scales showed consistent performance.a Comparison of model performance across different transformer backbones, illustrating architectural flexibility. b Impact of key design components on classification performance, including the frame classifier architecture (Temporal transformer encoder), pretraining weights (Kinetics-400), and backbone optimization methods (Partial unfreezing, low-rank adaptation), measured as relative changes from the baseline system. c Model performance across increasing training data volumes, demonstrating robustness in low-data scenarios and scalability to new sites. Experiments across backbones and data scales showed consistent performance. a Comparison of model performance across different transformer backbones, illustrating architectural flexibility. b Impact of key design components on classification performance, including the frame classifier architecture (Temporal transformer encoder), pretraining weights (Kinetics-400), and backbone optimization methods (Partial unfreezing, low-rank adaptation), measured as relative changes from the baseline system. c Model performance across increasing training data volumes, demonstrating robustness in low-data scenarios and scalability to new sites. Impact of architectural and training modifications To quantify the contributions of key design components, we conducted studies (see “Methods”), measuring changes in frame-level classification AUC relative to the baseline configuration (Fig. 5). The components assessed included the frame classifier architecture, pretraining weights, and backbone optimization methods. Incorporating a Temporal Transformer Encoder (TTE) and enhancing the backbone with Low-Rank Adaptation (LoRA)32 both led to modest AUC gains, indicating that targeted improvements to temporal modeling and fine-tuning can further enhance classification accuracy. Conversely, using a backbone pretrained on Kinetics-40033 led to a 0.06 AUC drop compared to SSv234, highlighting the importance of domain-aligned pretraining. Unlike Kinetics-400’s coarse, long-duration actions, SSv2 contains fine-grained, short-term activities that better match the temporal characteristics of surgical gestures. Collectively, these results demonstrate that F2O is both a high-performing and adaptable framework for surgical gesture recognition. By avoiding dependence on computationally intensive pretraining and supporting targeted adaptation techniques, the framework promotes sustainable model development and enhances reproducibility, key priorities as video representation learning continues to evolve. Scalability to sites with limited data A key translational challenge for AI in healthcare is deployment to sites with limited annotated data. To assess the model’s scalability under such constraints, we performed data scaling experiments using subsets of the training data: 10%, 25%, 50%, 75%, and 100%. We evaluated performance with the AUC of the same test dataset (Fig. 5). The model exhibited strong data efficiency. With only 10% of the training data (˜24 videos), it achieved an AUC of 0.70 over 90% of the full-data AUC (0.78). The largest performance gains occurred before reaching 50% of the data. This behavior indicates that the system captures essential gesture patterns early in the learning curve, enabling rapid deployment even with limited training resources. These findings illustrate the model’s capacity to generalize to new clinical environments with minimal labeled data, facilitating broader adoption of surgical gesture recognition in diverse healthcare settings. Discussion We demonstrate that an automatic method for capturing standardized, fine-grained actions across surgical videos can lay the foundation for scalable analysis of intraoperative behavior and its relationship to postoperative outcomes. While various AI-based approaches to analyze surgical video have been explored, a systematic, standardized method to link intraoperative video to outcomes has remained elusive due to persistent challenges in terminology, annotation, data variability, and modeling complexity5,10,35. These gaps have collectively delayed the integration of intraoperative activity-based analysis into outcome research, limiting the ability to understand how specific surgical actions influence recovery, complications, and overall patient outcomes. In this study, we introduce F2O, an end-to-end system that translates intraoperative video into a sequence of standardized surgical gesture classes (average duration: 2 s), providing interpretable and clinically meaningful signals embedded in surgical behavior. This system addresses the longstanding need for automated, standardized intraoperative measurements and their relationship to clinical outcomes36. Our results have demonstrated that F2O (1) achieves accurate frame-level classification across surgical gesture classes, (2) enables automatic gesture sequence recognition across videos, and (3) supports interpretable downstream analysis of surgical outcomes. In contrast to prior methods reliant on coarse time resolution and manually annotated boundaries37–39, F2O leverages vision transformer-based spatial and temporal modeling and frame-wise classification to provide high-resolution predictions with a mean AUC of 0.80 (95% CI: 0.78-0.81) across ten dominant surgical gestures, which account for over 97% of gestures in the dataset. When applied to full-length tissue dissection videos, the system produces smooth and temporally coherent predictions that align with ground truth annotations (Fig. 2), achieving strong performance in its prediction of 12-month functional recovery, despite heterogeneity in case complexity and gesture distribution. This automated process effectively replaces the need for manual gesture annotation in surgical video analysis, which requires the development of procedure-specific guides for training personnel40,41. Although the current gesture recognition performance is sufficient for predicting recovery of erectile function after prostatectomy, further improvements would enhance robustness, enable broader downstream applications and automated data-driven analysis of surgical procedures. Areas for further advancement are most apparent among the lower-performing gesture classes, where performance likely reflects several intrinsic challenges of fine-grained surgical gesture recognition. First, some classes had limited and less diverse training samples, which constrained generalization. For example, assistant (a) occurred only three times within the video (Fig. 2). Second, certain gestures were brief, sparsely distributed, or occurred during rapid transitions within a procedure, making temporal boundaries more difficult to localize. This was particularly relevant for retraction (r). Third, some classes exhibited substantial visual similarity or co-occurred with other actions, increasing classification ambiguity. For instance, peel (p) and cut (c) are performed using the same instrument and often occur together, while peel (p) may visually appear with either open or closed instrument tips, making reliable separation more challenging. Addressing these challenges through larger and more diverse datasets, targeted class-specific optimization strategies, and noise-aware or diffusion-based temporal modeling may further improve overall AUC and class-level robustness. Beyond classification accuracy, our results demonstrate that F2O not only replicates the statistical significance of manual annotations (high overlap in top-ranked features and r = 0.96 Pearson correlation of Cohen’s d values) but also uncovers the same clinically relevant behaviors that experienced surgeons intuitively value. In particular, deliberate, sustained gestures—longer cumulative peel durations (dur_sum_p, d = +0.51 vs. +0.41 (F2O vs Ground truth respectively)) and extended spread runs (avg_run_s, d = +0.58 vs. +0.32) were enriched in the favorable-outcome cohort. These metrics align with careful tissue manipulation around neurovascular bundles and support nerve preservation42–44. Conversely, frequent coagulation gestures (freq_g, d = −0.50 vs. −0.39) and repeated energy transitions (trans_e_e, d = −0.44 vs. −0.43) correlated with poorer functional recovery, likely reflecting energy-related damage to the nerves45. Thus, F2O indicates similar interpretable behavioral markers to the ground truth dataset, such as transition ratios and n-gram patterns, that map directly onto surgical technique and patient outcome. Methodologically, the present framework differs from prior studies along four principal dimensions. First, it operates exclusively on raw surgical video, without incorporating instrument kinematics or auxiliary sensor data. While multimodal inputs have improved recognition performance in experimental settings18–21, video remains the universally retained intraoperative data stream, making video-only modeling essential for scalable clinical translation. Second, whereas much of the existing literature centers on suturing gestures derived from controlled dry-lab datasets such as JIGSAWS7, this study focuses on tissue dissection gestures occurring within real-world clinical procedures. Dissection introduces substantially greater anatomical variability, tissue interaction complexity, and temporal continuity, requiring modeling strategies that extend beyond predefined task units. Third, gestures are not treated as isolated clip-level classifications. While prior systems such as SAIS27 demonstrated strong classification performance across six dissection gestures, their predictions were primarily evaluated at the segment level. In contrast, F2O achieves comparable performance while extending into temporally ordered sequences spanning the entire dissection phase. This design preserves procedural continuity rather than limiting outputs to discrete recognition events. Finally, the framework is backbone-agnostic and demonstrates stable performance across multiple transformer architectures, indicating structural robustness independent of any single model instantiation. Together, these design principles distinguish the system from prior gesture-recognition approaches and position it as an automated behavioral quantification framework suitable for outcome-aligned surgical analytics. From a clinical perspective, the most intractable challenge has been the delayed nature of surgical end-points, which often only manifest months or years postoperatively46,47. Traditional feedback to the surgical team is therefore episodic and anecdotal, lacking granularity on which segment or instrument maneuver drove success or complication. By contrast, our sequential-analysis approach automatically digitizes every frame into gesture events and quantifies their impact on the eventual clinical outcome. This creates a closed feedback loop: surgeons can review which specific peel or spread gestures differentiated high-functioning cases, and they can iteratively refine their technique in near real time. Such granular and automated feedback has the potential to accelerate skill acquisition beyond what retrospective chart review can achieve48,49. In addition to its clinical relevance, F2O’s modular architecture underpins its generalizability. The frame-to-gesture model, the change-point detection, and gesture-based feature engineering can be retrained or altered on diverse endoscopic procedures, from laparoscopy, bronchoscopy, to arthroscopy, with minimal code changes, supported by a predefined standardized dissection gesture taxonomy. Outcome labels are equally flexible, extending beyond erectile functional recovery metrics to include hospital length of stay after surgery, post-operative complication rates, and patient-reported outcomes50. Consistent performance across various backbones demonstrates the system’s plugin capability, eliminating the need for weeks of computation on high-end GPUs51. As more pre-trained vision models and larger cohorts become available, the pipeline can be updated in a plug-and-play fashion, ensuring continuous improvement without imposing new burdens on the user52 and supporting resource-limited clinical environments. Furthermore, F2O exhibits strong data efficiency. With only 10% of training data (˜24 videos), it achieved over 90% of the full-data AUC. This enables practical implementation in new clinical sites where large-scale data collection is not feasible. Despite these strengths, several limitations warrant discussion. First, our cohort (N = 138) focused on a single procedure type and a binary outcome measure; larger, multi-center studies are needed to validate generalizability across institutions and more nuanced clinical endpoints. Second, although we achieved high agreement with manual annotation, black swan events, rare, or subtle gestures may escape robust detection due to limits on the number of types of gestures able to be analyzed and limited data to train the model on such gestures. Future algorithms should integrate temporal context or unsupervised gesture discovery to capture these events. Third, although the framework is computationally efficient at inference, real-time intraoperative deployment was not evaluated in this study. Under the tested configuration (single NVIDIA L40S GPU, 48 GB VRAM; 16 CPU cores; 256 GB RAM), gesture classification required ˜11 min to process a 20-min video (˜1.8× real-time throughput), with peak GPU memory utilization of 23 GB under FP16 precision. While these results provide an initial indication of computational feasibility, practical implementation would require a formal assessment of latency variability, hardware compatibility, and integration within existing surgical video infrastructure. A user-friendly interface that allows secure, de-identified inference and analysis of surgical videos would be particularly beneficial, especially for early-career surgeons. Finally, while the present study focused on gestures for clinical outcome prediction, patient factors may also contribute. Our prior work explored relevant patient factors and found that there was a negligible contribution from the patient characteristics in predicting recovery of Erectile Function after prostatectomy (EF). An eventual model utilized in clinical practice to predict outcomes should nonetheless account for patient factors. Looking forward, we can think of some examples where F2O’s framework shows promise in the near future: (1) Prospective validation and clinical trials. Embedding F2O into live cases and correlating its real-time feedback with subsequent outcomes will establish its efficacy as a training aid and quality-assurance tool. (2) Adaptive, outcome-weighted gesture coaching. By linking specific gesture scores to predictive risk, one could create a dashboard that alerts the surgeon if their current sequence deviates from patterns associated with good outcomes, enabling immediate corrective action. (3) Cross-domain transfer and meta-analysis. Aggregating gesture-outcome mappings across multiple procedure types may reveal universal principles of tissue handling and instrument use, informing best-practice guidelines and AI-augmented surgical curriculum. (4) Public tool integration for training and decision support. Deploying F2O within publicly accessible platforms could support surgical training, quality assurance, and outcome-informed decision-making by providing interpretable feedback and scalable gesture analytics across diverse clinical settings. In summary, we present F2O, an automated, end-to-end system that transforms intraoperative surgical video into standardized, fine-grained gesture representations and links these measurements to postoperative outcomes. Across a multi-institutional cohort of robot-assisted prostatectomy videos, the system demonstrates strong gesture recognition performance and concordance with expert annotations in outcome analyses, enabling high-resolution quantification of surgical behavior without reliance on manually annotated temporal boundaries. Importantly, F2O is designed as a measurement and analysis infrastructure, and the principal contribution lies in enabling scalable, standardized assessment of intraoperative behavior to support downstream outcome research, surgical quality analysis, and hypothesis generation. While the current study focuses on a single procedure and outcome, the modular design of the framework allows adaptation to other endoscopic procedures, outcome measures, and model backbones as data availability expands. By bridging raw surgical video and clinically meaningful behavioral metrics, F2O establishes a foundation for systematic investigation of how operative technique influences patient outcomes, advancing the role of data-driven analysis in surgical research and education. Methods F2O Leverage Transformer-based spatial and temporal modeling and frame-wise classification, along with standardized surgical gesture terminology and sequence feature engineering, the system translates untrimmed tissue dissection videos to sequences of consecutive standardized surgical gestures without human efforts and identifies important gesture patterns associated with the outcome. Ethics approval The study protocol was approved by the Institutional Review Board of the primary institution under protocol number STUDY00002828, preserving erectile function by quantifying the nerve-sparing step of the robotic prostatectomy. Informed consent was obtained. Surgical gesture data Surgical motion revolves around moment-to-moment gestures, which serve as the fundamental units of action, similar to binary code in computing. Decoding procedures at the gesture level enables standardized representations of surgical workflows and enhances both performance evaluation and postoperative outcome prediction. Videos from the nerve-sparing (NS) section of robot-assisted radical prostatectomy (RARP) exhibit rich gesture-level activity, with approximately 260 gestures in a 10 min video, each averaging 2 s (Fig. 1). These gestures flow continuously and, when annotated, provide valuable insights for outcome prediction and performance assessment. Previous work established a dissection surgical gesture classification system consisting of dissection gestures (e.g., cold cut) and supporting gestures (e.g., retraction) (Fig. 1) and demonstrated the relationship between gestures used during the NS step of RARP and postoperative erectile dysfunction rates. In this study, we used 294 annotated RARP-NS videos collected from four international institutions (University of Southern California [USC], Memorial Sloan Kettering Cancer Center [MSK], St. Antonius Hospital [SAH], and Stanford University Medical Center [SMC]). The dataset includes surgeries performed by 23 unique surgeons, representing the diverse real-world scenario. Each video captures either the left or right segment of a nerve-sparing procedure. Ground-truth gesture annotations were provided by trained human raters following previously established guidelines and taxonomy of dissection gestures8, where each gesture is defined as a discrete movement bounded by brief pauses in motion. In cases of simultaneous instrument activity, the predominant gesture corresponding to the dominant hand of the console surgeon was annotated. Temporal boundaries were defined sequentially: raters recorded the end time of each gesture, and the start time was implicitly given by the end of the preceding gesture, resulting in a continuous, non-overlapping segmentation of the video. On average, each video contains ˜260 gestures, each lasting about 2 s. Over 97% of gestures belong to 10 dominant classes, including cold cut (c), hook (h), clip (k), camera move (m), peel (p), retraction (r), spread (s), assistant (a), coagulation (g), and energy cut (e). The remaining 3% are labeled as X and excluded from automated gesture recognition due to insufficient training data for these infrequent classes. Gesture occurrence frequencies across the full dataset are shown in Fig. 2. For outcome prediction, only procedures with bilateral nerve-sparing video components were included, yielding a final cohort of 138 patients. Spatial and temporal modeling To capture high-level spatial features and temporal dependencies between frames, the system employs spatial and temporal modeling using vision transformers, which interpret video as a sequence of patches extracted from the individual frames, and each patch is mapped into an embedding and augmented with positional information. We adopted TimeSformer28, a vision transformer specifically designed for general video understanding. Given a sequence of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F$$\end{document}F RGB frames of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H\times W$$\end{document}H×W, TimeSformer divides each frame into \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=\frac{H\times W}{{P}^{2}}$$\end{document}N=H×WP2 non-overlapping patches of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P\times P$$\end{document}P×P. Each Patch \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{x}}}_{(p,t)}\in {{\mathbb{R}}}^{3{P}^{2}}$$\end{document}x(p,t)∈R3P2 is linearly projected into a D-dimensional embedding:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{z}}}_{(p,t)}^{(0)}={\bf{E}}{{\bf{x}}}_{(p,t)}+{{\bf{e}}}_{(p,t)}^{\mathrm{pos}}$$\end{document}z(p,t)(0)=Ex(p,t)+e(p,t)poswhere \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\bf{E}}\in {{\mathbb{R}}}^{D\times 3{P}^{2}}$$\end{document}E∈RD×3P2 is a learnable matrix, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{e}}}_{(p,t)}^{\mathrm{pos}}\in {{\mathbb{R}}}^{D}$$\end{document}e(p,t)pos∈RD is a learnable spatiotemporal positional embedding, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p\in \{1,\ldots ,N\}$$\end{document}p∈{1,…,N} is the spatial patch index, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t\in \{1,\ldots ,F\}$$\end{document}t∈{1,…,F} is the frame index. Temporal and spatial attention modules are applied sequentially, enabling the model to learn rich spatiotemporal features from frame sequences. An example visualization is shown in Fig. 1, spatial and temporal modeling, where red and green highlighted patches are used for self-attention computation of the blue patch. After processing through \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L$$\end{document}L layers of divided space-time attention, the Transformer outputs a sequence for batch size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$B$$\end{document}B:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\bf{Z}}=\left[{{\bf{z}}}_{(0,0)}^{(L)},{{\bf{z}}}_{(1,1)}^{(L)},\ldots ,{{\bf{z}}}_{(N,F)}^{(L)}\right]\in {{\mathbb{R}}}^{B\times ({NF}+1)\times D}$$\end{document}Z=z(0,0)(L),z(1,1)(L),…,z(N,F)(L)∈RB×(NF+1)×D We selected TimeSformer-HR, a high resolution variant that operates on 16 frames of size 448 × 448, over default TimeSformer (8 frames at 224 × 224) and the long range TimeSformer-L (96 frames at 224 × 224). TimeSformer-HR strikes a balance between temporal coverage and spatial fidelity, allowing it to model longer temporal dependencies without being distracted by minor inter-frame variations. For pretraining, we selected Something-Something V2 (SSv2) as pretrain weights, as its temporal complexity more closely resembles the high-frequency and short-duration surgical activity, compared to Kinetics-400 (K400), known for capturing general human actions. With the selected pre-trained backbone, our system processes each frame to extract high-level spatial features. Temporal relationships between frames are captured to understand action evolution over time. This dual approach ensures that both spatial and temporal information contribute to action recognition. Frame-wise classification Frame-wise classification plays a key role in detecting transitions between actions, especially given that gestures average only 2 s and often change rapidly. For instance, a clip gesture may transition to a cut gesture in less than a second, necessitating high temporal resolution in classification. Previous efforts have attempted to learn frame-level representations, but surgical activity recognition at this granularity has not yet been extensively developed. In our framework, temporal features extracted from TimeSformer are aggregated per frame. We excluded the classification token \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{z}}}_{(0,0)}^{(L)}$$\end{document}z(0,0)(L) and reshaped the remaining output tokens to recover frame-wise structure:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{Z}}}^{{\prime} }={\rm{reshape}}\left({{\bf{Z}}}_{\left[:,1:,:\right]},\left(B,F,N,D\right)\right)$$\end{document}Z′=reshapeZ:,1:,:,B,F,N,DTo obtain frame-level features, patch embeddings are averaged within each frame:4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{F}}}_{t}=\frac{1}{N}\mathop{\sum }\limits_{p=1}^{N}{{{\bf{Z}}}^{{\prime} }}_{[:,t,p,:]}\,{\rm{for}}\,{t}=1,\ldots ,F$$\end{document}Ft=1N∑p=1NZ′[:,t,p,:]fort=1,…,FThis produces the final frame-level representation:5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\bf{F}}\in {{\mathbb{R}}}^{B\times F\times D}$$\end{document}F∈RB×F×D To classify gesture classes at the frame level, we explored two lightweight frame-wise classifiers. A simple linear layer maps each 768-dimensional feature vector to a 10-dimensional vector of logits, effectively learning to classify each frame into one of the 10 gesture categories. A lightweight TTE on top of the TimeSformer backbone consists of a single Transformer encoder layer with four attention heads to capture diverse temporal patterns across gesture sequences. This is followed by a two-layer multilayer perceptron (MLP) that maps the temporally encoded frame features to class probabilities. The MLP includes a 256-dimensional hidden layer with ReLU activation and dropout regularization (p = 0.3), followed by a final linear projection to the number of gesture classes. These lightweight classifiers maintain computational efficiency while maximizing the utility of the pretrained backbone for fine-grained frame-level recognition. Video dataset processing and frame classifier training pipeline The videos were split into training (80%), validation (10%), and test (10%) sets. Each frame was labeled according to the manually annotated start and end times. Stratified and adaptive overlapping sequence sampling was performed at adjusted FPS, adjusted sampling stride based on class weights to handle diverse FPS across different hospitals, to enrich the minority classes, and to ensure an equal number of samples per gesture class. This step is to ensure diversity and sufficient data coverage across the train, validation, and test datasets, for example, the dataset comprising 80,000 frames per class in the training dataset for the 10 most common gestures, validation and test sets containing 8000 frames per class. Data augmentation techniques, including zoom and rotation, were applied to improve generalizability. See Supplementary Fig 1 for gesture class distributions across the training, validation, and test splits and institutions. Model training was conducted using a distributed data parallel (DDP) framework with mixed precision (AMP) to enable efficient multi-GPU training. We optimized a class-balanced cross-entropy loss using the Adam optimizer with weight decay and a cosine annealing learning rate schedule with linear warm-up. During training, data were shuffled to ensure representative sampling. Performance was evaluated on a validation set using frame-level area under the ROC curve (AUC) and loss. Early stopping was applied based on validation loss, and model checkpoints were saved every five epochs. In addition to AUC, we computed per-class AUC on the test dataset to assess class-specific performance (Fig. 2). Automatic gesture sequence recognition in a video Once the frame-level classification model was trained, it was applied to full-length, untrimmed videos using a sliding window approach. Videos were uniformly sampled at a fixed time interval (e.g., 0.1667 s), and non-overlapping frame sequences of length 16 were passed through the model. The model processes each sequence by capturing both spatial context and motion patterns, aggregating temporal embeddings, and producing frame-by-frame gesture class probabilities. For each video, class-wise AUCs were computed only for gesture classes present in the ground truth. The mean AUC across these present classes was then reported as the video-level AUC (Fig. 2). This approach ensures that model performance is assessed fairly and without penalization for absent classes. For each video, we generated visualizations of predicted gesture probability trajectories and exported structured probability files, which serve as inputs for downstream tasks such as change point detection and clinical outcome prediction. Gesture aggregation via change point detection In order to train downstream applications such as clinical outcome prediction, frame-level gesture predictions need to be aggregated into temporally meaningful gesture sequences akin to human-annotated gesture sequences. To create aggregated gesture sequences, where each gesture spans 2–3 s, the model’s class probability distributions associated with each frame were leveraged. These per-frame probability vectors were stacked into a 2D signal matrix and used as input to a change point detection algorithm to identify significant distributional shifts in probability53. The detected change points then partitioned the sequential frame-wise predictions into contiguous segments, analogous to the continuous annotation framework. Each segment was assigned a dominant gesture label based on the highest average class probability across frames within that segment, and a final sequence of gestures was created for the video. This design ensures that the resulting segments follow the same sequential, non-overlapping structure as the manual annotations, while remaining robust to frame-level noise. An implementation of linearly penalized segmentation (Pelt) in the ruptures Python library was used due to its ability to identify shifts within non-linear distributions54. The Pelt model parameter was set to rbf. The penalty value, which determines the sensitivity of the change point detector, was fixed after empirically assessing values in the range [0,1]. A penalty value of 0.5 resulted in a reasonable alignment between the lengths of the aggregated and human-annotated gesture sequences. Feature engineering for clinical outcome prediction To capture the temporal and sequential dynamics of user gestures, we engineered a comprehensive set of features from raw gesture, timestamp, and duration data. These features were categorized into several domains to reflect different behavioral and statistical properties of gesture sequences (Fig. 1). In total, 2484 features were created per case. Frequency-based features are normalized frequencies for each gesture type, quantifying relative occurrence across the entire sequence. This included global gesture frequencies, as well as exponential decay-weighted counts that emphasize more recent gestures. The decay-weighted approach ensures temporal sensitivity by assigning higher importance to gestures closer to the present moment. Temporal features are time-based features that capture both global and gesture-specific timing information. These included: (1) total sequence duration—time between the first and last gesture, (2) gesture rate—gestures per unit time, (3) time since last occurrence for each gesture, calculated relative to the current timestamp. These features characterize user activity intensity and temporal recency of individual gestures. Sequence structure features quantified the structural properties of gesture sequences. We included: (1) the number of unique gestures used, (2) the total number of gesture changes, (3) the Shannon entropy of the gesture distribution. These metrics reflect both the diversity and predictability of the user’s gestural behavior. N-gram and transition features incorporated n-gram statistics (for n = 2 and 3) to capture local sequential dependencies. Each n-gram feature represents the normalized frequency of a contiguous subsequence of gestures. Transition probabilities between gesture pairs were estimated, modeling the likelihood of one gesture following another. These features reflect the short-term temporal dynamics of gesture transitions. Dwell-time features computed inter-gesture timing statistics (dwell times) to capture rhythm and pacing, including: (1) mean, standard deviation, minimum, maximum, and median of dwell times, (2) higher-order moments including skewness and kurtosis of dwell distributions, (3) gesture-specific dwell time statistics preceding each gesture type. These features describe the temporal granularity of user interactions and may indicate hesitation or fluidity in motion. Gesture duration features extracted aggregate statistics for the duration of each gesture, including mean, standard deviation, skewness, kurtosis, and total duration. These features were computed both globally and separately for each gesture class, enabling characterization of gesture-specific temporal patterns and variability. Run-length encoding features modeled gesture repetition and persistence, included: (1) the maximum run length, (2) the average run length of consecutive occurrences. These features quantify how often users repeat or sustain gestures, which may relate to intent or error states. The concordance analysis was designed to evaluate whether features derived from automatically recognized gesture sequences capture similar outcome-relevant information as those derived from manual annotations. Redundancy was mitigated through preprocessing steps such as variance thresholding and subsequent model-based selection of discriminative features, which reduce the influence of repetitive or non-informative variables. Comparison across backbones The system leverages pretrained foundational models to maintain flexibility in adopting state-of-the-art architectures while minimizing computational demands. Recent vision Transformer models are typically trained on substantial hardware resources. For instance, TimeSformer, published in 2021, was trained using 32 GPUs, while pretraining a ViT-g model with VideoMAE V2 (2023) requires around two weeks on 64 A100 GPUs. In contrast, academic institutions often rely on limited and aging GPU infrastructure, creating barriers to adopting such resource-intensive workflows. To address this, our system is designed to maximize the utility of publicly available pretrained models. We benchmarked three Transformer backbones(TimeSformer, ViViT, and VideoMAE) within our frame-wise classification pipeline. To ensure a fair comparison, each backbone was frozen during training, and only the lightweight linear frame classifier was trained. Training and evaluation were conducted using consistent hyperparameters and data processing settings. Results are reported as frame-level AUCs, visualized in a boxplot (Fig. 5). VideoMAE backbone To evaluate the generalizability of our framework, we implemented a version of the model using VideoMAE-v2 as the backbone for frame-wise gesture classification. We adapted the vit_giant_patch14_224 model from the official VideoMAE repository and initialized it with pretrained weights fine-tuned on SSv2. The model processes sequences of 16 video frames and extracts spatiotemporal patch embeddings via the transformer encoder. These embeddings are pooled per frame to form a compact temporal representation, which is then passed through a lightweight linear classifier to predict gesture labels. ViViT backbone We also implemented a model variant using ViViT as the backbone for frame-wise gesture classification. We leveraged a pretrained ViViT model from Hugging Face to extract spatiotemporal patch embeddings from sequences of 16 video frames. The tokenized output, excluding the CLS token, was reshaped to recover per-frame representations by averaging over patches. These frame-level features were then passed through a lightweight frame classifier to classify each frame into gesture categories. This implementation allowed us to compare ViViT with other backbones under a consistent architecture and training setup. Model variant studies To evaluate the contribution of individual components to overall performance, we conducted a series of experiments around key components: frame classifiers, pre-trained weights, and backbone optimization. The experiments included adding a temporal transformer encoder in the frame classifier, comparing different pretrained weights (TimeSformer fine-tuned on SSv2 vs. Kinetics-400), and exploring various backbone fine-tuning strategies such as partial unfreezing and LoRA. We used TimeSformer-HR SSv2 with a linear frame classifier as the baseline, keeping similar hyperparameters, data processing, and augmentation. These experiments provided insights into how each architectural and training decision influenced temporal modeling and gesture classification performance. Data scaling Data scaling experiments were conducted to assess how model performance varies with increasing amounts of training data. Our dataset comprises 294 videos of NS sections of RARP surgeries collected from four international sites, involving approximately 23 surgeons. The number of videos per site and per surgeon varies substantially from around 10 to 140 surgeries, due to the complexity and difficulty of surgical video data collection and cross-institutional sharing. To evaluate the model’s scalability to new sites with limited data, we assessed its performance under restricted training data conditions, simulating deployment in settings where only small-scale datasets are available. We performed stratified random splits at the video level to divide the dataset into train/validation/test sets (80/10/10), ensuring balanced representation across sites. The test set was fixed across all experiments to enable fair comparisons between different data scales. The model was trained using 10%, 25%, 50%, 75%, and 100% of the training set, while keeping validation and test sets unchanged. For all experiments, we used the baseline TimeSformer-HR pretrained on SSv2, with a linear frame-level classifier, and maintained consistent hyperparameters, data processing steps, and augmentation strategies. Model performance was evaluated using AUC on the test set. Supplementary information Supplementary Information Supplementary Information