Investigating surgeon performance metrics as key predictors of robotic herniorrhaphy outcomes using iterative machine learning models: retrospective study Authors: Thomas H Shin, Abeselom Fanta, Mallory Shields, Georges Kaoukabani, Fahri Gokcal, Xi Liu, Ali Tavakkoli, O Yusef Kudsi Published: 2025 ABSTRACT AbstractBackgroundRobotic data streams allow for capture of objective performance indicators, providing the ability to quantify and analyse operator technique and movement in optimizing postoperative outcomes. This study provided proof-of-concept demonstration of how intraoperative surgeon-factors could influence post-robotic herniorrhaphy complications via machine learning analyses of objective performance indicators.Study designData on robotic-assisted ventral hernia repair were retrospectively reviewed between February 2013 and November 2022 at a single academic centre. Machine learning modelling on systematic chart review data correlated perioperative patient factors, intraoperative objective performance indicators, and postoperative outcomes. Complications were classified with the Clavien–Dindo scale. Endpoints of interest included postoperative complications at discharge, at postoperative day 30, and at the last follow-up. Machine learning models employed included linear, k-nearest neighbours, support vector, decision tree, random forest, adaptive boosting, and extreme gradient boosting regression algorithms.ResultsSome 520 patients undergoing robotic ventral hernia were included. Median age of patients was 56 years with 52.7% male and median body mass index 31.9 kg/m2. 92.7% of patients had at least one medical comorbidity peoperatively. Complications occurred in 33 (6.3%) patients at time of discharge. Machine learning models demonstrated an accuracy 0.95, a precision 0.92, a recall 0.95, and a F1 0.92 of objective performance indicator predicting complications and an accuracy 0.95, a precision 0.95, a recall 0.95, and a F1 0.94 by Clavien–Dindo grade at time of discharge. Thematic analyses of top ranked factors included operator-specific objective performance indicators alongside patient factors canonically associated with hernia complications.ConclusionsThis study showed the novel application of machine learning modelling to bridge objective performance indicators and clinical patient factors to postoperative clinical outcomes, demonstrating the relevance of dynamic intraoperative surgeon factors on clinical outcomes. Introduction With the growing rise of artificial intelligence (AI) applications through all aspects of medicine, surgical data science is emerging as an area of great interest, particularly in the optimization of postoperative patient outcomes in relation to technical precision and data-driven decision-making. Continued developments of the robotic platform have been leveraged to quantify how operator movement and behaviour, known as objective performance indicators (OPIs), are connected to postoperative endpoints1–4. Unlike traditional analysis that relies on predefined models and known variables, AI-powered machine learning (ML) dynamically adapts to data patterns without predefined structures with a level of robustness and depth not previously achievable. The marriage between ML modelling and OPI-based outcomes research has been established in robotic surgery5–9. Although predictive ML models have shown success in some surgical domains, their broad applicability warrants further exploration. Previous ML modelling studies linked preoperative patient and hospital factors to post-herniorrhaphy complications such as mesh infection, pulmonary failure, and venous thromboembolism8,10–13. AI and ML modelling have also been applied to investigate some features on cross-sectional imaging, which predicted complexity of abdominal wall reconstruction and the likelihood of developing an incisional hernia after other surgical procedures14–17. Data used were mostly static and collected during the preoperative period rather than dynamic and intraoperative. Operator-dependent data likely have a significant impact on patient postoperative outcomes and procedural efficiency4,18–21. This study exploited robotic data streams collected from robotic-assisted ventral hernia repairs to determine how OPIs, patient and operator factors could predict short-term outcomes. Methods Patient selection Data on robotic assisted ventral hernia repair were retrospectively reviewed between February 2013 and November 2022 at a single academic centre. All operations were performed with the Da Vinci® robotic systems (Intuitive, Sunnyvale, CA). Complications and patient outcomes were graded based on the Clavien–Dindo classification system22. Analysis and modelling OPIs and perioperative factors were used to predict postoperative endpoints using supervised ML methodology. OPIs and events were calculated from the robotic data streams and all OPIs are outlined in Table S1. All OPIs collected were consistent throughout the study period, robotic platform, and operative technique. OPIs are represented as total event counts, total event durations, rates (event count/event duration), normalized durations (total event duration/operation duration), and/or medians. Each operation and all associated OPIs were retrospectively paired to preoperative, intraoperative, hernia characteristics, and surgeon/operator factors (Table S1). The model also captured the completion of concomitant procedures at time of hernia repair, such as concomitant adhesiolysis, closure of the hernia defect, and inguinal hernia repairs. The primary endpoint was prediction of postoperative complications by Clavien–Dindo grade at time of discharge, 30 postoperative days, and at last follow-up. Logistic regression, k-nearest neighbours, support vector, decision tree, random forest, adaptive boosting, and extreme gradient boosting (XGBoost) classifiers were used to predict Clavien–Dindo grade. Further details on the ML algorithmic approach are provided in Supplementary Methods S1. Model variants from each model type with the highest F1 score on test data sets were selected as a representative model for predicting a desired outcome, where F1 score refers to the fidelity score derived as the harmonic mean between precision and recall. For Clavien–Dindo models, metrics were weighted by the number of grades in the training data set. To demonstrate the robustness of the models, the test data sets were split into five groups where model metrics (accuracy, precision, recall, F1, and log loss) were computed. Shapely additive explanations (SHAP) values were used to compute feature importance for the final model. To qualify model fit, F1 scores were used for model selection. Exploratory endpoints also included length of operative hospital stay and Charleston Co-morbidity Index score at last follow-up. Oversight This investigation was conducted under Institutional Review Board approval at Good Samaritan Medical Center with an approved waiver of informed consent (IRB protocol HW11-22). Funding for this research was provided by an Intuitive Surgical Foundation Research Grant (Sunnyvale, CA; August 2022). Results Patient characteristics and clinical outcomes A total of 520 consecutive robotic-assisted abdominal wall hernia patients were included in the study. The median ASA score was 3, reflecting how 92.7% of patients (N = 482) had at least one medical co-morbidity. The Composite Hernia Risk Factor score rubric, a modified categorization of the Ventral Hernia Working Group hernia classification system that incorporates patient clinical co-morbidities, wound class, and hernia characteristics to risk-stratify hernias, was utilized to find that 418 patients had at least one risk factor2,3,23. A total of 491 patients were operated in an elective setting with an equal distribution of approaches (P > 0.05). Emergency patients were 5.6% (N = 29) and were all completed via an intraperitoneal onlay mesh (IPOM) approach. Whereas a majority of operations were incisional or ventral hernia repairs in isolation, the most frequently concomitant procedure was adhesiolysis in 17.7% (N = 92), followed by inguinal hernia repairs (10.8%, N = 56; Table 1). Surgical volume by operator was also tracked, allowing for the model to account for operator variability using operative volume as a proxy. Table 1Patient and perioperative characteristics Patients (n = 520) Preoperative patient characteristics Age (years), median (i.q.r.)56 (46–67) Sex Male274 (52.7) Female246 (47.3) BMI (kg/m2), median (i.q.r.)31.9 (28.1–36.1) BMI > 30335 (64.4) ASA score, median (i.q.r.)3 (2–3) Patient comorbidities Presence of at least one co-morbidity482 (92.7) Cardiovascular Coronary artery disease44 (8.5) Hypertension294 (56.5) Hyperlipidaemia206 (39.6) Prior myocardial infarction10 (1.9) Pulmonary Chronic obstructive pulmonary disease63 (12.1) Non-COPD228 (43.8) Endocrine pathologies Diabetes mellitus99 (19.0) Neuropsychiatric diagnoses132 (25.4) Vascular pathologies68 (13.1) Composite Hernia Risk Factor418 (80.4) Smoking history Never smoker288 (55.4) Previous smoker109 (21.0) Active smoker123 (23.7) Hernia characteristics Hernia type Ventral263 (50.6) Incisional250 (48.1) Ventral and incisional7 (1.3) Incarceration329 (63.3) Recurrent hernia130 (25) Intraoperative characteristics Defect closure479 (92.1) Console time (min), median (i.q.r.)68.5 (47–177) Skin-to-skin operative time (min), median (i.q.r.)88 (61–191) Emergent*29 (5.6) Surgical approach Extended totally extraperitoneal171 (32.9) Hybrid approach†27 (5.2) Estimated blood loss (ml), median (i.q.r.)5 (5–10) Concomitant procedures Adhesiolysis92 (17.7) Inguinal hernia repair Unilateral24 (4.6) Bilateral32 (6.2) Non-hernia concomitant procedure4 (0.8) Mesh position IPOM149 (28.7) TAPP117 (22.5) Rives–Stoppa127 (24.4) TAR127 (24.4)Values are n (%) unless otherwise stated. ASA, American Society of Anesthesiologists; BMI, body mass index; COPD, chronic obstructive pulmonary disease; IPOM, intraperitoneal onlay mesh; TAPP, transabdominal preperitoneal; TAR, transversus abdominis release. *All emergent hernia repairs were completed via IPOM repair. †Hybrid approach entails use of open approach for portions of the surgery in addition to laparoscopic approach (that is laparoscopic lysis with open repair, etc.). Patient and perioperative characteristics Values are n (%) unless otherwise stated. ASA, American Society of Anesthesiologists; BMI, body mass index; COPD, chronic obstructive pulmonary disease; IPOM, intraperitoneal onlay mesh; TAPP, transabdominal preperitoneal; TAR, transversus abdominis release. *All emergent hernia repairs were completed via IPOM repair. †Hybrid approach entails use of open approach for portions of the surgery in addition to laparoscopic approach (that is laparoscopic lysis with open repair, etc.). The median follow-up time was 64.8 days (i.q.r. 32.9–84.4 days, Table S2). Complication rates at each timepoint ranged between 6.3% and 23.3% (at discharge—n = 33 or 6.3%; at postoperative day 30—n = 82 or 15.8%; at last follow-up—n = 121 or 23.3%), with most scored as Clavien–Dindo Grade 1. Each complication was considered an independent event, resulting in some patients having multiple complications of variable severity grade (Table 2). Figure S1 demonstrates exploratory investigations into the predictive performance of the OPI ML models on length of hospital stay and a composite complication index score generated by the sum of Clavien–Dindo grades for all complications per patient. Table 2Postoperative endpoints At dischargePOD30Last follow-up Complications Yes33 (6.3)82 (15.8)121 (23.3) Clavien–Dindo Grade Grade 1195158 Grade 262025 Grade 3a166 Grade 3b2225 Grade 4a605 Grade 4b112 Secondary endpoints Length of stay (days), median (i.q.r.)0 (0–1) Time of last follow-up (months), median (i.q.r.)64.8 (32.9–84.4) Inflation-adjusted total treatment cost (USD), median (i.q.r.)6113 (4433–15 693) Overall comprehensive complication index score, median (i.q.r.)0 (0–8.7)Values are n or n (%) unless otherwise indicated. USD, United States Dollar; POD30: postoperative day 30. Postoperative endpoints Values are n or n (%) unless otherwise indicated. USD, United States Dollar; POD30: postoperative day 30. Machine learning model selection Multiple iterations over a range of parameters were used with linear and logistic regression, k-nearest neighbours, and support vector machine, revealing highest predictive metrics on patient outcomes were achieved by XGBoost model variants followed by random forest models (Fig. S2). Tree-based and ensemble modelling were also employed to avoid overfitting, which resulted in 5828 models for each patient outcome factor. Models with the highest F1 score for each outcome timepoint were selected. Model accuracy, precision, recall, and F1 scores for models querying complications are summarized in Fig. 1a. Similar models for each Clavien–Dindo grade are summarized in Fig. 1b. As a harmonic mean between accuracy and precision, F1 scores were used as a proxy for model fidelity. The final models demonstrated robust performance for complications (accuracy 0.95, precision 0.92, recall 0.95, F1 0.92) and Clavien–Dindo grade predictions (accuracy 0.95, precision 0.95, recall 0.95, F1 0.94) at the time of discharge with effect degradation with farther timepoints. The same number of model variants for Clavien–Dindo grades were used in regression modelling for secondary endpoints with model performance using adjusted R2 outlined in Fig. S3. Iteration parameters and number of model variants for each model are outlined in Table S3. Machine learning model fidelity and performance predicting (a) any complications and (b) complications by Clavien–Dindo grade at time of operative hospitalization discharge, postoperative day 30, and time of last follow-up Ranked predictive features for overall complications and Clavien–Dindo grades Variables used to create models were ranked in order of predictive value for both overall complication incidence and Clavien–Dindo grades. Consistent with existing literature, age and BMI were among the top predictive factors for overall complications at each timepoint analysed. Numerous OPIs were consistently among the top-ranked variables (Fig. 2). Other patient-specific factors, such as medical co-morbidities and medical therapy used (that is anticoagulants), are not conserved across timepoints. Top-ranking OPIs conserved across models and timepoints were camera control total duration, camera control normalized duration, left master clutch normalized duration, and head-in rate (Fig. 2a–d). OPIs conserved across models are those relevant to camera manipulation, operator non-dominant hand use, and operator frequency of head within console. Ranked feature identification of variables used in predictive model for any complications at (a) time of operative hospitalization discharge; (b) postoperative day 30; and (c) time of last follow-up Additional analyses were pursued to examine whether similar results were found within predictive modelling for complications by Clavien–Dindo grade (Fig. 3). Age and BMI continue to appear as top-ranked patient predictors and are found to be strongest for Grade 1 complications. There is a continued overlap of ranked OPI factors associated with complications stratified by Clavien–Dindo grade, including camera duration, head-in console time, and non-dominant master clutch duration. These conserved factors are most associated with lower-grade complications. When analysing by stratified Clavien–Dindo grade, factors associated with monopolar thermal energy use arise as top-ranked predictors for higher-grade complications. The proportion of Clavien–Dindo predictive OPI factors was much higher at the time of discharge compared to later follow-up timepoints, demonstrating the contextual relevance of OPIs as intraoperative factors that impact short-term outcomes rather than those longer term. Although the model accounts for operator surgical volume and trainee assistance, neither of these factors consistently arises as a top-ranked factor in any of the models of interest (Fig. 3a–d). Ranked feature identification of variables used in predictive model for complications by Clavien–Dindo grade at (a) time of discharge; (b) postoperative day 30; and (c) time of last follow-up Discussion Through the application of ML modelling on robotic-assisted ventral and incisional hernias, the relevance of both patient factors and intraoperative OPIs on outcomes have been demonstrated. The findings were concordant with existing literature supporting key patient factors as principal predictors of postoperative complications, such as advanced age and higher BMI. This study aligned with findings from other related studies in urology and other procedural fields that OPIs may influence postoperative outcomes with reported precision of up to 87.2%5–7,24. This study demonstrated that ML-based extreme gradient boosting and random forest models should identify relationships between OPIs and patient outcomes. This work could provide novel insight into the clinical applicability of OPIs and into the relationships between surgeon-specific intraoperative behaviour and postoperative complications. OPIs that could predict overall complications included those related to camera manipulation, frequency of non-dominant hand clutch control, and head-in time. This may reflect higher-complexity patients requiring more adjustments in visualization and retraction. Various widely accepted patient-specific risk factors, including BMI, age, medical co-morbidities, and history of recurrent hernias, were noted as top-ranked features in this ML model predicting complications. The proportion of OPIs within the highest-ranked factors was greatest at earlier timepoints and decreased subsequently; this made sense because factors specific to the technical execution probably had the greatest impact on immediate postoperative outcomes, whereas long-term complications were subject to the patient’s co-morbidities that were not addressed by the hernia repair itself. The study model’s performance also decreased with long-term outcomes, probably because OPIs had more influence on outcomes closer to the operative date rather than on the longer term outcomes. Models stratifying for Clavien–Dindo grade demonstrated that different thematic OPIs are more associated with specific complication severity. Clavien–Dindo 1 complications were most closely associated with the conserved OPIs from models predicting overall complications. Stratification according to Clavien–Dindo grade resulted in OPIs related to thermal energy use, resulting in higher ranking. The strength of association between OPIs and higher Clavien–Dindo grades appeared greater at later timepoints versus at discharge. Taken together, the findings suggested that specific thematic categories of OPIs could be identified as stronger predictors of not only overall complications, but also complication severity at different postoperative timepoints. As with other investigations examining the role of OPIs on postoperative outcomes, it is important to consider the nuanced implications of these findings. This analysis is based on a single surgeon’s experience. The OPIs revealed could not be generalizable. OPIs from other robotic hernia surgeons at this institution were not available to provide similar analysis. There are several limitations that must be considered in interpreting these results. The clinical relevance of top-ranked OPIs must be individually criticized. The OPIs identified were correlative factors with predictive value but not necessarily causative in nature. For example, this study revealed that increased intraoperative camera manipulation is predictive of higher complication rates, but it did not support the contrary. Rather, it suggested that frequent camera adjustments may serve as a proxy for surgical difficulty, which is indicative of complications. Post-hoc evaluation of intraoperative events, such as left-hand clutch use and camera manipulation, may affect perioperative decision-making. The OPIs identified were specific to the operator in this study, limiting the applicability of these findings. The duration of follow-up and retrospective nature of the data set inherently limited the scope of the study findings. This study uncovers operative or technical factors that may guide a surgeon’s risk assessment of a patient and subsequent postoperative management. Implications of this study could be extended into applications of OPIs in surgical training, where surgeons could iteratively adapt and optimize technique. Further exploration into this application and OPI intersection with other aspects of digital surgery (that is video segmentation, visual augmentation) is warranted and remains a future direction for this study. Although OPI-specific findings are unique to the individual surgeon, this study illustrated the relevance and importance of measures of intraoperative technique on patient outcomes vis-à-vis canonical patient factors. Further work is needed to crystallize the clinical relevance of OPIs and intraoperative objective metrics on other aspects of perioperative outcomes, including case complexity, technical skill acquisition, and operative efficiency. Supplementary Material Contributor Information Thomas H Shin, Division of General and Gastrointestinal Surgery, Brigham and Women’s Hospital, Boston, Massachusetts, USA; Division of General Surgery, Department of Surgery, University of Virginia Medical Center, Charlottesville, Virginia, USA. Abeselom Fanta, Intuitive Surgical, Sunnyvale, California, USA. Mallory Shields, Intuitive Surgical, Sunnyvale, California, USA. Georges Kaoukabani, Department of Surgery, Good Samaritan Medical Center, Tufts University, Brockton, Massachusetts, USA. Fahri Gokcal, Division of General and Gastrointestinal Surgery, Brigham and Women’s Hospital, Boston, Massachusetts, USA; Department of Surgery, Good Samaritan Medical Center, Tufts University, Brockton, Massachusetts, USA. Xi Liu, Intuitive Surgical, Sunnyvale, California, USA. Ali Tavakkoli, Division of General and Gastrointestinal Surgery, Brigham and Women’s Hospital, Boston, Massachusetts, USA. O Yusef Kudsi, Division of General and Gastrointestinal Surgery, Brigham and Women’s Hospital, Boston, Massachusetts, USA; Department of Surgery, Good Samaritan Medical Center, Tufts University, Brockton, Massachusetts, USA; Department of Surgery, Acibadem Mehmet Ali Aydinlar University, School of Medicine, Istanbul, Türkiye. Author contributions Thomas H. Shin (Formal analysis, Investigation, Methodology, Project administration, Validation, Writing—original draft, Writing—review & editing), Abeselom Fanta (Data curation, Formal analysis, Methodology, Validation, Writing—original draft), Mallory Shields (Conceptualization, Data curation, Formal analysis, Writing—original draft), Georges Kaoukabani (Data curation, Formal analysis, Investigation), Fahri Gokcal (Conceptualization, Data curation, Investigation, Writing—review & editing), Xi Liu (Conceptualization, Investigation, Project administration, Writing—original draft), Ali Tavakkoli (Investigation, Supervision, Writing—original draft, Writing—review & editing) and O. Yusef Kudsi (Conceptualization, Data curation, Investigation, Supervision, Writing—original draft, Writing—review & editing) Funding Financial support from Intuitive Surgical Foundation Research Grant (Sunnyvale, CA, USA; August 2022). Disclosures No financial disclosures to declare. Supplementary material Supplementary material is available at BJS Open online. Data availability Data availability not applicable. References Associated Data Supplementary Materials Data Availability Statement Data availability not applicable.