Development and application of an intelligent assessment system for medical clinical skill training Authors: Qi Dai, Kaiyu Chen, Cheng Han, Boli Zheng, Enguang Zou, Yisu Ge, Changfeng Cheng, Yuqian Gan, Jiatao Song, Huiling Chen, Zhaomin Chen, Xuebo Zhu, Gang Chen Published: 2026 ABSTRACT AbstractThe digital transformation of medical education represents a significant global development trend. However, clinical skill assessment, a vital method to inspect education outcome, has historically depended on conventional manual evaluation methods. To further explore intelligent training paradigms and realize autonomous instruction and automated assessment, this study introduces contrastive learning methods to construct a novel clinical skill assessment system characterized by unified standard, visible process, and objective evaluation. Practical validation indicates that the model effectively differentiates varying levels of proficiency in the specific surgical procedures. Under the optimal parameter combination, the prediction of one operational procedure achieved a peak accuracy of 94.01%. With a one-point tolerance, accuracy further improved to 96.41%. Overall, depending on its objectivity and standardization, the model significantly enhances the scientific rigor and efficiency of the clinical skill training process, while also providing immediate feedback on performance.Subject terms: Computational biology and bioinformatics, Engineering, Mathematics and computing Introduction The digital transformation of medical education is a significant global development trend. Numerous nations are formulating related policies and strategic plans to promote the deep integration of medical education with modern information technology, thereby enhancing the quality of medical talent cultivation. The Accreditation Council for Graduate Medical Education (ACGME) in the United States has consistently emphasized the use of information technology to optimize assessment and feedback mechanisms for residents (e.g., 360-degree assessment, the Milestone system), aiming to improve educational quality1. The International Association for Medical Education (AMEE) has a dedicated technology-enhanced learning committee to guide its decisions regarding various aspects of artificial intelligence (AI) in teaching. The focus of medical education is progressively shifting from traditional pedagogical approaches to digital learning methods2. The assessment of clinical skills has long depended on traditional manual evaluation methods, such as standardized patients (SP), direct observation of procedural skills, and the objective structured clinical examination. While these methods possess high face validity, their scoring process relies on the conventional model of “on-site expert observation + checklists,” which is time-consuming, susceptible to human bias, and often fails to accurately reflect medical students’ clinical skill proficiency and training outcomes. Existing research has introduced methods like video recording to enhance assessment accuracy and efficiency, yet the evaluation process remains costly and time-intensive. Furthermore, studies indicate that even with standardized scoring rubrics and examiner training, significant discrepancies persist in ratings for the same clinical skill among different examiners. The average inter-rater group variance exceeds 5%, representing a substantial gap from the high-consistency standards required for objective assessment3. The upfront investment is often disproportionate to the effectiveness, and the feedback provided is typically delayed and lacks specific detail4,5. Consequently, constructing a novel clinical skill assessment system characterized by “unified standards, visible process, and objective evaluation” has become a critical breakthrough point for medical education reform. Presently, the rapid advancement of computer science, particularly AI, is progressively turning autonomous instruction and automated assessment into realities. Increasing emphasis is being placed on the application and integration of innovative technologies, such as augmented reality (AR), in medical education, with their use cases expanding beyond skills training and assessment6. In the practice of clinical skills teaching, AI can be applied to process recognition, motion analysis, error detection, and score determination, thereby enabling real-time recognition and detailed modeling of operational behaviors7. AI-assisted instructional systems, including virtual patients, VR simulation training platforms, and intelligent tutoring systems, are being incrementally introduced into skill training. These tools help students develop operational thinking, reinforce standard procedures, and achieve personalized learning in a risk-free environment8,9. However, most of the existing research focuses on identifying single skill points or providing preliminary grading. These systems are not yet capable of comprehensively scoring the complete operational workflow and struggle to cover comprehensive dimensions such as procedural sequence, motion precision, operational logic, and humanistic care10,11. Basic surgical skills, like dressing changes and catheterization, are multi-step and highly standardized, making it difficult for current methods to meet the high-precision scoring requirements. Therefore, there is an urgent need to construct a novel assessment system that integrates AI with standardized principles. Such a system must overcome the bottlenecks of traditional assessment—namely, subjectivity, lack of feedback, and data gaps—to realize a full-process, multi-dimensional, and quantifiable intelligent evaluation of basic operational skills. This study targets the wound dressing change procedure—a core skill in basic clinical practice, licensing examinations, and standardized residency training—as its entry point. It proposes and validates an interpretable and scalable unified framework combining contrastive learning and support set retrieval for clinical skill assessment, which is also adaptive to new scenarios by simply updating the support set. As one of China’s innovative intelligent assessment systems for basic clinical skills, this project holds threefold groundbreaking value: It establishes a nationally leading clinical procedure video dataset, filling a gap in standardized teaching resource libraries. It establishes a nationally leading clinical procedure video dataset, filling a gap in standardized teaching resource libraries. The innovatively developed automatic scoring system achieves an accuracy of 94.01% in assessing operational details. The innovatively developed automatic scoring system achieves an accuracy of 94.01% in assessing operational details. Through the deployment of a cloud-based assessment engine, it achieves, for the first time, cross-regional and cross-institutional homogenized management of clinical skills assessment on a unified platform. Through the deployment of a cloud-based assessment engine, it achieves, for the first time, cross-regional and cross-institutional homogenized management of clinical skills assessment on a unified platform. This system serves three purposes: first, it provides an objective evaluation tool for clinical skills assessment; second, it offers a new autonomous training model of intelligent instruction for clinical skills training; and third, it leverages AI to drive a paradigm shift in clinical education from experience-driven to data-driven. Results The data used in this study were sourced from video recordings of residents performing a complete wound dressing procedure in a simulated environment at The First Affiliated Hospital of Wenzhou Medical University. The experimental videos were captured from three distinct perspectives: a frontal view, a trolley-side view, and a non-trolley-side view. All footage was recorded using Logitech cameras with a minimum resolution of 720p HD. The frontal camera was positioned 2 m away from the operator at a height of 1.6 m, while the trolley-side and non-trolley-side cameras were each placed 1 m away at an approximate height of 2 m. In cases where a specific camera angle was missing, such as when one of the three views was not recorded, the corresponding data segment was discarded entirely. Moreover, any recordings exhibiting significant camera shake or obstructions during the operation were also excluded. All raw videos underwent a preliminary screening process to ensure completeness and visual clarity. To enhance data diversity, multiple valid operation videos from the same operator that met the established scoring criteria were included in the final dataset. Each video was meticulously scored by trained senior surgical experts according to the standardized objective structured assessment of technical skills (OSATS) scale. The scoring criteria for this OSATS scale are detailed in Supplementary Table 1. The scale disaggregates the dressing change procedure into key stages and core skill points, with each scoring element possessing explicit criteria and corresponding point values. Specific scoring categories included operational judgment, pre-operative preparation, operational process, post-operative management, overall evaluation. The experts assigned scores based on the operator’s performance in each segment, culminating in an overall skill score for the procedure. These expert ratings constitute the ground truth labels for model training and evaluation in this study. In addition, on this basis, this study established a surgical video resource website (http://tool.valiant-hcc.com/index.html) to demonstrate the results and realize resource sharing and information exchange. The video data utilized for the experiments focused on the content within the operational process category. Given that critical skill information is predominantly reflected in the surgeon’s fine hand movements and the specific conditions of the surgical area captured from the frontal perspective, subsequent video analysis experiments focused primarily on the frontal view footage. This research primarily focuses on four types of surgical operational procedure videos, designated as operational procedure 3.1, operational procedure 3.2, operational procedure 5.2, and operational procedure 6. In total, the dataset comprises 2686 training videos, 107 support set videos, and 687 validation videos. The data composition for each procedure is as follows: operational procedure 3.1 includes 870 total videos, consisting of 670 training, 28 support, and 172 validation videos; operational procedure 3.2 includes 870 total videos, with 674 training, 24 support, and 172 validation videos; operational procedure 5.2 includes 870 total videos, comprising 676 training, 22 support, and 172 validation videos; and operational procedure 6 includes 867 total videos, made up of 666 training, 33 support, and 171 validation videos. In addition, on this basis, this study established a surgical video resource website (http://tool.valiant-hcc.com/index.html) to realize resource sharing and information exchange. Hyperparameter comparison experiments The selection of critical hyperparameters significantly influences the performance of the contrastive learning model. These experiments utilized the six-action videos corresponding to the operational process as the benchmark test. Adjustments were made to two key hyperparameters: the backbone network freezing strategy and the total loss function weight λ. Adjusting the freezing strategy balances the utilization of pre-trained knowledge with the learning of task-specific features from the dressing change videos. The weighting coefficient λ controls the contribution of the cross-set contrastive loss Lloss relative to the combined intra-class contrastive loss. Supplementary Table 2 lists the model parameter names corresponding to the hyperparameter strategies. The freeze strategy F1 denotes fine-tuning only the final bottleneck block of layer 4 in the backbone. The freeze strategy F2 denotes fine-tuning all layer 4; and F3 denotes fine-tuning both layer 3 and layer 4. Supplementary Table 3 presents the accuracy (AUC) and macro accuracy (MacroAUC) for the corresponding models. As shown in the table, θ2,2 achieved a strict accuracy of 79.39%, the highest among all combinations. In assessments with a higher tolerance margin, its accuracy (≤1 tolerance margin) was 87.27%, indicating a very high proportion of predictions falling within 1 point of the expert scores, second only to the 88.48% achieved by θ1,2. For the ≤2 tolerance margin, θ1,3 reached 95.15%, while other models also exceeded 91% accuracy, demonstrating that the prediction performance within a 2-point tolerance is substantial. Further analysis of performance under specific conditions reveals that when fixing the loss weight λ at 0.5 and comparing θ2,1, θ2,2, and θ2,3, θ2,2 achieved higher accuracy than the other two within the 1-point tolerance margin. This indicates that the optimal number of fine-tuned parameters is neither maximized nor minimized. The F2 strategy provided a more balanced and holistically superior performance. Likewise, fixing the freeze strategy at F2 and comparing different λ values (θ1,2, θ2,2, and θ3,2), θ2,2 was significantly superior to the other two models at strict accuracy. Although θ2,2 did not achieve the absolute optimum on every single metric, it demonstrated robust and well-balanced performance across all evaluation dimensions. Notably, given the inherent subjectivity in the expert scoring of these procedural videos, the θ2,2 model, which excelled within the ≤1 tolerance margin, was selected as a robust and efficient model for the subsequent skill assessment tasks. Model performance on different actions The support set is a standard term in the few-shot learning and meta-learning literature, denoting a small set of labeled reference examples used during inference. Unlike a validation set, which serves hyperparameter tuning, the support set directly participates in the inference pipeline. The query videos are scored by computing feature similarity against the support reference samples. This paradigm was first formalized by Vinyals et al.12 in matching networks and has since been widely adopted in prototypical networks13, meta- learning14, relation network15, and subsequent works. In our framework, the support set comprises expert-reviewed representative procedure videos at each score level, serving as the reference library for the retrieval-based scoring system at inference time. In our specific implementation, we selected samples from operational procedure 3.1, operational procedure 3.2, operational procedure 5.2, and operational procedure 6. Due to the sparsity of videos for certain rating levels, mandating a fixed-size support set could lead to an insufficient number of videos or the inclusion of low-quality, non-representative samples. To address this data sparsity, this study dynamically adjusts the support set size based on the following criteria: If the video count for a label is ≤5, 1 video is selected for the support set. If the video count for a label is ≤5, 1 video is selected for the support set. If the count is > 5 and ≤10, 2 videos are selected. If the count is > 5 and ≤10, 2 videos are selected. If the count is > 10 and ≤20, 3 videos are selected. If the count is > 10 and ≤20, 3 videos are selected. If the count is > 20, 5 videos are selected. If the count is > 20, 5 videos are selected. The specific video counts per label and the resulting support set sizes for each action are detailed in Fig. 1. Data sample distribution. a Distribution of the number of samples in the training, test, and support sets across different operations. b Distribution of the number of videos for each operation within the training, test, and support sets. Supplementary Table 4 presents prediction accuracy and MacroAUC for the four actions. As shown, all actions except operational procedure 6 achieved an accuracy of over 90% within a 1-point tolerance margin. Figure 2 displays the per-class AUC values, as well as the MacroAUC and Micro-AUC, for the four actions on the test set. It should be noted that some rating levels are absent from the figures because they were too sparse to be present in the test set. From Fig. 2, it is evident that the model generally achieves satisfactory predictive differentiation for the various rating levels across operational procedures 3.1, 3.2, 5.2, and 6. However, Fig. 2c indicates that the model performed poorly in distinguishing videos with a score of 12 for operational procedure 5.2, registering an AUC of only 0.4437. AUC values for each action grade, MacroAUC, and MicroAUC on the test set under the four action prediction scenarios. a AUC values for operational procedure 3.1. b AUC values for operational procedure 3.2. c AUC values for operational procedure 5.2. d AUC values for operational procedure 6. Figure 3 depicts the changes in loss and accuracy over epochs during training for the five actions. The accuracy for “operational procedure 3.1,” “3.2,” and “5.2” rose rapidly in the early stages, quickly reaching and stabilizing near 1.0. In contrast, the accuracy for “operational procedure 6” increased more slowly and continued to show improvement on the validation set even late in the training process. Although “operational procedure 6” exhibited greater training volatility, its loss eventually stabilized around 0.6 from a higher starting point, indicating a successful training process. The training for “operational procedure 5.2” was exceptionally stable. Changes in loss and accuracy over epochs during the training process for the four action prediction scenarios. The curves for different actions are distinguished by line styles and colors as indicated in the figure legend. Based on this experimental analysis, the predictive performance for most action-rating levels is acceptable, with the notable exception of the poor differentiation for “operational procedure 5.2” at score 12. This poor performance may be attributed to a lack of representativeness in the selected support set for that specific class. It is important to note that the data in this study is multi-center, involving different video capture methods. Some videos may not clearly contain the specific action. Furthermore, as this study utilized only a single-view perspective, the trolley-side view might have been occluded in some instances, leading to predictive inaccuracies. Future work may require model improvements focused on support set selection strategies and the application of multi-view analysis. Comparison experiments of different backbones To evaluate the impact of different backbones on overall model performance, this experiment conducted a comparison with three other representative video classification models: multiscale vision transformers (MVIT)16, temporal relation network (TRN)17, and temporal segment network (TSN)18. Additionally, a temporal shift module (TSM) model pre-trained on Kinetics-400 was included. Specific model information is visible in Supplementary Table 4. This experiment used the operational procedure 6 dressing change videos as the benchmark test for prediction. It should be noted that the backbones for TRN, TSN, and TSM are all based on ResNet-50. Their output feature dimension is (2048, 7, 7). In this study, we applied average pooling to compress the last two dimensions, resulting in a 1D feature vector of 2048. For MVIT, the feature output is the embedding of the class token, resulting in a 1D vector of length 768. As shown in Supplementary Table 5, TSM (sthv2) and TSM (kinetics) demonstrated the best or second-best performance. TSM (sthv2) achieved the highest strict accuracy at 79.39% and, along with TSM (kinetics), reached the peak accuracy of 93.33% within a 2-point tolerance. In terms of MacroAUC, which measures overall classification performance, TSM (kinetics) achieved 0.9289, indicating the strongest ability to discriminate between classes. These results suggest that TSM’s efficient temporal shift module effectively captures critical temporal information. To assess statistical reliability, we compute 95% confidence intervals (CI) for all performance metrics using bootstrap resampling (N=1000). For each iteration, we draw a sample of the same size as the original test set from the predictions, compute all metrics, and report the mean and 95% CI (2.5th and 97.5th percentiles) across all iterations. Detailed information can be found in Supplementary Tables 6, 7, and 8. To further illustrate the superiority of our contrastive learning approach, Supplementary Table 9 also presents the prediction results derived directly from the classification and regression models. The hyperparameters for TSM(sthv2) models strictly adhere to the configurations specified in their respective original papers, utilizing the support set as validation data to identify the optimal weights. For all four actions, we trained both a direct classification baseline and a regression model built upon the TSM (SthV2) architecture, subsequently averaging the video accuracy across the four action categories. Specifically, the models were trained on the training set, utilizing the support set as validation data to determine the optimal model weights. Final evaluations were conducted on the test set, with detailed results provided in Supplementary Table 9. For the regression model, the predicted scores were rounded to the nearest integer before being compared against the ground truth video labels to determine prediction correctness. Compared to the identical backbone architecture, our proposed contrastive learning approach outperforms both the classification and regression baselines in terms of precision, yielding an average accuracy improvement of 10.7% (87.09% vs 76.39% vs 0.76%). In contrast, the TRN model lagged significantly across all metrics, with a MacroAUC of only 0.6822, making it unable to accurately classify the data used in this study. Furthermore, the results for MVIT show that while its strict accuracy was relatively low at 69.70%, its MacroAUC (0.9242) was superior even to the TSM (sthv2). This may imply that while MVIT has biases in determining the single most likely class, its overall grasp of inter-class differentiation remains strong. This could be related to the characteristics of vision transformer architectures requiring massive-scale pre-training. Figure 4 illustrates the AUC results for each model, and Fig. 5 shows the loss and accuracy during training. Apart from TRN, all models could perfectly identify with the highly distinct Class 5. However, Class 13 and Class 14 were problematic for all models, consistently showing low AUC scores. For example, all models scored below 0.9 AUC in Class 14, and both MVIT and TSM (sthv2) also performed poorly in Class 13. This consistency across models suggests that these two classes possess inherently ambiguous visual features, perhaps with high overlap with other classes or large intra-class variance. This indicates a fundamental challenge within the dataset itself, rather than a limitation of the models. AUC values for each action grade, MacroAUC, and MicroAUC on the test set, predicted by different backbone models. a AUC values predicted by the MVIT backbone models. b AUC values predicted by the TRN backbone models. c AUC values predicted by the TSN backbone models. d AUC values predicted by the TSM (kinetics) backbone models. e AUC values predicted by the TSM (sthv2) backbone models. Changes in loss and accuracy over epochs during the training process for the five models. The curves for different models are distinguished by line styles and colors as indicated in the figure legend. Visualization of retrieval results To gain a more intuitive understanding of the model’s retrieval behavior and to identify its strengths and potential areas for improvement, this section presents a visualization and analysis of selected retrieval results from the test set. Specifically, the experiment used operational procedure 6 as the benchmark task. Videos from each rating level were randomly selected from the test set, and for each test video, the model identified the most similar video from the support set. The retrieved video pairs are shown in Figs. 6 and 7, with 6 uniformly sampled frames displayed for each video. Figure 6c of each figure displays the cosine similarity scores between the selected test video and the support set videos. Since there could be more than one support set video per score, we display the highest similarity score for each score level. Furthermore, as all videos with scores of 5 and 8 were classified correctly, no examples for these scores are presented in Fig. 7. Visualization comparison of a correctly classified test video frame and its most relevant corresponding frame from the support set. a The test video frame. b The most similar matching video frame from the support set. c Cosine similarity between the test frame and the most similar support frame for each score category. The y-axis represents the corresponding video score. Visualization results for an incorrectly classified test video frame and its most relevant corresponding frame from the support set. a The test video frame. b The most similar matching video frame from the support set, whose score differs from the ground truth. c Cosine similarity between the test frame and the most similar support frame for each score category; the highest similarity corresponds to the predicted score, which diverges from the ground truth. The y-axis represents the corresponding video score. The correctly matched results demonstrate the robustness of the proposed model. For instance, in the matching cases with ground truth scores of 11 and 12 in Fig. 7, the operators in the test and support videos wear different colored attire. There are also noticeable differences in patient mankind, the operating room background, and the video’s lighting and camera angles. Despite these variations in scene and target appearance, the model accurately identified the corresponding skill level and made a correct match. This strongly suggests that the model can effectively disregard irrelevant variables and focus on the key features of the surgical procedure itself, demonstrating good generalization and robustness. The incorrect matching results in Fig. 8 show that the model demonstrates a pronounced overestimation tendency. Specifically, samples with ground truth scores of 10, 11, and 12 were misclassified as 14 or 15, resulting in a score discrepancy of 3 to 4 points. Flowchart of the contrastive learning method for action recognition in surgical dressing change videos. The flowchart shows the overall workflow. Surgical videos are divided into training, support, and test sets. Each set undergoes preprocessing and feature extraction. Training set features optimize the model via contrastive learning; support set features are stored as a reference library. Test set features are compared against the library to predict skill scores based on similarity. While this gap appears numerically large, an analysis of the similarity distribution reveals that the similarity margin between the true class and the predicted class is often minimal. For example, in the sample with a true score of 12, the similarity to the true class was 0.842, while the similarity to the predicted class was 0.856. This negligible difference of 0.014 resulted in a 3-point score deviation. Similarly, for the sample with a true score of 10, the similarity margin was only 0.077, yet it was ultimately assigned to a class 4 points higher. This phenomenon suggests that the model effectively recognizes the proximity between different skill levels in the similarity space. However, because the final output is a discretized single score, these minimal similarity differences are amplified into large score errors. Conversely, there were also minor misclassifications, such as underestimation (a true score of 15 predicted as 14) or slight overestimation (a true score of 14 predicted as 15). These 1-point errors can be reasonably attributed to the subjectivity and ambiguity inherent in the manual scoring process. However, more concerning than these minor deviations is the model’s systematic bias towards higher scores across multiple examples. This phenomenon suggests that during feature learning, the model forms tighter clusters for high-score samples but fails to adequately capture the non-standard (or highly variable) features of low-score samples. This results in a tendency to align with high-score classes during similarity ranking. Furthermore, in the sample with a true score of 11, a tie in the highest similarity occurred (0.795 for both Class 13 and Class 14). It was ultimately classified as 14 due to the decision rule, further exacerbating the overestimation trend. In summary, while the model can generally distinguish between different skill levels, it suffers from a systematic overestimation problem at the fine-grained level. The root cause appears to be the high inter-class similarity among high-score categories, combined with a discretization-based decision mechanism that amplifies minimal similarity differences, especially for boundary samples. Therefore, future research direction should consider introducing a regression-based prediction model. This would better align with the continuous nature of surgical skill assessment and avoid the numerical deviations caused by discrete classification. Discussion This study establishes an intelligent automated scoring system for fundamental clinical skills, leveraging the advantages of AI to achieve objective, accurate, and efficient assessment of medical students’ clinical abilities with timely feedback, thereby creating a closed-loop system. First, the system systematically validates the utility of an intelligent scoring system based on support set retrieval and contrastive learning in the context of dressing change procedures, which is significant for ensuring the quality of clinical skills training. Second, the system’s computations rely on extensive clinical skill operation data and standardized assessment metrics. Through quantitative analysis, it effectively controls inter-rater variability, thereby enhancing inter-rater reliability and consequently improving the scientific validity and reliability of the assessment results19. Furthermore, the system can record and analyze students’ operational processes in real-time, promptly identifying problems and deficiencies in their clinical skill execution. It provides specific feedback and improvement suggestions, helping students engage in targeted learning and training and promoting the continuous enhancement of their clinical skills20. Moreover, the automated scoring system enables large-scale, high-frequency skill examinations, meeting the needs of medical institutions at various levels. It also provides medical educators with rich teaching data, helping them understand student learning situations and skill mastery levels, thereby optimizing teaching content and methods and improving the quality and effectiveness of clinical skills training. Basic operational skills are a fundamental requirement across all clinical disciplines. Standardized execution is directly linked to nosocomial infection control, patient safety, and prognosis. For example, inadequate attention to aseptic technique details during a dressing change can lead to severe infections, potentially jeopardizing patient recovery21. However, current residency training assessments still widely rely on examiner experience, suffering from significant subjectivity and distinct inter-group variability22,23. The intelligent scoring system proposed in this study, through data-driven quantitative scoring, allows skill training to overcome the limitation of “reliance on senior faculty” and achieve homogenization across institutions. Instantly, fine-grained feedback helps young doctors quickly identify operational weaknesses, thereby shortening their learning curve and reducing error rates. In the long term, the adoption of such a tool is expected to improve training equity and ultimately translate into improved patient outcomes. Compared to related research in recent years, existing AI-based skill assessments predominantly focus on binary classification or coarse-grained action recognition, often lacking in multi-level scoring, detailed granularity, and cross-center consistency. Hira et al. used attention networks for binary grading of surgical skill videos but failed to cover skill-level differentiation24. Wan et al. also used spatio-temporal attention mechanisms to distinguish experts from novices, remaining within a binary classification framework25. In JAMA Network Open, Kitaguchi et al. developed a 3D convolutional network for skill assessment in laparoscopic operations, but the results primarily served to differentiate operational levels rather than providing detailed feedback26. Igaki et al. also reported a scoring system based on intraoperative videos; although it verified the objectivity of AI scoring, it lacked pedagogical explainability27. Furthermore, systematic reviews and multi-institutional studies indicate that most current video AI systems suffer from insufficient external validity, a lack of standardization, and limited interpretability28–30. In contrast, this study explicitly embeds expert scoring via support set retrieval, enabling the AI output to be not just a “score” but also traceable back to reference samples. This enhances generalization capabilities in few-shot, multi-level scenarios and provides explainable feedback highly consistent with expert evaluations. The support set retrieval and contrastive learning loss function design adopted in this study enables the model to maintain stable performance on multi-center, heterogeneous video datasets. Specifically, the intra-loss constrains feature similarity among samples within the same class, while the cross-loss establishes consistency between the training set and the support set, thereby enhancing the discriminative power of the feature representations for the multi-level scoring task. Unlike traditional direct classification methods, the scoring results of this study can be traced back by referencing support set samples, ensuring the transparency and interpretability of the prediction process. The experimental results demonstrate that the framework achieved a peak accuracy of 94.01% in scoring specific actions. When the tolerance was relaxed to ≤1 (a one-point margin), the accuracy improved to 96.41%. This indicates that the framework can achieve highly consistent effects with expert scoring at both the action level and the overall assessment level, providing a reliable basis for establishing an objective and generalizable medical skill assessment system. Despite the rapid development of automated scoring systems, their practical application in clinical training still faces several challenges: First, data dependency: A lack of high-quality labeled data impacts model training, and annotation costs are high31,32. Second, clinical applicability: Most systems are still confined to simulated environments, with limited application in real clinical scenarios. Third, lack of standardization: Scoring criteria have not been unified across different systems, making horizontal comparisons difficult. Fourth, scoring comprehensiveness needs improvement: Most research focuses on identifying single skill points or preliminary grading. These systems are not yet capable of comprehensively scoring the entire operational process, failing to cover integrated dimensions such as procedural sequence, motion finesse, operational logic, and humanistic care33,34. Overall, this framework enhances the scientific rigor and generalizability of clinical skills training through its objective and standardized scoring mechanism, offering an important reference for constructing a unified medical education assessment system. In the future, this research will focus on the following aspects: First, expanding the dataset by incorporating more clinical operation items and real-world clinical videos from multiple centers to enhance the model’s generalization capabilities. Additionally, more rigorous evaluation strategies, such as leave-one-center-out testing or independent validation on unseen centers, will be adopted to determine whether the model captures truly generalizable skill-related features. Furthermore, the research will explore domain generalization or adaptation strategies to improve robustness under distribution shift. Second, exploring multi-view information fusion techniques by integrating video streams from different camera angles to build a more comprehensive understanding of the operational scene and overcome single-view occlusion issues. Third, investigating the transition from the current discrete retrieval framework to a continuous regression prediction model to directly output a continuous skill score, making the evaluation more aligned with the nature of clinical skill assessment and potentially resolving scoring biases introduced by discrete decision-making. Fourth, deploying this model in clinical training by setting up appropriate training venues, collecting and uploading data from multiple cameras, providing real-time analysis and feedback, and truly achieving autonomous instruction and autonomous intelligent examination. Methods Dataset construction and scoring system establishment In this study, the entire process of trainees performing procedures was recorded in a standardized clinical skills training room using multi-angle, fixed-position cameras (front, left, and right-side multi-perspectives) and an ambient sound acquisition system. This setup ensured complete coverage of all operational steps and potential common errors. The subjects included interns and resident physicians from different grades. Before filming, informed consent forms were distributed, and procedural and operational instructions were provided to ensure the performance was as close as possible to an actual training scenario. After video acquisition, professional video processing tools were used for frame splitting and compression. Subsequently, two or more experts with clinical teaching experience annotated each video segment. The annotations included: sequential marking of operational steps, localization of key time nodes, identification of common erroneous behaviors, assignment of scoring labels, and subjective evaluation of the operation’s overall completeness and standardization. Upon completion, another expert reviewed all annotations, performing corrections and score merging to generate a video dataset with multi-dimensional scoring labels. On this basis, referring to the Supplementary Table 1 and integrating expert consensus, a scientific, clear, and quantifiable scoring metric system was developed. This system defined the scoring dimensions and key points for each operational item. Subjective scoring dimensions, such as demonstration of humanistic care, were supplemented through expert discussions. All scoring items were ultimately converted into standardized scoring labels for model training input. To ensure the universality and computability of the scoring system, all scoring items were index-coded in a pre-operative definition checklist and managed in a structured tabular format, facilitating subsequent automatic matching and result generation within the platform system. The analysis in this study primarily focused on single, frontal-view videos. During the data preprocessing stage, the originally captured compressed-format videos were first decoded into processable image frames. Subsequently, bilinear interpolation was used to uniformly resize each frame to 1920×1080 pixels. Following this, 16 frames were randomly sampled from each complete video to serve as the input sequence for the model. To enhance the model’s generalization capabilities and reduce the risk of overfitting, a data augmentation strategy was adopted. Specifically, image frames in the training set were subjected to multi-crop and random horizontal flipping with a probability of 0.5. In contrast, frames in the support and test sets underwent standard center cropping, resulting in final image dimensions of 224×224 pixels. Model architecture This study reframes the scoring problem as a video retrieval task. The core idea is to utilize contrastive learning to retrieve the most similar reference video from a pre-constructed support set corresponding to the video being evaluated. The expert score of this reference video is then assigned to the video under evaluation, thereby achieving an automated score. For instance, assume the support set contains Video A, expertly rated at 9 points, and Video B, rated at 5 points. When a new surgical video, X, is input to the model, the model, through contrastive learning, determines that Video X is most similar to Video A in terms of key operational features. Consequently, the automated score for Video X will tend toward 9 points. This research introduces a supervised contrastive learning framework to optimize the backbone model’s comprehension capabilities for video. First, representative videos from each scoring level are selected to form the support set. Subsequently, the backbone network is used to extract video features. The model is continuously trained by pulling the same-class videos closer together and pushing different-class videos further apart in the feature space. Finally, based on the learned model and the support set samples, skill score predictions are made for new videos under evaluation. Specifically, the primary workflow of the contrastive learning-based clinical surgical video scoring method proposed in this study is illustrated in Fig. 8. The entire process begins with the acquisition and preparation of surgical video data, which includes collecting video materials of surgical procedures and their corresponding skill scoring labels provided by experts. After data preparation, the dataset is meticulously partitioned into three independent subsets: a training set, a support set, and a test set. The support set contains representative video samples for each skill score level, serving as a benchmark for subsequent contrastive learning and score prediction. Next, the video data in the training, support, and test sets all undergo a unified preprocessing procedure. This aims to augment the raw video data and convert it into a format suitable for model input. Following preprocessing, the same feature extraction network is employed to extract high-dimensional feature representations from the videos in the training and support sets, respectively. During the model training phase, the spatiotemporal features from the training and support sets are primarily used to optimize the backbone model parameters. Based on the contrastive learning framework, features from the training set are compared against those from the support set. A defined loss function is used to minimize the distance between same-class features while maximizing the distance between different-class features. This training process is conducted iteratively until a preset number of epochs is reached. Upon completion of training, the trained model is used to re-extract and store the features for all video samples within the support set, thereby constructing a support set feature library. This library will be used for subsequent skill score predictions. For any video in the test set, or any new video to be evaluated, it undergoes the same preprocessing and feature extraction pipeline. The resulting spatiotemporal features are then compared against all sample features in the support set feature library to calculate similarity. Finally, the skill score level for the query video is predicted based on the highest similarity found between its spatiotemporal features and the spatiotemporal features of the support set samples. After prediction, the model’s results are compared with the ground-truth expert scoring labels, and the model’s performance is comprehensively assessed using a series of evaluation metrics. Contrastive learning module The structure of the contrastive learning model proposed in this study is shown in Fig. 9a. Common video processing model backbones, such as MVIT16, TRN17, and TSN18, are employed as video feature extractors. To enhance the model’s video understanding capabilities, the backbone parameters θ are initialized using weights pre-trained on the large-scale, general-purpose video action recognition dataset, Something-Something V235. Architecture diagram of the contrastive learning-based method for action recognition in surgical dressing change videos. a Illustrates the overall workflow of the contrastive learning framework. After data augmentation, both the training set and the support set are fed into the backbone network for feature extraction. The resulting features are then passed through an average pooling layer to generate the final feature representations. Model training is achieved by calculating the intra-set loss within each set and the inter-set loss between the sets. b The TSM module helps capture temporal features at each layer by shifting channels between adjacent video frames. c Both intra-set and inter-set losses are derived by computing the similarity between each sample and all other samples. The objective of this process is to increase the similarity among samples of the same class while decreasing the similarity among samples of different classes. d Similarity is measured using the cosine similarity metric. During the prediction phase, a query sample is assigned to the label of the support sample with which it shares the highest cosine similarity score. The preprocessed video tensors Tinput are fed into the selected backbone network M(⋅;θ). The network performs forward propagation, outputting feature vectors Foutput=MTinput;θ that represent the high-level semantic information of the input video clips. The raw features Foutput output by the feature extraction network require further feature mapping for the contrastive loss calculation. Global adaptive average pooling Poolavg⋅ is applied to compress the spatial dimensions of the feature maps, yielding a fixed-length feature vector f=PoolavgFoutput that retains information from all channels. Finally, the total loss is computed from the resulting features by calculating the intra-loss and cross-loss. Temporal shift module To efficiently capture temporal dependencies between video frames without significantly increasing computational cost, this study integrates a TSM into the feature extraction network, with its specific structure illustrated in Fig. 9b. Traditional 2D convolutional neural network (CNN)-based video analysis methods typically process each frame independently, overlooking temporal information interaction. Conversely, while 3D CNNs can extract spatio-temporal features simultaneously, they are computationally intensive and demand significant hardware resources. The TSM addresses this issue through a novel and efficient approach. The core idea of this module is to shift a portion of the feature map channels along the temporal dimension before performing the 2D convolution operation. Specifically, for a given feature map tensor, some channels are shifted one temporal step backward, another portion is shifted one temporal step forward, while the remaining channels are left unchanged. In this way, when the subsequent 2D convolutional kernel operates on the feature map of the current frame, its receptive field effectively incorporates information from adjacent frames. When processing the features of the tth frame, the convolution operation can simultaneously see partial features from the (t-1)th and (t+1)th frames due to the channel shift. This parameter-free temporal information fusion operation enables standard 2D CNNs to effectively model temporal information with zero additional computational overhead. This module enhances the model’s ability to understand dynamic changes in video at a very low cost, which is crucial for identifying and evaluating clinical operational steps that possess continuity and a specific sequence. Loss function design The core of this study lies in utilizing contrastive loss functions to guide feature learning, as specifically illustrated in Fig. 9c. Model training is conducted by combining two types of contrastive losses: intra-class contrastive loss and cross-set contrastive loss. The intra-class loss is calculated independently for the training set batch and the support set batch, respectively. The objective of designing the loss function is to pull features of the same-class samples closer together in the feature space while pushing features of different-class samples further apart. This study employs cosine similarity to measure the distance between features. For a feature vector f, it is first L2-normalized according to Eq. (1). The intra-loss is designed to enhance the intra-class compactness and inter-class separability of the features. For a batch of N normalized features {fi}i=1N sampled from the training set, along with their corresponding skill rating labels {yi}i=1N, the raw similarity score matrix L is computed via Eq. (2). where τ is the temperature hyperparameter, set to 0.07 in this study. The valid exponential score matrix Eij=expLij×1−δij is computed. The δij is the Kronecker delta function, where self-pairs (i=j) are excluded from the denominator. Next, a positive mask P is constructed such that Pij=1 when samples i and j share the same label, and Pij=0 otherwise. The intra-class contrastive loss for each sample lintra,i is calculated as shown in Eq. (3). The final intra-class loss for the batch Lintra is the average loss over all N samples, as shown in Eq. (4). The cross loss utilizes the support set as a reference to guide the feature learning of the training set samples. Let the training set batch contain Ntrain normalized features {ftrain,i}i=1Ntrain and labels {ytrain,i}i=1Ntrain, and the support set contain Nsupp normalized features {fsupp,j}j=1Nsupp and labels {ysupp,j}j=1Nsupp. The raw similarity score matrix Lcross between the training set features and all support set features is computed using Eq. (5). Analogously, a cross-set positive mask Mposcross is defined, where entries are 1 when the training sample and support sample share the same label. The cross-set contrastive loss for each training sample lcross,i, is calculated as shown in Eq. (6): The final cross-set loss Lcross is the average loss over all N training samples, as detailed in Eq. (7). The multiple loss terms calculated above are combined via a weighted sum to form the final total loss function Ltotal, as shown in Eq. (8). where λ, α, and β are preset weighting coefficients used to adjust the contribution of each loss type to the overall optimization. Model inference After the model training is complete, for each sample in the test set requiring prediction, its cosine similarity with every sample in the support set is calculated. The sample is then assigned to the label of the support set sample with the highest cosine similarity, as illustrated in Fig. 9d. Specifically, each video in the support set saves its final, post-processed feature vector {fsupp,j}j=1Nsupp. These feature vectors, along with their corresponding known skill rating labels, are stored to form a reference feature library. For a new video requiring skill level assessment, feature extraction is performed to obtain its feature vector ftest. The cosine similarity between the feature vector of the video to be assessed and all feature vectors in the support set feature library is computed, yielding a similarity vector Stest=s1,s2,…,sNsupp. The index j*=argmaxjsj corresponding to the maximum value in the similarity vector is identified. The label ysupp,j* of the j* sample in the support set is assigned as the predicted score ytestˆ for the video under assessment. Implementation details All experiments were conducted on the same hardware setup. The CPU is an Intel(R) Xeon(R) Gold 6430 operating at 1900MHz frequency. The system is equipped with 512GB of RAM. The GPU device is an NVIDIA GeForce RTX 4090 D. The PyTorch 2.4.1 deep learning framework and MMAction2 were used as the foundational framework for the video understanding task. The feature extraction network in the experiment employs the TSM as the backbone, which primarily includes layers such as conv1, maxpool, layer1, layer2, layer3, and layer436. Among these, conv1 and maxpool contain only a small number of parameters and have minimal impact on the overall backbone. The model was pre-trained on the Something-Something V2 dataset35. When fine-tuning for this specific task, only the parameters of layer 4 in the backbone were trained. During training, the Adam optimizer was used with a learning rate of 0.001 and a weight decay of 0.0001. The model was trained for a total of 200 epochs. The parameters yielding the highest matching accuracy with the support set during the training process were recorded for subsequent validation on the test set. The batch size for the training, test, and support sets was set to 16. The temperature hyperparameter τ for the loss function was set to 0.07. The coefficients λ, α, and β in the total loss function were all set to 0.5. For data augmentation, the scales for multi-crop were (1, 0.875, 0.75, 0.66), and the random horizontal flip probability was 0.5. After extracting the corresponding support set, the remaining data was partitioned such that the test set comprised 20%, and the rest constituted the training set. Statistical analysis When evaluating model accuracy, a predicted skill level for the test set is considered correct only if it exactly matches the ground-truth label. However, considering that skill level assessment by experts can involve inherent subjectivity, the boundaries between adjacent scores may be ambiguous, particularly for operations with similar performance levels. Based on this, to more comprehensively reflect the model’s practical utility and its tolerance for slight rating discrepancies, this study introduces two additional tolerance accuracy metrics. A prediction is deemed correct if the absolute difference between the predicted skill level and the ground-truth label is less than or equal to 1. Similarly, a second metric considers a prediction correct if the absolute difference is less than or equal to 2. Furthermore, the calculation of the MacroAUC requires class scores for each test sample, indicating the likelihood of it belonging to each respective class. In conventional classification models, these scores are typically obtained directly from the model’s final output layer. However, the prediction method adopted in this study, which is based on contrastive learning and nearest-neighbor principles, derives those class confidence scores differently. Specifically, for a video from the test set awaiting evaluation, its normalized spatio-temporal features are first extracted. Then, the cosine similarity between this feature and the features of all samples in the support set feature library is computed. To obtain the confidence score Scoreftest,c that the test video belongs to a specific skill level c, this study takes the maximum similarity value found between the test sample’s feature fsupp,j and all sample features in the support set that belong to level c. This is formally defined in Eq. (9). where j needs to satisfy ysupp,j=c. These category scores are then used to construct ROC curves for each level and calculate the corresponding AUCc, and finally, the overall MacroAUC is obtained using formula (10). These class scores Sc are subsequently used to construct a ROC curve and calculate the corresponding AUCc for each class. Finally, the overall MacroAUC is derived using Eq. (10). where C is the total number of skill rating classes. Ethics statement This study was approved by the Institutional Review Boards of the First Affiliated Hospital of Wenzhou Medical University (KY2026-022). All the procedures adhered to the ethical standards outlined in the Declaration of Helsinki. Written informed consent was obtained from all participants prior to sample collection and data acquisition. Additionally, written informed consent was obtained from all participants for the publication of the identifiable videos and images included in this manuscript. Supplementary information Supplementary Information Acknowledgements This work was supported by Research Foundation of National Health Commission of China–Major Medical and Health Technology Project for Zhejiang Province (Grant No. WKJ-ZJ-2438), the National Natural Science Foundation of China (U25A20450) and Zhejiang Provincial Natural Science Foundation of China (Grant No. LMS26F020041). Author contributions Q.D., K.C., C.C., Y.Ga. and J.S. collected the data. B.Z., Y.Ge., H.C. and K.C. analysed the data. G.C., X.Z. and Z.C. conceived and supervised the project. C.H. and E.Z. designed the website. Q.D., K.C., C.H. and B.Z. contributed to data interpretation, method design, critical review, writing and revision of the manuscript. All authors have read and approved the final version of the manuscript. Data availability The datasets generated and/or analyzed during the current study are not publicly available due to restrictions imposed by the ethics approval to protect participant privacy, but are available from the corresponding author on reasonable request. Code availability The source codes of contrastive learning models and others used in this study have been deposited to GitHub (https://github.com/zboliff/Development-and-Application-of-an-Intelligent-Assessment-System-for-Medical-Clinical-Skill-Training). The following software versions were used: Python 3.8, PyTorch 2.4.1, and MMAction2 (v.1.2.0). Key hyperparameters and data preprocessing parameters used to generate and analyze the current datasets are detailed in the Methods section. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. These authors contributed equally: Qi Dai, Kaiyu Chen, Cheng Han, Boli Zheng. Contributor Information Zhaomin Chen, Email: chenzhaomin123@gmail.com. Xuebo Zhu, Email: xuebo.zhu@wmu.edu.cn. Gang Chen, Email: chen.gang@wmu.edu.cn. Supplementary information The online version contains supplementary material available at https://doi.org/10.1038/s41746-026-02877-y. References Associated Data Supplementary Materials Supplementary Information Data Availability Statement The datasets generated and/or analyzed during the current study are not publicly available due to restrictions imposed by the ethics approval to protect participant privacy, but are available from the corresponding author on reasonable request. The source codes of contrastive learning models and others used in this study have been deposited to GitHub (https://github.com/zboliff/Development-and-Application-of-an-Intelligent-Assessment-System-for-Medical-Clinical-Skill-Training). The following software versions were used: Python 3.8, PyTorch 2.4.1, and MMAction2 (v.1.2.0). Key hyperparameters and data preprocessing parameters used to generate and analyze the current datasets are detailed in the Methods section.