Automating Feedback Analysis in Surgical Training: Detection, Categorization, and Assessment These authors contributed equally
Firdavs Nasriddinov, Rafal Kocielnik, Arushi Gupta, Cherine Yang, Elyssa Wong, Anima Anandkumar, Andrew J. Hung · 2024
Read Automating Feedback Analysis in Surgical Training: Detection, Categorization, and Assessment These authors contributed equally by Firdavs Nasriddinov, Rafal Kocielnik, Arushi Gupta, Cherine Yang, Elyssa Wong, Anima Anandkumar, Andrew J. Hung (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00760.txt.
Introduction
Firdavs Nasriddinov and co-authors built the first system to rebuild surgical teaching talk from raw recordings in a 2024 paper. During live operations, trainers give spoken feedback that keeps patients safe and builds trainee skill over time. That talk is messy and full of specialist terms, so it has resisted study at scale.
Their framework finds feedback in the recordings, sorts it into types, and rates it. Turning loose talk into structured data lets programs count what kinds of guidance trainees get and when. Gaps in feedback could then be fixed with targeted teaching. The work opens a data path for improving how surgeons are trained, grounded in what is actually said in the room.
Catalog record
| CREATOR | Firdavs Nasriddinov, Rafal Kocielnik, Arushi Gupta, Cherine Yang, Elyssa Wong, Anima Anandkumar, Andrew J. Hung |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00760
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Automating Feedback Analysis in Surgical Training: Detection, Categorization, and Assessment These authors contributed equally Firdavs Nasriddinov, Rafal Kocielnik, Arushi Gupta, Cherine Yang, Elyssa Wong, Anima Anandkumar, Andrew J. Hung Abstract This work introduces the first framework for reconstructing surgical dialogue from unstructured real-world recordings, which is crucial for characterizing teaching tasks. In surgical training, the formative verbal feedback that trainers provide to trainees during live surgeries is crucial for ensuring safety, correcting behavior immediately, and facilitating long-term skill acquisition. However, analyzing and quantifying this feedback is challenging due to its unstructured and specialized nature. Automated systems are essential to manage these complexities at scale, allowing for the creation of structured datasets that enhance feedback analysis and improve surgical education. Our framework integrates voice activity detection, speaker diarization, and automated speech recognition, with a novel enhancement that 1) removes hallucinations (non-existent utterances generated during speech recognition fueled by noise in the operating room) and 2) separates speech from trainers and trainees using few-shot voice samples. These aspects are vital for reconstructing accurate surgical dialogues and understanding the roles of operating room participants. Using data from 33 real-world surgeries, we demonstrated the system’s capability to reconstruct surgical teaching dialogues and detect feedback instances effectively (F1 score of 0.79 ± \pm 0.07). Moreover, our hallucination removal step improves feedback detection performance by ≈ \approx 14%. Evaluation on downstream clinically relevant tasks of predicting Behavioral Adjustment of trainees and classifying Technical feedback, showed performances comparable to manual annotations with F1 scores of 0.82 ± \pm 0.03 and 0.81 ± \pm 0.03 respectively. These results highlight the effectiveness of our framework in supporting clinically relevant tasks and improving over manual methods. 1 1 footnotemark: 1 † † volume: 259 † † year: 2024 † † submitted: LEAVE UNSET † † published: LEAVE UNSET † † workshop: Machine Learning for Health (ML4H) 2024 keywords robot-assisted surgery, teaching, feedback, surgical assessment, automated speech recognition Data and Code Availability Due to the nature of the data it will only be available on request. The code is publicly available on our github . Institutional Review Board (IRB) The data used was collected under the IRB of the University of Southern California (HS-17-00113). Figure 1: Overview of our automated surgical feedback detection and assessment framework, organized into: Dialogue Reconstruction, which integrates (A) Voice Activity Detection (VAD) to detect timespans of speech. (B) Speaker Diarization (DIA) to differentiate speech from different speakers. (C) Automated Speech Recognition (ASR) to transcribe the audio into text. Dialogue Refinement focuses on (D) Hallucination Removal & Trainer/Trainee Identification which ensures the correct identification of speakers and removes irrelevant audio snippets. Clinical Tasks step applied our framework to clinically relevant tasks from Wong et al. 2023 , including (E) Feedback Detection (Task 1) which identifies when feedback is delivered. (F) Feedback Effectiveness (Task 2) which evaluates the impact and effectiveness of the feedback. (G) Feedback Components (Task 3) which categorizes the feedback into Anatomic, Procedural, and Technical. 1 Introduction Importance: Formative verbal feedback delivered by a trainer to a trainee during surgical procedures is crucial for immediate correction and guidance ( Wong et al. 2023 ) as well as for fostering long-term skill acquisition ( Agha et al. 2015 ) . High-quality feedback has been shown to significantly enhance intraoperative performance ( Bonrath et al. 2015 ) , accelerate surgical skill development ( Ma et al. 2022 ) , and bolster trainees’ sense of autonomy ( Haglund et al. 2021 ) . The quality of communication has also been directly linked to operative outcomes ( D’Angelo et al. 2020 ) . This underscores the importance of understanding current practices of feedback delivery on the quality of surgical education ( El Boghdady and Alijani 2017 ) . Given the critical role of feedback in surgical training and patient outcomes, automating feedback analysis is essential. It can help streamline the understanding of current practices, enhance trainer-trainee communication, and reduce inconsistencies in manual analysis. By improving training methods and creating structured datasets, automation enables the development of standardized, cost-efficient guidance systems ( Ma et al. 2024 ) , ultimately elevating the quality of surgical education and patient care. Challenges: Effectively quantifying and analyzing real-world surgical feedback at scale presents significant challenges. The contextual understanding of surgery, including its phases, specific procedures, and instructional tasks, necessitates deep domain expertise ( Haque et al. 2024 ) . Differentiating genuine feedback from mere comments or unrelated remarks requires a thorough familiarity with operating room interactions ( Wong et al. 2023 ) . Moreover, discerning which feedback is actionable and merits detailed annotation demands surgical teaching expertise. Additionally, the often extensive durations of surgeries coupled with the complex social dynamics among medical staff lead to many exchanges that do not qualify as clinically relevant feedback. Due to these challenges, previous studies have depended on labor-intensive manual identification and annotation of feedback by trained human raters ( Wong et al. 2023 ; Hauge et al. 2001 ; Blom et al. 2007 ) . Such manual annotation requires considerable time and effort from skilled annotators and is impractical to perform at scale, thereby hindering both the systematic quantification of feedback and the creation of structured, clinically-aligned datasets essential for developing automated feedback delivery systems ( Laca et al. 2022 ) . Approach: We analyze full-length recordings of live surgical cases to automatically detect surgical feedback delivery moments based on the clinically validated definition of feedback from Wong et al. 2023 . We further automatically categorize the content of this feedback and assess its effectiveness. We develop a framework for reconstructing teaching interactions in the operating room consisting of 3 consecutive steps as shown in Figure 1 : 1. Dialogue Reconstruction : Combines Voice Activity Detection (VAD) to identify speech timespans in continuous audio, Speaker Diarization (DIA) to provide timespans for speech from different speakers, and Automated Speech Recognition (ASR) to transcribe audio within each timespan. 2. Dialogue Refinement : Removes ASR and DIA hallucinations and identifies trainer/trainee speaking turns by leveraging cosine similarity between embeddings of candidate speech segments and few-shot embedded voice samples from the trainer and trainee present in the operating room. 3. Clinical Task Evaluation : We demonstrate the crucial importance of each step on downstream clinically relevant tasks form Wong et al. 2023 : Feedback Detection , Feedback Effectiveness Assessment , and Feedback Components Identification . We perform experiments on recordings of 33 surgeries from prior work on feedback ( Wong et al. 2023 ) , which contains clinically validated human annotations of 4.2k feedback instances along with component and trainee behavior annotations (Table 1 ). Table 1: Statistics of the dataset obtained from Wong et al. 2023 containing full video and audio recordings of 33 surgeries along with available human annotations for different clinically relevant tasks which we support in our feedback detection framework. Task Dimension Definition Len/Count % Pos Dialogue - Verbal interactions in the operating room. 78h\>52m - Feedback Detection - Any dialogue intended to modify trainee thinking or behavior. 4210 - Component Classification Anatomic Familiarity with anatomic structures and landmarks. 1194 28.4% Procedural Pertains to timing and sequence of surgical steps. 851 20.2% Technical Performance of a discreet task with appropriate knowledge of exposure, instruments, and traction. 3489