BREAST+: An augmented reality interface that speeds up perforator marking for DIEAP flap reconstruction surgery Authors: Rafaela Timóteo, David Pinto, Pedro Matono, Carlos Mavioso, Maria‐João Cardoso, Pedro Gouveia, Tiago Marques, Daniel Simões Lopes Published: 2024 ABSTRACT AbstractDeep inferior epigastric artery perforator flap reconstruction is a common technique for breast reconstruction surgery in cancer patients. Preoperative planning typically depends on radiological reports and 2D images to help surgeons locate abdominal perforator vessels before surgery. Here, BREAST+, an augmented reality interface for the HoloLens 2, designed to facilitate accurate marking of perforator locations on the patients' skin and to seamlessly access relevant clinical data in the operating room is proposed. The system is evaluated in a controlled setting by conducting a user study with 27 medical students and 2 breast surgeons. Quantitative (marking error, task completion time, and number of task repetitions) and qualitative (perceived usability, perceived workload, user preference and user satisfaction) data are collected to assess BREAST+ performance during perforator marking. The average time taken to mark each perforator is 7.7 ± 6.5 s, with an average absolute error of 6.8 ± 2.6 mm and an estimated average deviation of 3.6 ± 1.4 mm. The results revealed non‐negligeable biases in user estimates likely attributed to depth perception inaccuracies. Still, the study concluded that BREAST+ is both accurate and considerably more efficient (∼6 times faster) when compared to the conventional perforator marking approach. INTRODUCTION In Europe, 20%–40% of breast cancer patients are submitted to mastectomies, leaving the affected women in need of a reconstruction surgery to regain the female shape [1]. Deep inferior epigastric artery perforator (DIEAP) flap reconstruction surgeries are a safe, common, and reliable technique for immediate or delayed breast reconstruction surgery [2, 3]. According to the American Society of Plastic Surgeons, over 150 thousand breast reconstruction surgeries were performed in 2022, from which almost 20 thousand were performed using the DIEAP flap method [4]. The procedure consists of harvesting the lower abdomen's excess skin and fat, which contains perforator vessels that will act as the main blood suppliers of the reconstructed breast tissue [2]. To preserve the perforators' vessels, surgeons ensure a cautious dissection of the perforators' vicinity. Nevertheless, pinpointing the location and trajectory of the perforators is a long and costly endeavour [5, 6]. This is primarily due to discrepancies between subjective preoperative assessments and personal observations during surgery. This is aggravated by the absence of a standardized protocol for preoperative perforator selection and the considerable variability in the location and trajectory of perforators among individuals [7, 8]. Preoperatively, all patients go through computed tomography angiography (CTA) imaging to confirm the presence of perforator vessels, revealing their locations and intramuscular trajectory of the source artery [5, 9]. In the operating room (OR), perforators are marked on top of the patient's abdominal skin using a pen and a measuring ruler based on the coordinates listed in the corresponding CTA imaging report, without any visual aid. Overall, given an average of eight perforators per procedure, this conventional way of marking perforators can take 6 to 8 min according to interviewed experienced breast surgeons. Augmented reality (AR) is an interaction paradigm that merges virtual objects in the real‐world environment. In the context of breast reconstruction surgery, AR has the potential to empower surgeons with ‘x‐ray vision’ allowing them to ‘see through’ the patient's skin by merging digital models of the patient's anatomy [10, 11, 12, 13]. AR headsets and projectors have been explored in DIEAP flap surgeries to display 3D models aligned with the patient's body to assist surgeons in decision‐making and perforator marking [14, 15, 16]. Nevertheless, the use of an AR headset offers the advantage of a hands‐free interaction and 3D visualization of a patient's high‐fidelity digital twin in the OR. Optical see‐through (OST) AR headsets have already proven their clinical potential by enabling access to information in real‐time without interrupting the surgeon's workflow or needing to move their gaze from the patient to another screen [17, 18]. However, the accurate display of the digital twin aligned with the patient's body is a common challenge. To use these technologies, it is necessary to implement a well‐calibrated system with a proper registration process to ensure that there are no relevant deviations in the OR [13]. Here, we developed BREAST+, an AR interactive application that runs in a HoloLens 2 headset, to allow the user to access the CTA imaging report in the OR and visualize the digital twin of the abdominal skin and perforators overlayed on the patient. The proposed study provides an AR tool to assist surgeons during perforator marking on the preoperative planning of DIEAP flap reconstruction surgeries. We assessed usability, perforator marking task accuracy, and task completion time of BREAST+ with 27 medical students and 2 breast surgeons. MATERIALS AND METHODS BREAST+ Our AR system displays two main visual components: a 3D model of the DIEAP flap region and a CTA report menu. While BREAST+ was developed to work with patient‐specific 3D digital twins, we conducted the usability study in a controlled environment with a medical mannequin (Laerdal Nursing Anne Simulator1). We built a 3D model of the outer surface of the mannequin with patient 3D anatomical data (Figure 1a) to grant a layer of realism, using the 3D modelling software Blender.2 First, we performed a manual surface scan of the Nursing Anne Simulator mannequin using a handheld device (CreaformTM3 Go!Scan 3D handheld) (Figure 1b,c). Then, from the patients who were recruited and consented to participate in this study, we chose one patient's CTA scan to segment the skin, fascia, umbilicus, and perforators a computer vision‐assisted method and reconstructed them in 3D [9]. Perforator information included caliber and 3D geometric paths (intramuscular and subcutaneous) of each vessel. A total of six perforators were modelled based on the CTA images, which were distributed across the four quadrants centred on the umbilicus, to assess potential correlations between their positions and marking error. To better distinguish the intramuscular and subcutaneous paths of perforators, the surgeons involved in the co‐design sessions recommended the colours of red and green, respectively (Figure 1). Finally, we manually aligned the 3D mannequin data with the 3D patient‐specific data in Blender2 and digitally measured the position of each perforator in a coordinate system centred in the umbilicus (ground truth). Different points‐of‐view of the 3D model created in Blender (previous page). (a) The lower abdominal area with perforators (green and red lines) and the fascia (blue area) are based on real patient data. (b) Side perspective of the abdomen showing the perforators’ paths. (c) The white circles at the skin's surface represent the projection of the point where the perforators cross the fascia, that is, the marking point. Top view of the complete model: lower abdominal anatomy plus surface scan of the mannequin. The CTA report menu, on the other hand, appears as a floating panel adjacent to the patient, and displays the same information about the perforators (e.g., caliber, intramuscular path, and localization) as in the conventional paper‐format report. Additionally, this report menu incorporates buttons for selecting which anatomical parts are displayed on the model, a fine‐tuning button for precise model positioning, two pin buttons for menu and model anchoring, and a slider for adjusting model skin transparency (Figure 2a). The AR application was developed using Unity4 (version 2020.3.8f1), Microsoft Reality ToolKit5 (MRTK) (version 2.7.3), and Microsoft Visual Studio6 (version 17.1.1). The application was deployed to the Microsoft HoloLens 2.7 (a) BREAST+ user interface. Report‐like menu to interact with the model. (b) The light‐blue button shows that the fine‐tuning box is enabled. Fine‐tuning box surrounding the model with handles for translation and spheres for rotation. Several MRTK gesture interactions were made available to the user: (i) grabbing the menu using one or both hands to position it; (ii) pressing the buttons with a finger for enabling or disabling functionalities; (iii) using a pinching motion with the index finger and the thumb to interact with the slider; (iv) grabbing the 3D model using one or both hands for manipulation; (v) using a pinching motion gesture to interact with the widgets on the fine‐tuning box (for translation and rotation on each axis). User study Two breast surgeons from the Breast Unit, Champalimaud Foundation and 27 medical students from the Lisbon School of Medicine, University of Lisbon were recruited and consented to participate in the user study. Participants were briefed on the system's recognized gestures for interface interaction. First, participants were asked to perform a habituation task, freely interacting with basic 3D models and learning input gestures such as grabbing, selecting, and pinching in a HoloLens 3D viewer app. Afterwards, participants completed three main tasks to assess the accuracy and usability of BREAST+: (i) marking the six perforators on the mannequin according to the 3D model displayed through the HoloLens using children's makeup and a cotton swab (Figure 3b,c); (ii) selecting which anatomy elements of the model are displayed using the menu buttons and transparency slider; (iii) manually aligning the 3D model to the mannequin, using the fine‐tuning box to make fine adjustments (Figure 3d). Before each participant began the marking task, the experiment leader (experienced in manual alignment) aligned the digital model to the mannequin, to minimize variability across markings by different subjects. Experimental setup. (a) Experimental setup containing a computer for answering questionnaires and a mannequin. (b) Participant wearing a HoloLens 2 executing the first task: marking the perforators on the mannequin's skin using a cotton swab. (c) Resultant localization of the perforators marked by the participants. Axes used throughout the experiment are represented in the bottom left corner). (d) Participants manoeuvring digital content in the augmented space while executing Task 2 and Task 3. To evaluate the system's accuracy, we collected quantitative data relative to perforator marking error (measured along the X and Y axes with a ruler (see Figure 3c), perforator marking completion time, manual alignment completion time, and number of task repetitions. For each participant and each perforator, we calculated the marking error along the two axes (Δx and Δy), and the absolute error. We estimated the systematic errors along the two axes by calculating the average of the errors of all the perforators for each participant. This systematic error encompasses translational deviations in the alignment of the 3D digital twin with the mannequin's body, as well as any eventual perceptual biases along each axis. We then calculated the deviation error for each perforator disregarding the systematic error. Subsequently, we analysed the marking errors along the two axes, the absolute error, and the deviation error in our subject population (n = 29) for each perforator separately. To evaluate the usability of BREAST+, upon task completion, we gathered qualitative data regarding perceived usability, workload, user preference, and user satisfaction. We used standardized usability evaluation metrics, such as System Usability Scale (SUS), NASA‐TLX, and perceived satisfaction forms, to evaluate BREAST+. We also conducted semi‐structured interviews to gather subjective information from participants. RESULTS AND DISCUSSION The participants included 27 medical students and 2 experienced breast surgeons (one with 15 years of experience and the other with less than 10 years of experience). In the student's cohort, the age ranged between 18 and 31 (average of 22.2 ±6.3 years), and there was a diverse representation: 12 medical students from the first year, 7 from the second, 3 students from the third, 2 students from the fourth, 1 student from the fifth, and 4 students from the sixth. Most students (83.9%) had no prior exposure to AR with the HoloLens (83.9%), and a notable portion had no knowledge of DIEAP flap reconstruction surgeries (71%). Except for the experienced breast surgeons, none of the students had observed or engaged in such procedures. Accuracy evaluation We tested the BREAST+ AR system for perforator marking with 29 participants. With our interface and 3D model aligned with the mannequin, participants demonstrated rapid proficiency in marking perforators, achieving an average completion time of 46 ± 39 s in total (7.7 ± 6.5 s per perforator). Only five participants repeated marking a single perforator. Insights from domain experts indicate that marking a single perforator conventionally takes approximately 45 s. Thus, our system's performance reflects a substantial enhancement, yielding an estimated speed up of ∼6 times. Our time measurement does not include the registration of the digital model to the patient, since this registration was manually performed by the experiment leader for simplification and does not reflect the overall time savings expected in real‐world applications. Thus, we emphasize our solution's potential in enhancing the marking task itself through immediate display of perforator locations, considering future work on more adequate registration methods that will not be performed manually by the surgeon. The average absolute error measured was 6.8 ± 2.6 mm (Figure 4a) while the estimated average deviation, after accounting for systematic errors, was 3.6 ± 1.4 mm (Figure 4b). Given that experts consider an error value of 5 mm acceptable, our system performed within a tolerable range. (a) Box plot with accuracy error and outliers per perforator. (b) Box plot with average deviation (after accounting for systematic errors) and outliers per perforator. (c) Box plot with errors along the X and Y axes per perforator. Analysing the difference between these types of errors provides some interesting observations about the system performance. The fact that after accounting for systematic deviations the error was almost reduced to half, suggests that the system is relatively precise but systematic errors introduce biases in the perforator marking position. We hypothesize that there are two major contributions for these systematic errors. First, translational alignment errors between the 3D model and mannequin introduce biases common to all perforators. However, since for each participant, the model is aligned independently by the experiment leader, when looking over all participants, these errors should cancel out. There can also be systematic errors that are common to all participants, for example, the same error along a certain direction. This type of errors represents intrinsic biases in our experimental setup. We first looked into the effect of perforator location in marking error (see Figure 4c) and observed that there was no significant effect (p‐value = 0.72, t = 0.58, t‐critical = 2.27, single factor ANOVA test). This suggests that the magnitude of the error does not depend on the position of the perforator in the mannequin's body. We then looked at errors along the two directions (X and Y axes). Interestingly, we observed directional biases in errors along the X axis, consistently towards the left for all subjects and across perforators (X = [−5.1; −2.8], p‐value = [1.1 × 10−6; 3.4 × 10−3], t = [−5.9; −2.9], one sample, one‐tailed t‐test). Errors along the Y axis did not show similar biases (p‐value = [2.1 × 10−6; 1.95 × 10−1], t = [0.87; 3.98], one sample, one‐tailed t‐test). This bias along the X axis is likely attributable to incorrect depth estimates, given that the perforator marking task was always performed with the user positioned on the left side of the mannequin. Subjects consistently marked the position of the perforators towards their side. There may also be errors arising from the headset calibration differences between users. However, those errors should not significantly impact the observed systematic errors in our study, as they would vary among participants and do not fully explain the consistent bias we detected. Additionally, this study did not include the time estimation for starting the application, performing eye calibration, automatic registration, and the marking task. These limitations will be addressed in future work. Usability evaluation The usability SUS score average of 80.1 (scores range from 1 to 100), which stands in an acceptable and positive range within the scale, reflects an above‐average user experience and commendable level of usability [19]. Analysis of the raw NASA‐TLX scores reveal noteworthy performance by participants (73 score), despite medium frustration (22 score), somewhat high mental workload (31 score), a high effort required for task success (52 score), and medium physical (28 score) and temporal (29 score) workload [20, 21]. The raw NASA‐TLX scores show above‐average efforts due to: (i) sensitivity of the gesture recognition; (ii) inexperience of the users; (iii) challenges on the manual alignment with manipulation of the 3D model using the fine‐tuning box. Feedback gathered from both the satisfaction questionnaire and semi‐structured interview unanimously highlighted participants' appreciation for the augmented 3D content visualization and interactive capabilities offered by the system. However, many users expressed encountering challenges when engaging with the fine‐tuning box, attributing this difficulty to a lack of familiarity with the gestures required and the system's sensitivity. A commonly reported issue was the persistent recognition of gestures even when users attempted to release objects, resulting in unintended movements. These issues were primarily related to the manipulation of the model, which explains the high NASA‐TLX scores. Despite these challenges, most participants recognized the potential of this system as a viable alternative to conventional methods in the future, contingent upon improvements in accuracy and user confidence, insertion in the education of medical students, and consequent practice. A limitation here is the absence of both quantitative and qualitative task performance analysis related to anatomy selection and manual alignment. Here we did not conduct this analysis because both tasks intended to assess usability. Our decision is considering a future implementation of automatic alignment of the digital twin to the patient, which would eliminate the need for manual alignment by the surgeon, reducing the relevance of these functionalities. Therefore, this study focuses on demonstrating the potential of AR in expediting the marking process itself, independent of alignment tasks, which we considered outside the primary scope of our study. BREAST+ demonstrates potential in clinical effectiveness by speeding up surgery's preoperative planning by means of a user‐friendly interface, that easily integrates into clinical workflows. Our system contains the fundamental technological components to ensure cohesive operation. For future adoption in clinical practice, it is crucial to address aspects of health technology assessment. This will encompass the utilization of a medically certified AR headset, such as the Magic Leap 2,8 correction of depth perception errors by providing more cues (reflections, highlights, and shadows), and implementation of a robust automatic alignment algorithm to achieve sub‐millimetre levels of precision and reduce registration time. Regarding usability, future research should be towards optimizing interaction techniques, as well as revisiting colour schemes, to be more inclusive towards colour‐blinded users. CONCLUSION The present study proposes BREAST+ as a novel solution for perforator marking for free flap reconstruction surgery, using the abdominal wall tissue (DIEAP flap). Our user‐friendly AR interface suitable for the HoloLens 2 facilitates the marking of perforator vessels on the patient skin preoperatively, offering access to clinical information within the OR. This immersive solution promotes a significant acceleration in perforator marking, achieving a speed up of around six times compared to conventional methods, with acceptable deviation error. The observed accuracy errors are very likely due to depth perception inaccuracies common in AR environments [22]. In addition, challenges related to gesture recognition sensitivity were noted among most participants. Future research efforts should prioritize both gesture recognition sensitivity and depth perception issues to further enhance the accuracy and usability of AR‐based surgical planning systems. AUTHOR CONTRIBUTIONS Rafaela Timóteo: Data curation; formal analysis; investigation; methodology; software; validation; visualization; writing—original draft; writing—review and editing. David Pinto: Conceptualization; investigation; methodology; supervision; validation; writing—review and editing. Pedro Matono: Data curation; software; writing—review and editing. Carlos Mavioso: Investigation; validation; writing—review and editing. Maria João Cardoso: Conceptualization; project administration; resources; writing—review & editing. Pedr o Gouveia: Conceptualization; funding acquisition; project administration; resources; supervision; writing—review and editing. Tiago Marques: Formal analysis; investigation; methodology; project administration; supervision; writing—review and editing. Daniel Simões Lopes: Conceptualization; funding acquisition; investigation; methodology; supervision; visualization; writing—review and editing. CONFLICT OF INTEREST STATEMENT The authors declare no conflicts of interest.