FBG‐driven simulation for virtual augmentation of fluoroscopic images during endovascular interventions Authors: Valentina Scarponi, Juan Verde, Nazim Haouchine, Michel Duprez, Florent Nageotte, Stéphane Cotin Published: 2024 ABSTRACT AbstractEndovascular interventions are procedures designed to diagnose and treat vascular diseases, using catheters to navigate inside arteries and veins. Thanks to their minimal invasiveness, they offer many benefits, such as reduced pain and hospital stays, but also present many challenges for clinicians, as they require specialized training and heavy use of X‐rays. This is particularly relevant when accessing (i.e. cannulating) small arteries with steep angles, such as most aortic branches. To address this difficulty, a novel solution that enhances fluoroscopic 2D images in real‐time by displaying virtual configurations of the catheter and guidewire is proposed. In contrast to existing works, proposing either simulators or simple augmented reality frameworks, this approach involves a predictive simulation showing the resulting shape of the catheter after guidewire withdrawal without requiring the clinician to perform this task. This system demonstrated accurate prediction with a mean 3D error of 2.4 ± 1.3 mm and a mean error of 1.1 ± 0.7 mm on the fluoroscopic image plane between the real catheter shape after guidewire withdrawal and the predicted shape. A user study reported an average intervention time reduction of 56% when adopting this system, resulting in a lower X‐ray exposure.Keywords: biomedical imaging, finite element analysis, image enhancement, optical sensors INTRODUCTION AND RELATED WORKS Over the last decades, endovascular procedures have revolutionized the diagnosis and treatment of a broad range of cardiovascular diseases. The success of endovascular procedures is due to their efficiency in treating these pathologies and to their minimal invasiveness. Compared to surgical approaches, they lead to fewer adverse events, less pain, faster recovery times, and shorter hospital stays, thus significantly reducing costs [1, 2]. However, the limiting factor of endovascular procedures is their complexity, requiring extensive training and access to specially equipped facilities [3]. During the intervention, the patient is placed on the operating table, and a tiny incision is made in the femoral artery. Then a flexible guidewire is inserted through the incision, and a thin hollow tube (the catheter) is slid over the wire. Both instruments, guidewire, and catheter, are manually manipulated from their proximal end, located outside the patient at the incision site, and the only visual feedback used to determine the location of the instruments inside the human body is given by 2D X‐ray images (fluoroscopy). The main challenge of these interventions is to navigate through the “vascular highway” until reaching the area to be treated. This navigation can be divided into two main tasks: the navigation through the lumen of a vessel and the cannulation of collateral branches. The latter consists of accessing, from a large main vessel, to small side vessels (Figure 1) and is sometimes repeated several times until reaching the target that can be, in the case of neuro‐vascular interventions, more than 1 meter away from the insertion point. Initially, the guidewire leads the navigation (its very soft tip avoids puncturing the vessel wall) and the catheter slides over the wire to progress through the artery. Both devices have different shapes and stiffness. In some cases, the guidewire is more flexible than the catheter. In other cases, it is stiffer. Both devices also have different tip shapes designed according to the geometry of the local vascular anatomy. Real fluoroscopic image showing catheter and guidewire navigation in the abdominal artery and renal artery cannulation (left) (CC BY 3.0). A synthetic fluoroscopic image (right) augmented with our predictive catheter model is used for assisted renal artery navigation. An essential aspect of endovascular navigation consists in modifying this predefined tip shape to facilitate the progression of the device through the anatomy. This modification of the tip shape is done manually by pushing or pulling the guidewire through the catheter, as illustrated in Figure 2. Mastering this manipulation requires significant training and even experienced clinicians usually work by trial and error, particularly during the cannulation step. Indeed, the intervention is performed under fluoroscopic imaging which provides only a two‐dimensional view of a complex 3D anatomy. The acquisition of these images involves the use of X‐rays, which are harmful to both the patient and the clinical team [4]. The injection of a contrast agent, required to visualize the vessels during navigation, is also deleterious for the patient and can lead to kidney failure [5]. Exposures to X‐ray and contrast agent injections are amplified when the clinician struggles with navigation or cannulation. Illustration of the straightening effect of the catheter that occurs when a stiff guidewire is pushed through the catheter. As the guidewire advances, its stiffer section changes the shape of the catheter. The parameter ‘s’ on each image indicates the local curvilinear coordinate of the guidewire within the catheter. Since the visual information provided by X‐rays is insufficient, mixed reality (MR) and augmented reality (AR) systems able to improve and enrich the information provided to the clinician during the interventions are becoming increasingly popular. In their study, Garcia et al. [6] developed a system to support the caregivers during endovascular aortic repair, visualizing 3D and 2D information through Microsoft Hololens. To assist the clinician during transcatheter aortic valve implantation, Soulami et al. [7] created augmented fluoroscopy images showing the patient's aortic root. These works present essentially visualization systems that provide more detailed information about the 3D anatomy of the patient, but they do not fulfil the requirements of guidance systems. Other works focus on the development of simulators for a safer training of new interventional radiologists. In this case, it is crucial to reproduce as accurately as possible the behaviour of the real instruments through accurate physical models. To this purpose, Korzeniowski et al. [8] exploited the Cosserat model, Wang et al. [9] modelled the devices as a set of linked rigid rods and Luboz et al. [10] modelled the guidewire through linked rigid bodies. All these studies propose simulated environments suitable for surgical training, but do not aim to provide intraoperative support to the surgeon. Contribution In the present work, we propose a mixed reality framework that facilitates endovascular navigation. Our system is able to overlay, on the fluoroscopic image, not only the virtual vascular anatomy but also the predicted shape of the device for a predetermined action. By fusing fluoroscopic image augmentation techniques and physics‐based simulations, our system is able to provide more informative fluoroscopic images, supporting clinicians with additional visual information. We developed a data‐driven endovascular navigation method that couples physics‐based models with a shape sensing system based on fibre Bragg grating (FBG) technology. Our system makes it possible to predict the device's position and shape in situations that have not yet occurred. To the best of our knowledge, this is the first time a system providing this kind of support has been proposed: not only do we visually augment fluoroscopic images with 3D anatomical information, but we also show the result of the clinician's actions through real‐time, predictive simulation (Supporting Information). This additional support can reduce the number of failed attempts and, consequently, shorten the duration of the intervention, as well as the number of contrast agent injections and X‐ray exposure for both the patient and the caregivers. Section 2 describes the shape‐sensing solution. Sections 3 and 4 present the physics‐based model and how it is used to regularize the sensing data and predict future shapes based on the current user input. Section 5 reports the details of the experimental setup we used to test our system, and Section 6 shows our results both in terms of prediction accuracy and usefulness for the physician when performing endovascular procedures on a phantom. THE TRACKING SYSTEM To achieve our goal, it is essential to have a reliable system to track the position and the 3D shape of the devices in the real world. Nowadays, this information is either extracted from 2D images [11], exploiting for example convolutional neural networks [12, 13], which can only provide 2D information, or using electromagnetic sensors [6, 14], whose data might become unreliable if metallic components are present in the operative field. In this work, the reconstruction of the 3D shape and position of the devices relies on FBG sensors, which, presenting a diameter in the order of 125 μm, are suited to be integrated into catheters and guidewires. FBG is a particular type of optical fibre in which the index of refraction, describing the light‐bending ability of a medium, has permanently been changed in a periodic pattern along the fibre's core by optical absorption of UV light [15]. The region that underwent the modification of the index of refraction is called grating. When an incident light spectrum propagates inside the grating, the whole spectrum is propagated, except for a specific wavelength, which is reflected. This wavelength takes the name of Bragg wavelength and can be obtained as λB=2nefΛ, where nef is the optical fibre refraction index, and Λ is the grating period. Nowadays, optical fibres containing multiple cores (MultiCore Fibres, MCF) in the same cladding are developed (Figure 3). Over the length of a single optical fibre, multiple FBGs can be inscribed, allowing for the reconstruction of the instrument deformation along its length. (a) Representation of a 4‐core MCF. λi is the wavelength of the ith core, θi is the counter‐clockwise angle between the x‐axis and the ith core, while θb is the bending plane angle with respect to the x‐axis. (b) Isometric view of a MCF. lz is the distance between two consecutive FBG. When the optical fibre undergoes an axial deformation, the Bragg wavelength λB experiences a proportional shift. By measuring this change, it is possible to exploit FBGs as strain sensors, able to measure the deformation. For each core i, the strain ε is linked to the curvature κ through: where d represents the distance from the neutral bending plane and θi the counter‐clockwise angle between the x‐axis and the ith core (see Figure 3). In this formula, the only two unknown parameters are the curvature κ and the angle of the bending plane θb. Thus, only two outer core measurements are needed to find these two values, but more cores can be added to improve the robustness. Twisting and buckling are not taken into account in (1) and the symmetry around the bending plane is considered. The strain change can be induced either by mechanical strain or by thermal expansion. MCFs usually present a central core that coincides with the neutral axis of the fibre: if the axial strain is negligible, the Bragg wavelength changes of this core (ΔλB,1) can be used to compensate for wavelength changes due to temperature fluctuation in the outer cores. Given these considerations, the strain changes of the core i can be defined as: where Sε is the strain sensitivity coefficient. To reconstruct a shape starting from a strain, both a set of unstrained Bragg wavelength λ0[n]=λB0[n], where n defines the number of FBG inscribed in the fibre, and a set of measurements of the discrete FBGs wavelength λ[n]=λB[n] are needed. Calling s the arc length variable, κ(s) and τ(s)=dθbds describe how the curve's tangent T evolves along its length. The Cartesian position Cs(s) can then be obtained by integrating Equation (3), which can be solved using various methods. For further details, please refer to [16]. THE SIMULATED ENVIRONMENT This section mainly describes how the catheter and the guidewire are modelled (Section 3.1). Being able to precisely simulate the physical behaviour of these objects is crucial to obtaining an exact prediction of the position of the catheter after guidewire withdrawal. For this reason, particular attention was paid to the modelization of their mechanical and geometrical properties. The obtained models were then used to set up a numerical simulation in which the two devices are navigated together into a vascular network (Section 3.2). In this context, it is essential to accurately detect and manage the contacts between the devices and the wall of the vascular network, but also to handle the interaction between the two devices. To provide immediate visual feedback, it is crucial to perform these operations in real‐time. Catheter and guidewire modelling Both the catheter and the guidewire are modelled by exploiting the finite element method (FEM) and, in particular, the Timoshenko beam theory [17]. The FEM allows finding an approximate numerical solution to the mathematical model of a physical problem. The key aspect is the discretization of a continuum system, which allows obtaining the solution at a finite number of points by solving algebraic equations instead of solving differential equations in the continuum domain. The matrix form of the system of equations to be solved is reported in Equation (4). The solution is computed by adopting an Euler implicit solver. M denotes the mass matrix, K the stiffness matrix, dt the time step and v and Δv the velocity and the velocity variation respectively. The term f, which is a function of the current positions x(t), represents the internal and external forces applied to the system. The forces are computed through the Timoshenko beam model, based on the Kirchoff rod theory. This formulation is suitable to simulate 1D flexible structures, like guidewires and catheters, and has been exploited for this purpose by many other works (see [18] for instance). To solve Equation (4), matrix (M−dt2K) needs to be inverted using a solver. The particular structure of the problem ensures to have a tridiagonal matrix, allowing the use of a block tridiagonal (BTD) linear solver for calculating the solution of the system, which reduces computational time and makes it possible to simulate the deformation of the virtual device in real‐time. For the simulation to be accurate, it is essential to correctly tune the geometrical and mechanical characteristics of the devices. This work adopts real device characteristics and choses the Young Modulus of all the devices by means of a bending test, performed both in the real and in the simulated environment. The Young Modulus of the models was iteratively changed until the value that minimized the difference between simulated and real data was found. Numerical simulation of endovascular navigation Interactive simulation of the catheter or guidewire within the vascular anatomy is essential to reach our objective. The 3D anatomy of the patient can be easily reconstructed from pre‐operative magnetic resonance or computed tomographic angiography images. Furthermore, the progress of deep learning techniques allows nowadays to automatically segment and reconstruct a 3D model of the anatomy [19]. The challenge lies in the real‐time simulation of the contacts between deformable objects. The more contacts, the higher the computational and time cost of the operation. In our work, two types of interactions are present: the interactions between the devices and the vessel wall and the interaction between the two devices themselves. In particular, the catheter and the guidewire, because of their relative sizes and since they are inserted inside each other, are constantly in contact in the navigation phase. Solving the contacts between these two instruments in a classical way would involve a high computational cost. This is the reason why two different contact models were considered in our framework, a classical model for the interactions between the devices and the vessel wall (Section 3.2.1) and a modified method suited to our case to simulate the interaction between the instruments (Section 3.2.2). Interaction between the devices and the vessel The process of detecting and handling the contacts between two objects in the simulated environment involves two steps: the identification of all the elements that are possibly in contact and the resolution of the physics of the colliding objects according to their interaction. To compute the dynamic of these objects, a Lagrange multiplier approach and a single linearization by time step were used. Equation (4) becomes: where HTλ is the vector of constraint forces contribution, with H containing the constraint directions and λ the Lagrange multipliers. Equation (5) is solved in two steps: 1Each object is solved independently through the resolution of the equation AΔvfree=b. Each object is solved independently through the resolution of the equation AΔvfree=b. 2The constraints are solved by, first of all, individuating the points that risk being in contact and then solving Equation (6), which returns a corrective change in the velocity (vcorr). (6)AΔvcorr=dtHTλ,δ˙=W·λ,where W=HA−1HT is the matrix of the linearized constraint system and δ˙ is the interpenetration velocity. The constraints are solved by, first of all, individuating the points that risk being in contact and then solving Equation (6), which returns a corrective change in the velocity (vcorr). (6)AΔvcorr=dtHTλ,δ˙=W·λ,where W=HA−1HT is the matrix of the linearized constraint system and δ˙ is the interpenetration velocity. Once both vfree and vcorr are calculated for each interacting object, it is possible to compute the new position for each model's element, exploiting the Gauss–Seidel algorithm. Given the large number of contacts, the lightweight and the high stiffness of the instruments, the resulting system is affected by convergence issues due to the ill‐conditioning of the problem, in particular of the matrix A. This results in slower computational times or instabilities when resolving the contact problem. To improve the condition of A matrix, we apply a diagonal scaling [20], a particular type of matrix preconditioning. Applying a diagonal scaling on the matrix A has a similar effect as adding mass to the system: this operation, besides stabilizing the system resolution, modifies the physics of the problem. To avoid modifying the physics of the simulated device, we apply the diagonal scaling of A only during the constraint resolution step, before performing the computation of W. This allows to stabilize the system, without modifying the physics. Note that in our current simulation, the vessel walls are considered rigid, so only the configuration of the devices needs to be recomputed when contact happens. Models combination According to the method described in the previous section, the higher the number of contacts, the more the computational complexity augments. When considering the guidewire inserted inside the catheter, the number of contacts and constraints to be met is huge. To eliminate this problem, in this work, instead of considering the catheter and the guidewire as two distinct objects interacting with each other, the two devices are modelled as a single beam with mechanical and geometrical properties varying based on the relative insertion of the two instruments, with an approach similar to [21]. The beam is represented by means of a curvilinear abscissa and to simulate the sliding of one instrument over the other, the mechanical properties of the model are gradually modified as the guidewire slides inside the catheter. COUPLING FBG DATA WITH BIOMECHANICAL SIMULATION Data‐driven endovascular navigation The system setup involves two elements: the replica in the simulated environment of the motion of the actual devices manipulated by the user and the computation, starting from this configuration, of additional information provided to the clinician directly on the fluoroscopic images (Section 4.2). Concerning the computation of the mixed view, having a reliable physics‐based model (see Section 3.1), is crucial to anticipate the behaviour of the device and show the clinician a truthful prediction to help her/him perform the task more efficiently. The prediction is computed at each time step, using the real device shape and position and accounting for the interaction between this simulated catheter and the real vessel walls, reproduced in the simulated environment. The real device shape is reconstructed by exploiting FBG data and the physics‐based model described in Section 3.1. Since the data gathered by the sensor are intrinsically affected by errors, this choice was necessary: the process reported in Section 2, which describes how the shape is reconstructed starting from the measurements of wavelength shift, introduces an error between the real shape and the shape reconstructed by the sensor. Al‐Ahmad et al. [16] measured this error both in static and dynamic conditions, finding a maximum value of 2.5 mm, which can increase up to 21.2 mm when sharp bends occur. Even if occasional, such errors can compromise the robustness of our predictive system. For this reason, to regularize the data coming from the sensor, we adopted the physics‐based model described in Section 3.1 to simulate also the real guidewire and catheter. The fibre is attached to the catheter, and shape data are collected every 0.1 s. This shape information is then transformed in the same reference system as the anatomy through a rigid registration. The regularization is finally computed according to Equation (7), through which the balance of forces is imposed: Here, fint, fext, fcorrection are respectively the internal, external and correction forces of the beam. In this first study, the vessel walls are considered rigid and the effect of blood flow is neglected: fext are therefore only constituted by the contact forces, computed as explained in Section 3.2. The correction term fcorrection is calculated by weighting the difference between the position of the devices computed by the simulation xsim and the position measured by the sensors xFBG by a constant α. We use α = 0.8, which was tuned experimentally. Details on the accuracy of the reconstructed shape are reported in Section 6.1. Visualization The mixed view superimposes the predicted virtual shape directly onto the fluoroscopic image. This choice was made to make our system more intuitive for clinicians. In clinical practice, indeed, the operator is guided by 2D fluoroscopic images. For this reason, even if the data computed by our system are 3D data, rather than displaying the model on a separate screen, the predictive model is shown on the same plane as the fluoroscopic images. Therefore, the 3D model of the virtual vascular anatomy augmented with the actual instruments' shape and the virtual catheter shape is registered on the live fluoroscopic images through rigid registration methods [22, 23, 24]. For the registration of device shapes into the 3D vascular model, two alternatives are possible: the use of the centreline of the vessels which can be extracted from the 3D model of the anatomy, or the use of a sensor, such as an optical tracker, placed at the proximal end of the device. This makes the system less intrusive since it does not require additional equipment and does not modify the operating setup to which the clinician is accustomed. EXPERIMENTAL SETUP In this section, all the technical characteristics of our framework are reported. Regarding the optical fibre, FBGS International (Geel, Belgium) commercial system was used. The fibre is placed inside a protective Nitinol tube with an external diameter of 0.3 mm. FBG data are collected by means of a laptop placed close to the operative table, and streamed through TCP protocol to a remote computer in which the simulation is performed. Both the local and the remote computers are equipped with an IntelCore i7 processor running at 3.40 GHz. The complete setup of the operating room, including our instrumentation, is shown in Figure 4. The usability of our framework is proven during renal artery cannulation. In this first study, an anatomical rigid 3D‐printed vascular network was adopted and the effect of blood flow was neglected. To evaluate the robustness and usefulness of the system, a validation study (Section 5.2) and a user study (Section 5.3) were performed. Operating room setup. The phantom (a) is inserted inside a mannequin (b), the laptop and the instrument to elaborate optical fibre data are placed far from the insertion point of the instruments (not visible in the image) and the screen (c) in which the predictive mixed reality images are shown is placed to the left of the operating table. The simulated environment The simulation was built on top of SOFA framework [25], to which the optimizations reported in Section 3.2.1 were added. The adopted physical and geometrical characteristics of the models are reported below. The catheter considered in this work is a Cobra catheter, suited for navigating abdominal aorta transversal vessels. The undeformed shape of the catheter into which the optical fibre was inserted was used as the rest shape of the model. An external diameter of 1.7 mm, an internal diameter of 1.4 mm and a length of 430 mm were measured. Special attention was paid to the catheter tip, characterized by a lower stiffness. After a fine‐tuning of the parameters, a Young's Modulus of 3.4 and 0.7 MPa were found for respectively the body and the tip of the catheter. Concerning the guidewire, a straight device with a diameter of 0.24 mm was chosen to be inserted inside the Cobra catheter. A length of 400 mm was adopted and a 500 MPa Young's Modulus was found through a bending test. Validation study During the cannulation, the instruments are manipulated in order to place the catheter's tip at the entrance of the vessels to be reached, which in this case are the right and left renal arteries. These vessels have a diameter of 6 mm: to provide support during the procedure, the predictive model must produce a prediction whose error is lower than this value. For this reason, the accuracy of the model was tested by performing multiple insertions in a rigid 3D‐printed phantom reproducing a human abdominal aorta anatomy (Figure 5a,g) and comparing the catheter shape predicted by our system (Figure 5 in green) with the real catheter shape obtained after guidewire withdrawal (Figure 5 in purple). Validation study. The guidewire and the catheter equipped with the optical fibre are inserted inside a phantom in the right (top row) and left (bottom row) iliac artery. The system, besides the real instruments' shape and position ((b) and (h) in purple), shows the predicted catheter shape after guidewire withdrawal ((b) and (h) in green). (c–f,i–n) The comparison between the predicted catheter shape computed by our system (in green) and the real catheter shape obtained after guidewire withdrawal (in purple) and acquired through the optical fibre system. A mean 3D error of 2.4 ± 1.3 mm and 2D error of 1.1 ± 0.7 mm were measured between the two shapes. The experience involved the insertion of the guidewire and the catheter, equipped with the optical fibre, both in the right and in the left femoral artery and the navigation of the two instruments toward the origin of the renal arteries. During the navigation, the instruments were translated together and our system showed the current position of the instruments in real‐time, exploiting the data collected through the optical fibre (Section 4.1), and the predicted catheter shape after guidewire withdrawal (Figure 5b,h). User study The usefulness of the system was assessed through a user study in which four different subjects were involved: three non‐expert users and an experienced interventional radiologist. Each subject was introduced to the experimental setup composed of a rigid 3D‐printed phantom representing the anatomy of the abdominal aorta, inserted inside a mannequin as shown in Figure 4: only the access point, placed at the level of the femoral aorta was visible. The fluoroscopic images were shown on a screen placed in front of the operator, to reproduce as well as possible the setting of a real intervention. To avoid unnecessary X‐ray exposure for the users, synthetic fluoroscopic images were generated using the Beer–Lambert law [26] and the 3D mesh of the organs of the same subject from whom the anatomy of the abdominal aorta was obtained. The catheter (containing the optical fibre) and the guidewire manipulated during the experience are those described in Section 5.1. Each subject was required to perform a total of three experiences in three different scenarios without the possibility of repeating them to prevent a bias due to the learning curve. The aim of each experience was to perform a successful cannulation. At the beginning of the test, the catheter and the guidewire were inserted in the phantom's right renal artery. Then, manipulating both devices, the user performed the cannulation guided by fluoroscopic images containing for each trial different information: Classical fluoroscopic images, which do not show the anatomy of the vessels (Figure 6a). To reproduce as closely as possible the situation of real endovascular interventions, the 2D anatomy of the vessel was shown in the upper left part of the screen, to simulate the static road map typically used during these procedures. Classical fluoroscopic images, which do not show the anatomy of the vessels (Figure 6a). To reproduce as closely as possible the situation of real endovascular interventions, the 2D anatomy of the vessel was shown in the upper left part of the screen, to simulate the static road map typically used during these procedures. Fluoroscopic images augmented with the 3D model of the vessel (Figure 6b). Fluoroscopic images augmented with the 3D model of the vessel (Figure 6b). Fluoroscopic images augmented by our mixed reality system (Figure 6c). In this case, once the region close to the origin of the lateral vessels was reached, the predicted shape of the catheter (represented in green) started to be visualized. Fluoroscopic images augmented by our mixed reality system (Figure 6c). In this case, once the region close to the origin of the lateral vessels was reached, the predicted shape of the catheter (represented in green) started to be visualized. User study illustration. (a) First, the user is asked to navigate using the fluoroscopic image without contrast agent, seeing only a static roadmap in the upper left corner. (b) Second, the user navigates using a fluoroscopic image augmented with the 3D vessel anatomy. (c) Finally, the user tries to reach the target using our predictive mixed navigation approach. Each user performed the cannulation three times, each time with one of the three image guidance shown in these pictures. For all the subjects, the use of our system reduced the time required to perform the cannulation. During the experience, the user was observing the screen in which the movement of the combined devices was reconstructed in real‐time as the instruments were navigated in the phantom. The user was required to manipulate the devices and withdraw the guidewire at the right time to perform the cannulation. For each subject and for each experience, the time elapsed from the insertion until the successful cannulation was measured. If, after guidewire withdrawal, the target was not reached, the guidewire was re‐advanced inside the catheter and the guidewire withdrawal was repeated until the success of the operation. RESULTS AND DISCUSSION In this section, the results obtained in the validation and user study detailed respectively in Sections 5.2 and 5.3 will be reported. The catheter and guidewire described in Section 5.1 were adopted. To test the accuracy of the physical model illustrated in Section 3, the combined shapes obtained in the real world and in the simulated environment by inserting the guidewire inside the catheter were compared. A mean error of 2.0±0.9 mm was measured between the two shapes. It is important to notice that this combined model is obtained by computing the interaction of the two separate models described in Section 3.1, as explained in Section 3.2.2. Validation study For both the insertion in the right and in the left iliac artery, the navigation was stopped in four different locations and the accuracy of the system was measured by calculating the distance of the catheter shape predicted by our system and the real catheter shape obtained after guidewire withdrawal and measured through the optical fibres (Figure 5c–f,i–n). Since the predicted catheter shape was only shown in the region in which the renal arteries originate, only device positions close to this region were considered. A total of eight different locations were evaluated and are shown in Figure 5. A mean 3D error of 2.4 ± 1.3 mm was found, with a mean of 1.3 ± 0.6 mm for the navigation from the right iliac artery to the right renal artery and 3.5 ± 0.8 mm from the left iliac artery toward the left renal artery. Since the 3D model is superimposed on 2D fluoroscopic images, also the 2D error on the same plane as the fluoroscopic images was computed. A total mean 2D error of 1.1 ± 0.7 mm was found, with a value of 0.7 ± 0.5 mm for the right navigation and 1.4 ± 0.6 mm for the left one. The values obtained for each test are reported in Table 1. TABLE 1Validation study—Errors between the catheter shape computed by our system and the real catheter shape obtained after guidewire withdrawal in the locations shown in Figure 5. Error (mm) Error (mm) Right insertion 3D 2D Left insertion 3D 2D Loc. c1.90.5Loc. i2.51.2Loc. d0.70.4Loc. l3.52.4Loc. e0.90.5Loc. m4.41.3Loc. f1.81.5Loc. n3.50.9Mean (mm)1.3 ±0.60.7±0.5Mean (mm)3.5 ±0.81.4 ±0.6Total mean2.4 ± 1.3Total mean1.1 ±0.73D error (mm)2D error (mm) Validation study—Errors between the catheter shape computed by our system and the real catheter shape obtained after guidewire withdrawal in the locations shown in Figure 5. We believe that the higher errors found in the navigation from the left iliac artery toward the left renal artery are due to larger uncertainties in the shape reconstructed by the FBG sensor: since the phantom is rigid and the interior surface is non‐smooth, the tip of the fibre can get stuck in some regions during the navigation. This produces peaks in the bending of the fibre, with consequent deterioration of shape reconstruction accuracy. User study The results of the user study are reported in Table 2. Timings are measured in seconds, the value in parenthesis corresponds to the number of failed attempts. TABLE 2User study results. For each user, the time required for the navigation is reported, while the number of failed attempts is shown in parentheses. User ID Fluoroscopic images only Fluoroscopic images anatomy Predictive mixed navigation Non‐expert 181 s52 s34 sNon‐expert 274 s70 s (2)34 sNon‐expert 3136 s (2)49 s38 sExpert29 s20 s17 s User study results. For each user, the time required for the navigation is reported, while the number of failed attempts is shown in parentheses. A reduction in the time required to perform the cannulation can be observed from this table. In particular, the use of our system leads to an average procedural time reduction of 56% compared with classical fluoroscopic imaging guidance and 30% compared with the guidance of fluoroscopic images augmented with the 3D model of the vessel. Furthermore, a differentiation can be observed between the procedural time required by the expert user and the non‐expert users, whose performance is levelled when using our system. CONCLUSION AND PERSPECTIVES In this work, a predictive mixed reality system to support clinicians during endovascular interventions, in particular during navigation and cannulation tasks, is developed. These procedures are guided through 2D‐fluoroscopy images in which the vessels are only visible upon contrast agent injection. Our predictive mixed reality system aims to help overcome this limitation by superimposing on the fluoroscopic images the information of the anatomy of the vessel, and augmenting the visual feedback with information about the outcome of the current action, before actually performing it. To the best of the authors' knowledge, this is the first time that such a system has been developed. In the current work, real‐time physical simulations are used together with optical fibre measurements to obtain a reliable 3D reconstruction of the current position and movement of the real devices on the fluoroscopic images. This shape, together with the physics‐based model, is then used to compute a prediction able to support the clinician intra‐operatively, especially during the cannulation. The reliability of the model was tested through a validation study in which the predicted shape of the catheter was compared to the real shape of the device after guidewire withdrawal on the same plane as the fluoroscopic images that guide clinicians during the intervention. A mean error of 1.1 ± 0.7 mm was found, which is sufficient to offer support to the intervention. This validation study was seconded by a user study that reported a reduced duration and a reduced number of trials to obtain a successful cannulation when using our system. Our future work will aim at making predictions more accurate and more realistic, by considering the effect of vessel wall deformation and blood flow. Improved visual aids will also be investigated. In the current system implementation, as clinicians are used to being guided by 2D‐fluoroscopic images, our prediction is shown in the same 2D view. However, in our framework, all the computation is performed in 3D, which we could exploit to provide further support for the operator by indicating whether the catheter needs to be rotated to position its tip at the vessel entrance. AUTHOR CONTRIBUTIONS Valentina Scarponi: Conceptualization; data curation; formal analysis; investigation; methodology; software; validation; visualization; writing—original draft; writing—review and editing. Juan Verde: Validation; writing—review and editing. Nazim Haouchine: Formal analysis; validation; visualization; writing—review and editing. Michel Duprez: Conceptualization; data curation; formal analysis; methodology; supervision; validation; writing—review and editing. Florent Nageotte: Conceptualization; funding acquisition; investigation; methodology; project administration; resources; supervision; writing—review and editing. Stéphane Cotin: Conceptualization; formal analysis; funding acquisition; methodology; project administration; resources; supervision; writing—original draft; writing—review and editing. CONFLICT OF INTEREST STATEMENT The authors declare no conflicts of interest. Supporting information Supplemental Video 1 ACKNOWLEDGEMENTS The authors would like to express their deepest appreciation to IHU's experimental platform and preclinical unit personnel. This work of the Interdisciplinary Thematic Institute HealthTech, as part of the ITI 2021‐2028 program of the University of Strasbourg, CNRS and Inserm, was supported by IdEx Unistra (ANR‐10‐IDEX‐0002) and SFRI (STRAT'US project, ANR‐20‐SFRI‐0012) under the framework of the French Investments for the Future Program and was supported by French state funds managed within the “Plan Investissements d'Avenir” and by the ANR (reference ANR‐10‐IAHU‐02). Scarponi, V. , Verde, J. , Haouchine, N. , Duprez, M. , Nageotte, F. , Cotin, S. : FBG‐driven simulation for virtual augmentation of fluoroscopic images during endovascular interventions. Healthc. Technol. Lett. 11, 392–401 (2024). 10.1049/htl2.12108 DATA AVAILABILITY STATEMENT Data sharing not applicable – no new data generated, or the article describes entirely theoretical research. REFERENCES Associated Data Supplementary Materials Supplemental Video 1 Data Availability Statement Data sharing not applicable – no new data generated, or the article describes entirely theoretical research.