Toward Reliable AR-Guided Surgical Navigation: Interactive Deformation Modeling with Data-Driven Biomechanics and Prompts Thanks: This work was supported in part by the Hong Kong Innovation and Technology Fund (Project No. GHP/167/22SZ), in part by the Research Grants Council of Hong Kong Special Administrative Region, China (Projects No. N_CUHK410/23 and No. T45-401/22-N), in part by the National Natural Science Foundation of China (Project No. 62322318), in part by the General Research Fund of Hong Kong Research Grants Council (Project No. 15218521), and in part by the Shenzhen Science and Technology Program (Project No. SGDX20230116092200001). We sincerely thank Dr. Kai Wang from the Department of General Surgery at Nanfang Hospital in Guangzhou, China, for his clinical support. Corresponding authors: Yingfang Fan and Qi Dou Thanks: Zheng Han, Jialun Pei, and Qi Dou are with the Department of Computer Science and Engineering, The Chinese University of Hong Kong, HKSAR, China. (e-mail: zheng-han@link.cuhk.edu.hk). Thanks: Jun Zhou and Jing Qin are with the Center of Smart Health, School of Nursing, The Hong Kong Polytechnic University, HKSAR, China. Thanks: Yingfang Fan is with the Department of Hepatobiliary Surgery, The Third Affiliated Hospital of Southern Medical University, Guangzhou, China. Thanks: DOI: 10.1109/TMI.2025.3577759 Zheng Han, Jun Zhou, Jialun Pei, Jing Qin, Yingfang Fan Abstract In augmented reality (AR)-guided surgical navigation, preoperative organ models are superimposed onto the patient’s intraoperative anatomy to visualize critical structures such as vessels and tumors. Accurate deformation modeling is essential to maintain the reliability of AR overlays by ensuring alignment between preoperative models and the dynamically changing anatomy. Although the finite element method (FEM) offers physically plausible modeling, its high computational cost limits intraoperative applicability. Moreover, existing algorithms often fail to handle large anatomical changes, such as those induced by pneumoperitoneum or ligament dissection, leading to inaccurate anatomical correspondences and compromised AR guidance. To address these challenges, we propose a data-driven biomechanics algorithm that preserves FEM-level accuracy while improving computational efficiency. In addition, we introduce a novel human-in-the-loop mechanism into the deformation modeling process. This enables surgeons to interactively provide prompts to correct anatomical misalignments, thereby incorporating clinical expertise and allowing the model to adapt dynamically to complex surgical scenarios. Experiments on a publicly available dataset demonstrate that our algorithm achieves a mean target registration error of 3.42 mm. Incorporating surgeon prompts through the interactive framework further reduces the error to 2.78 mm, surpassing state-of-the-art methods in volumetric accuracy. These results highlight the ability of our framework to deliver efficient and accurate deformation modeling while enhancing surgeon-algorithm collaboration, paving the way for safer and more reliable computer-assisted surgeries. \FAILED \FAILED I Introduction Augmented Reality (AR) has the potential to transform surgical procedures by intuitively visualizing critical anatomical structures within the surgical field. This is achieved by superimposing virtual organ models—typically reconstructed from preoperative computed tomography (CT) or magnetic resonance imaging (MRI) scans—onto the patient’s intraoperative anatomy [ 1 ] . As illustrated in Fig. 1 (top), AR overlays vascular structures and tumor locations onto live endoscopic views, thereby providing surgeons with spatially contextualized guidance during complex procedures such as tumor resections or vessel preservation. Such enhanced visualization is especially critical in minimally invasive surgery, where the field of view is narrow and tactile feedback is limited. Accurate alignment between virtual models and intraoperative anatomy enables the identification of structures that are occluded or difficult to discern from endoscopic views alone. Fig. 1: Interactive deformation modeling for AR-guided surgical navigation. The proposed data-driven biomechanics algorithm (BiomPINN-PBMs) deforms the preoperative 3D model based on intraoperative imaging and presents the result for surgeon evaluation. This human-in-the-loop process allows the surgeon to either accept the deformation for AR overlay or provide corrective prompts to further refine the model. A video demonstration of the interactive process is available at: https://github.com/med-air/Interactive-registration . However, preoperative models represent static anatomy captured days or weeks prior to surgery and therefore fail to reflect the dynamic intraoperative environment. Factors such as pneumoperitoneum pressure, instrument interaction, and ligament dissection can induce substantial deformation of soft tissues, leading to spatial misalignment between the virtual model and actual anatomy [ 2 ] . Without addressing these deformations, AR overlays may become inaccurate or misleading, potentially compromising surgical safety and outcomes. Patient-specific biomechanical models (PBMs) have been widely adopted to compensate for intraoperative deformations, offering physically plausible representations of soft tissue viscoelasticity [ 3 ] . Grounded in continuum mechanics, PBMs typically employ finite element (FE) methods to simulate organ deformation under prescribed material properties, boundary conditions, and loading scenarios [ 4 ] . However, acquiring comprehensive input data for FE simulations in the surgical setting is often infeasible. Key parameters—such as contact forces and torques—are difficult to measure due to their invasive nature and the lack of integrated sensors in surgical instruments [ 5 ] . As a result, intraoperative information is usually limited to visual observations, such as laparoscopic images or reconstructed organ surfaces, which only capture partial and superficial aspects of deformation. Inferring full volumetric deformation from such sparse and incomplete surface cues remains a fundamental challenge to achieving anatomically accurate AR navigation. To address these practical limitations, image-driven surface-based PBMs have been developed [ 6 , 7 , 8 , 9 , 10 , 11 ] . These methods infer organ deformation by leveraging sparse surface displacements derived from geometric correspondences between the preoperative model and intraoperative observations, as illustrated in Fig. 2 . The observed displacements are imposed as Dirichlet boundary conditions to drive the surface alignment [ 7 ] , while FE-based regularizers are introduced to ensure biomechanical plausibility beyond observed regions. A critical component of this formulation is the choice of regularization weight, which balances geometric fidelity against physical plausibility. Overly stiff regularization may hinder surface alignment, whereas overly flexible settings can lead to non-physical deformations. To achieve an optimal balance, iterative optimization schemes are employed, gradually tuning the regularization weight to minimize alignment error while maintaining deformation coherence [ 9 ] . However, such iterative tuning is computationally expensive, as each iteration requires solving a large-scale linear system governed by the stiffness matrix. Furthermore, the inherently sequential nature of these solvers limits their ability to leverage parallel computation on modern hardware architectures, further constraining their applicability in time-sensitive surgical scenarios [ 12 ] . Fig. 2: Illustration of the surface-based deformation modeling process. After rigid registration, sparse correspondences with the intraoperative surface are established (left). These local displacements are biomechanically propagated to estimate the full volumetric deformation field, including unobserved regions (right). In parallel, deep learning–based models have been explored as alternatives to PBMs, aiming to approximate biomechanical deformation with lower computational costs [ 13 ] . Early approaches [ 14 , 5 ] discretized organs into voxelized 3D grids and employed convolutional neural networks (e.g., U-Net) to regress displacement fields. While effective in capturing global deformation patterns, these voxel-based approaches are constrained by limited resolution and voxelization artifacts. To overcome these issues, more recent works have adopted flexible geometric representations, such as point clouds [ 15 ] and implicit neural fields [ 16 , 17 ] , which better preserve fine anatomical details and enable richer deformation modeling. However, these representations often lack explicit spatial coupling among neighboring points, making it difficult to propagate local displacements into globally consistent volumetric deformation. Consequently, predicted strain fields frequently exhibit physical inconsistencies, including non-smoothness and discontinuities [ 18 ] . Another major challenge is data availability. Ground truth volumetric deformations are rarely obtainable in surgical settings, as acquiring them would require repeated intraoperative CT or MRI scans—procedures that are both ethically and logistically impractical. As a result, most data-driven models are trained on synthetic datasets generated through forward FE simulations. This offline training–testing paradigm introduces a domain shift: models trained on simulated anatomies and deformation patterns often fail to generalize to real, patient-specific conditions. Thus, despite their computational efficiency, current data-driven approaches remain constrained in clinical deployment by distribution shifts between training and intraoperative data and by insufficient guarantees of physical plausibility. Additionally, complex surgical scenarios involving large anatomical deformations pose inherent challenges for surface-based deformation modeling methods, including both physics-based and data-driven approaches. These methods rely on establishing accurate correspondences between the preoperative model and intraoperative anatomy—a critical prerequisite for driving deformation using sparse surface displacements. However, under conditions of severe morphological change, such correspondences often become unreliable. For example, in liver surgery, resection of the falciform ligament can induce structural alterations, such as pronounced separation between the left and right lobes, that are not reflected in the preoperative model. This anatomical discrepancy, compounded by the lack of distinctive geometric features on the organ surface, further hampers the establishment of reliable correspondences. As a result, such challenging scenarios often exceed the capabilities of fully automated pipelines, leading to inaccurate alignments and compromising the reliability of AR-guided navigation. These limitations underscore the urgent need for an adaptive framework that incorporates human expertise to correct deformation results when automated methods fall short [ 19 ] . In summary, we identify three key challenges in intraoperative deformation modeling: (1) the high computational burden associated with biomechanical models; (2) the lack of guaranteed physical plausibility in data-driven predictions, compounded by domain shift when applying synthetic-data-trained models to real surgical settings; and (3) the limited effectiveness of existing methods under scenarios involving large anatomical deformations. In response to these challenges, we present the following contributions: • Biomechanically-Constrained Neural Network: We introduce BiomPINN , a P hysics- I ntegrated N eural N etwork that incorporates Biom echanical priors to enforce physical laws and biomechanical plausibility. BiomPINN adopts a per-instance optimization paradigm , eliminating the need for offline training and avoiding distribution shift between synthetic and intraoperative data. • BiomPINN-PBMs Integration: We present a hybrid approach that integrates BiomPINN with PBMs, preserving FE–level deformation accuracy while significantly reducing computational cost through one-pass optimization. • Interactive Deformation Modeling Framework: We develop an interactive, human-in-the-loop framework tailored for AR-guided surgery. When automated predictions are unsatisfactory, surgeons can provide intuitive corrective prompts, which are incorporated into the modeling process to dynamically adapt to complex intraoperative scenarios. • Superior Performance: Our approach demonstrates superior volumetric deformation accuracy compared to state-of-the-art methods, validated on a publicly available phantom dataset. II Preliminary Information Given two organ surfaces—a source point cloud \mathcal{X}=\{\mathbf{x}_{i}\in\mathbb{R}^{3}\mid i=1,\dots,n\} , representing the complete geometric structure of the organ surface reconstructed from preoperative CT, and a target point cloud \mathcal{Y}=\{\mathbf{y}_{i}\in\mathbb{R}^{3}\mid i=1,\dots,m\} , representing an incomplete organ surface reconstructed from intraoperative imaging—the objective is to determine a deformation field \mathcal{U}=\{\mathbf{u}_{i}\in\mathbb{R}^{3}\mid i=1,\dots,n\} that warps the preoperative model \mathcal{X} to align with the intraoperative observations \mathcal{Y} . To ensure that the deformation of \mathcal{X} follows physically plausible behavior, a biomechanical model is introduced as a global geometric constraint to enforce volumetric consistency. II-A Biomechanical Model Representation The biomechanical model is defined as a volumetric tetrahedral mesh \Omega , constructed based on the surface node set \mathcal{X} , with internal nodes automatically generated during the tetrahedralization process. The mesh \Omega consists of n_{v} nodes, denoted as \mathcal{X}_{\Omega}=\{\mathbf{x}_{\Omega,i}\in\mathbb{R}^{3}\mid i=1,\dots,n_{v}\}\in\mathbb{R}^{3n_{v}}. The point set \mathcal{X} is a subset of the nodes in \Omega and forms a closed surface mesh, denoted by \partial\Omega . When surface tractions are applied on the boundary \partial\Omega , the resulting displacements within the volumetric domain \Omega , denoted as \mathcal{U}_{\Omega}=\{\mathbf{u}_{\Omega,i}\in\mathbb{R}^{3}\mid i=1,\dots,n_{v}\}, can be obtained by solving the static equilibrium equation: \mathbf{K}\mathbf{u}_{\Omega}=\mathbf{f}, (1) where \mathbf{u}_{\Omega}\in\mathbb{R}^{3n_{v}} is the global displacement vector obtained by stacking all nodal displacements in \mathcal{U}_{\Omega} , and \mathbf{K}\in\mathbb{R}^{3n_{v}\times 3n_{v}} is the global stiffness matrix, which encodes the organ’s geometry and material properties as defined by the FE model. The force vector \mathbf{f} represents the nodal forces assembled from the applied surface tractions on \partial\Omega . To relate the volumetric displacements to the surface boundary displacements, an interpolation matrix \Phi\in\mathbb{R}^{3n\times 3n_{v}} is introduced: \mathbf{u}=\Phi\,\mathbf{u}_{\Omega}, (2) where \mathbf{u}\in\mathbb{R}^{3n} is the vectorized displacement of the surface nodes, obtained by stacking the deformation field \mathcal{U} . Following the node ordering strategy in [ 9 ] , the nodes in \mathcal{X}_{\Omega} that exclusively belong to the surface \partial\Omega are appended to the end of the node list. Consequently, the interpolation matrix \Phi takes the following block form: \Phi=\begin{bmatrix}I_{3n\times 3n}&0\end{bmatrix}. (3) Here, the identity block extracts the displacements of surface nodes (assumed to be ordered first), while the zero block corresponds to the remaining internal nodes. This matrix provides a linear mapping from the full volumetric displacement field to surface displacements, enabling the use of the biomechanical model as a constraint in surface-based deformation modeling (see Fig. 3 for illustration). Fig. 3: Illustration of the interpolation matrix \Phi that maps volumetric displacements to boundary displacements. The tetrahedral mesh \Omega is partitioned into boundary nodes \mathcal{X}\subset\partial\Omega and interior nodes, yielding a block matrix structure. A cross-sectional view reveals the internal mesh structure, while the surface mesh \partial\Omega retains only the boundary points. II-B Stiffness Matrix The biomechanical model’s geometry and material properties are encoded in the stiffness matrix \mathbf{K} of the FE model. As shown in Eq. ( 1 ), the global stiffness matrix \mathbf{K} establishes the relationship between the nodal displacement vector \mathbf{u}_{\Omega} and the force vector \mathbf{f} . For a linearly elastic tissue model [ 20 ] , the mechanical behavior is characterized by two material constants: Young’s modulus ( E ) and Poisson’s ratio ( \nu ), which define the constitutive relationship between stress and strain. In practice, the global stiffness matrix \mathbf{K} is assembled from the local contributions of all elements in the tetrahedral mesh. For an individual element e , the local stiffness matrix \mathbf{K}_{e} is given by: \mathbf{K}_{e}=\int_{V_{e}}\mathbf{B}_{e}^{\mathsf{T}}\mathbf{D}_{e}\mathbf{B}_{e}\,dV, (4) where V_{e} is the volume of the element, \mathbf{B}_{e} is the strain-displacement matrix derived from the element shape functions, and \mathbf{D}_{e} is the material stiffness matrix, which encodes the linear elastic constitutive law. For isotropic materials, \mathbf{D}_{e}\in\mathbb{R}^{6\times 6} is defined as: \mathbf{D}_{e}=\begin{bmatrix}\lambda+2\mu&\lambda&\lambda&0&0&0\\ \lambda&\lambda+2\mu&\lambda&0&0&0\\ \lambda&\lambda&\lambda+2\mu&0&0&0\\ 0&0&0&\mu&0&0\\ 0&0&0&0&\mu&0\\ 0&0&0&0&0&\mu\end{bmatrix}, (5) where \lambda and \mu are Lamé parameters, computed from E and \nu via \lambda=\frac{E\nu}{(1+\nu)(1-2\nu)} , \mu=\frac{E}{2(1+\nu)} . The global stiffness matrix is then obtained by summing the local contributions over all elements through standard finite element assembly: \mathbf{K}=\sum_{e}\mathbf{A}_{e}^{\mathsf{T}}\mathbf{K}_{e}\mathbf{A}_{e}, (6) where \mathbf{A}_{e} is the element-to-global mapping operator. The quadratic form \mathbf{u}_{\Omega}^{\mathsf{T}}\mathbf{K}\mathbf{u}_{\Omega} quantifies the total strain energy within the deformable domain. The stiffness matrix \mathbf{K} encodes the tissue’s mechanical properties and governs the propagation of stress and strain, promoting physically coherent deformations throughout the organ. II-C Forward Problem In FE modeling, the forward problem refers to computing the displacements of free (unconstrained) nodes within the volumetric domain \Omega , given prescribed displacements at constrained boundary nodes. When the constraints span the entire model boundary \partial\Omega , this process represents mapping known boundary displacements to interior deformation, corresponding to the reverse direction illustrated in Fig. 3 . Let the global displacement vector of the volumetric mesh \Omega be denoted as \mathbf{u}_{\Omega}\in\mathbb{R}^{3n_{v}} , which is partitioned into: \mathbf{u}_{\Omega}=\begin{bmatrix}\mathbf{u}_{f}\\ \mathbf{u}_{c}\end{bmatrix}, (7) where \mathbf{u}_{f}\in\mathbb{R}^{3n_{f}} denotes the displacements of the free nodes, and \mathbf{u}_{c}\in\mathbb{R}^{3n_{c}} denotes the displacements of the constrained nodes, with n_{f}+n_{c}=n_{v} . Substituting this partition into the static equilibrium equation in Eq. ( 1 ) yields the following block matrix system: \begin{bmatrix}\mathbf{K}_{ff}&\mathbf{K}_{fc}\\ \mathbf{K}_{cf}&\mathbf{K}_{cc}\end{bmatrix}\begin{bmatrix}\mathbf{u}_{f}\\ \mathbf{u}_{c}\end{bmatrix}=\begin{bmatrix}\mathbf{f}_{f}\\ \mathbf{f}_{c}\end{bmatrix}, (8) where the stiffness submatrices \mathbf{K}_{ff} , \mathbf{K}_{fc} , etc., represent the interactions among free and constrained nodes, and \mathbf{f}_{f} , \mathbf{f}_{c} are the corresponding nodal force vectors. Assuming that no external force is applied to the free nodes (i.e., \mathbf{f}_{f}=\mathbf{0} ), the displacement of free nodes is computed by solving the reduced system: \mathbf{u}_{f}=-\mathbf{K}_{ff}^{-1}\mathbf{K}_{fc}\mathbf{u}_{c}. (9) The complete displacement vector \mathbf{u}_{\Omega} is then reconstructed by concatenating the free and constrained components as described in Eq. ( 7 ). II-D Correspondence Matrix The correspondence matrix \mathbf{C}\in\mathbb{R}^{n\times m} establishes the mapping between points in the preoperative organ model \mathcal{X}=\{\mathbf{x}_{i}\}_{i=1}^{n} and the intraoperative point cloud \mathcal{Y}=\{\mathbf{y}_{j}\}_{j=1}^{m} . It encodes the geometric relationships between the two surfaces, facilitating their alignment and enabling the estimation of the deformation field on observable regions. Depending on the formulation, \mathbf{C} can represent either probabilistic or deterministic correspondences. In its probabilistic form, \mathbf{C} is a soft correspondence matrix where each entry \mathbf{C}_{i,j}\in[0,1] indicates the likelihood that \mathbf{x}_{i}\in\mathcal{X} corresponds to \mathbf{y}_{j}\in\mathcal{Y} . In contrast, a binary matrix \mathbf{C}\in\{0,1\}^{n\times m} represents deterministic one-to-one matching, where \mathbf{C}_{i,j}=1 denotes a strict correspondence between \mathbf{x}_{i} and \mathbf{y}_{j} . In this study, we adopt the binary formulation to achieve precise correspondence. To construct the binary matrix \mathbf{C} , a soft correspondence matrix is first predicted using either learning-based or geometry-based approaches [ 19 ] . The soft matrix is then converted into a binary form by solving a linear assignment problem (LAP), enforcing strict one-to-one matching: \sum_{j}\mathbf{C}_{i,j}=1,\forall i;\ \sum_{i}\mathbf{C}_{i,j}=1,\forall j;\ \text{and\ }\mathbf{C}_{i,j}\in\{0,1\}. (10) However, in the presence of partial overlaps between \mathcal{X} and \mathcal{Y} , such strict matching may yield spurious correspondences due to missing regions or noise. To mitigate this, additional outlier pruning is performed after LAP solving to remove low-confidence matches and refine the final matrix \mathbf{C} . Notably, this approach does not rely on any pre-existing rigid registration between the two point sets. Alternatively, when a reliable rigid alignment \mathcal{R} has already been obtained, correspondences can be directly inferred via mutual nearest neighbors with geometric filtering: \mathbf{C}_{i,j}=\begin{cases}1,&\text{NN}(\mathcal{R}(\mathbf{x}_{i}),\mathcal{Y})=\mathbf{y}_{j},\text{NN}(\mathcal{X},\mathcal{R}^{-1}(\mathbf{y}_{j}))=\mathbf{x}_{i},\\ &\quad\quad\quad\text{and }\|\mathcal{R}(\mathbf{x}_{i})-\mathbf{y}_{j}\|_{2}