BridgeSplat: Bidirectionally Coupled CT and Non-Rigid Gaussian Splatting for Deformable Intraoperative Surgical Navigation
Maximilian Fehrentz, Alexander Winkler, Thomas Heiliger, Nazim Haouchine, Christian Heiliger, Nassir Navab · 2025
Read BridgeSplat: Bidirectionally Coupled CT and Non-Rigid Gaussian Splatting for Deformable Intraoperative Surgical Navigation by Maximilian Fehrentz, Alexander Winkler, Thomas Heiliger, Nazim Haouchine, Christian Heiliger, Nassir Navab (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-18501.txt.
Introduction
Surgery guided by scans faces a gap: the scan is taken before the operation, but organs shift and bend during it. Maximilian Fehrentz and five co-authors present BridgeSplat to close that gap. Their 2025 paper couples live 3D reconstruction with the patient's prior CT scan. The live model uses 3D Gaussians, soft blobs that render a scene, tied to the CT mesh.
Each Gaussian is set relative to a triangle of the mesh, so the two stay aligned. Joint tuning adjusts both the Gaussians and the mesh bend, guided by the surgery video. The bend can then flow back to update the CT itself. Tests ran on pig surgeries and on a simulated human liver. The CT bent in sensible ways from a single camera view. The authors share code and data in public.
Catalog record
| CREATOR | Maximilian Fehrentz, Alexander Winkler, Thomas Heiliger, Nazim Haouchine, Christian Heiliger, Nassir Navab |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-18501
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BridgeSplat: Bidirectionally Coupled CT and Non-Rigid Gaussian Splatting for Deformable Intraoperative Surgical Navigation Maximilian Fehrentz, Alexander Winkler, Thomas Heiliger, Nazim Haouchine, Christian Heiliger, Nassir Navab Abstract We introduce BridgeSplat, a novel approach for deformable surgical navigation that couples intraoperative 3D reconstruction with preoperative CT data to bridge the gap between surgical video and volumetric patient data. Our method rigs 3D Gaussians to a CT mesh, enabling joint optimization of Gaussian parameters and mesh deformation through photometric supervision. By parametrizing each Gaussian relative to its parent mesh triangle, we enforce alignment between Gaussians and mesh and obtain deformations that can be propagated back to update the CT. We demonstrate BridgeSplat’s effectiveness on visceral pig surgeries and synthetic data of a human liver under simulation, showing sensible deformations of the preoperative CT on monocular RGB data. Code, data, and additional resources can be found at https://maxfehrentz.github.io/ct-informed-splatting/ . 1 Introduction Laparoscopic surgery has shown the potential to lower complication rates, shorten hospital stays, and be cost-effective while maintaining R0 resection rates and showing no adverse effects on 90-day mortality [ 3 ] . However, it also comes with challenges, as tactile feedback is limited, spatial understanding is demanding, handling is challenging, and learning curves are flat. To mitigate those issues, Augmented Reality (AR) has been proposed to overlay relevant information from preoperative scans with little interruption to the clinical workflow. The potential for AR-guided (minimally invasive) interventions in visceral surgery has been demonstrated for the liver [ 1 ] , kidney [ 10 ] , pancreas [ 21 ] , and spleen [ 18 ] . Although the results are promising, those methods either stop at a rigid registration or require interruption of the surgical workflow for surgeon interaction (e.g., outlining contours). Continuous non-rigid registration is still an open challenge. Thinking from first principles, we would like a system that can reconstruct the intraoperative scene in real time while also estimating the camera pose. Therefore, much effort has been directed at non-rigid SLAM for surgery [ 14 , 12 , 17 ] . Despite all the progress on SLAM, as concluded in [ 13 ] , tracking of the camera is still desirable, if not necessary, to move towards clinically feasible systems. For posed cameras in surgical scenes, methods capable of 4D reconstruction for dynamic photorealistic novel view synthesis have emerged. Those methods are grounded in NeRF [ 20 ] and 4D Gaussian Splatting (4DGS) [ 24 , 9 , 8 , 22 , 7 ] . Prior work has been done based on both NeRF and 4DGS to go beyond mere reconstruction and towards intelligence, tracking the surgical scene [ 4 , 6 ] . However, NeRF does not have an explicit representation, and 4DGS is usually not bound to a surface. Attempting a monocular single-view reconstruction commonly leads to artifacts like floaters in the view frustum that allow overfitting on the individual observations but do not guarantee a meaningful 3D reconstruction of tissue over time, despite high image reconstruction quality. In [ 19 ] , the authors propose to extract a preoperative 3D model, equip it with a biomechanical model, and perform an initial rigid registration based on an intraoperative point cloud acquired with stereo depth. To allow for a non-rigid registration, they track the registered structure and deform the biomechanical model, using optical flow. Also building on biomechanical models, the authors in [ 2 ] propose a CNN for non-rigid registration. Although some of the introduced methods leverage selected 3D information from preoperative scans, none of them is directly coupled to the preoperative scan, using the entire 3D prior for guiding the reconstruction and enabling direct deformation propagation to the CT. Contribution We deform a preoperative CT from an intraoperative monocular RGB video. By rigging Gaussians to a registered CT mesh, we constrain the ill-posed monocular reconstruction problem through a canonical space prior, resolving the depth ambiguity. This reframes the problem of non-rigid registration from a non-rigid reconstruction to a non-rigid tracking problem with the preoperative CT as template. Unlike existing 4D Gaussian Splatting methods, our approach is bound to patient-specific anatomy, and its deformations are directly coupled to the CT. 2 Method Figure 1: We initialize 3D Gaussians by rigging them onto the registered mesh, extracted from the preoperative CT. Given the first camera pose, we identify the visible Gaussians, derive the visible vertices, and subsample them to initialize a sparse deformation field. For all consecutive frames, we optimize for the mesh vertices, the deformation vectors, and the appearance properties of the 3D Gaussians. Deformations are propagated to the CT. 2.1 Problem Formulation We start with a preoperative mesh \mathbf{M} of the abdominal cavity, obtained from the preoperative CT. At the beginning of surgery, a rigid registration \mathbf{T}_{CT} is performed to align the CT (and thus \mathbf{M} ) with the patient. We then introduce a set of 3D Gaussians, which are rigged onto the registered mesh \mathbf{T}_{CT}\mathbf{M} . The 3D Gaussians have learnable parameters \boldsymbol{\Theta}_{t}\ =\{\boldsymbol{\mu}_{t},\;\boldsymbol{\alpha}_{t},\;\mathbf{\Sigma}_{t},\;\mathbf{c}_{t}\} : means \boldsymbol{\mu}_{t} , opacities \boldsymbol{\alpha}_{t} , covariance matrices \mathbf{\Sigma}_{t} , and spherical harmonic coefficients \mathbf{c}_{t} . Additionally, a deformation field \theta_{t} is defined to capture mesh deformations over time. During surgery, a tracked laparoscope provides images \mathbf{I}_{t} from known camera poses \mathbf{P}_{t} . Our main objective is to minimize a photometric energy term \mathbf{E}_{\mathrm{photo}} between the observed image \mathbf{I}_{t} and a rendered image \hat{\mathbf{I}}_{t} via 3D Gaussian Splatting of the deformed mesh and its rigged 3D Gaussians. Formally, we seek to solve: \argmin_{\theta_{t},\;\boldsymbol{\Theta}_{t}}\,\mathbf{E}_{\mathrm{photo}}\;=\;\argmin_{\theta_{t},\;\boldsymbol{\Theta}_{t}}\Bigl\|\,\mathbf{I}_{t}\;-\;\hat{\mathbf{I}}_{t}\!\bigl(\mathbf{P}_{t},\;\theta_{t}(\mathbf{T}_{CT}\mathbf{M}),\;\boldsymbol{\Theta}_{t}\bigr)\Bigr\|. (1) where \theta_{t} deforms the registered mesh \mathbf{T}_{CT}\mathbf{M} . 2.2 Coupling 3D Gaussians and Mesh Unlike conventional 3D Gaussian Splatting methods that treat the means \boldsymbol{\mu}_{t} and covariances \mathbf{\Sigma}_{t} as unconstrained parameters, we tightly anchor our Gaussians to the mesh, inspired by GaussianAvatars [ 11 ] . Specifically, each triangle f^{i} of the deformed mesh \theta_{t}(\mathbf{T}_{CT}\mathbf{M}) holds one or more Gaussians whose means are expressed via optimizable barycentric coordinates: \boldsymbol{\mu}_{t}^{i}\;=\;\sum_{k=1}^{3}\mathbf{b}_{t}^{i,k}\,\mathbf{v}_{t}^{i,k},\quad\text{with}\quad\sum_{k=1}^{3}\mathbf{b}_{t}^{i,k}\;=\;1,\quad\mathbf{b}_{t}^{i,k}\;\geq\;0, (2) where \mathbf{v}_{t}^{i,k}=\theta_{t}\bigl(\mathbf{v}_{\mathrm{init}}^{i,k}\bigr) are the vertices of triangle f^{i} after deformation, and \mathbf{b}_{t}^{i}=(b_{t}^{i,1},b_{t}^{i,2},b_{t}^{i,3}) are the barycentric coordinates for the i -th Gaussian. By enforcing nonnegative \mathbf{b}_{t}^{i,k} that sum to 1, the mean \boldsymbol{\mu}_{t}^{i} cannot leave the parent face. This is in contrast to GaussianAvatars, where Gaussians are allowed to wander off the surface for enhanced visual fidelity. In our case, however, we try to estimate mesh deformations from single-view monocular RGB input. Therefore, we have to ensure that what we render is closely related to what the mesh actually represents, otherwise, Gaussians can satisfy the photometric energy by roaming and stretching freely beyond their parent triangles without the mesh having to deform. For \mathbf{\Sigma}_{t}^{i} , we decompose the covariance into a rotation (orientation) and an anisotropic scale. In particular, we fix the Gaussian’s orientation \mathbf{q}_{t}^{i} to align with the triangle’s normal, thus keeping the Gaussian aligned with the surface of its parent triangle. The corresponding scale vector \mathbf{s}_{t}^{i} has only two freely optimizable components, the component in normal direction is fixed to a small constant. We further cap the remaining scales to keep them within a factor of the initial scale, which is computed from a mesh-dependent heuristic. This prevents the Gaussians from stretching significantly beyond their parent triangles. 2.3 Deformation Field As shown in Equation 2 , the