Rapid patient-specific neural networks for intraoperative X-ray to volume registration
Vivek Gopalakrishnan †, David-Dimitris Chlorogiannis, Andrew Abumoussa, Anna M. Larson, Nazim Haouchine, Darren B. Orbach, Sarah Frisken, Neel Dey †, Polina Golland † · 2025
Read Rapid patient-specific neural networks for intraoperative X-ray to volume registration by Vivek Gopalakrishnan †, David-Dimitris Chlorogiannis, Andrew Abumoussa, Anna M. Larson, Nazim Haouchine, Darren B. Orbach, Sarah Frisken, Neel Dey †, Polina Golland † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16309.txt.
Introduction
Vivek Gopalakrishnan and co-authors present xvr in 2025, a method that aligns 3D scans with live X-ray images during procedures. Surgeons need that alignment to navigate, but older methods need careful tuning per patient or large hand-labeled data sets tied to one body part. Their framework trains a fresh small network for each patient.
Physics-based simulation builds training data from the patient's own pre-procedure scan, so no manual labels are needed. A foundation model pre-trained on thousands of whole-body scans adapts to any body region in about five minutes of fine-tuning. In the largest real fluoroscopy test of its kind, the method reached high accuracy in seconds across body parts, scan types, and hospitals, beating rivals by a wide margin.
Catalog record
| CREATOR | Vivek Gopalakrishnan †, David-Dimitris Chlorogiannis, Andrew Abumoussa, Anna M. Larson, Nazim Haouchine, Darren B. Orbach, Sarah Frisken, Neel Dey †, Polina Golland † |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16309
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rapid patient-specific neural networks for intraoperative X-ray to volume registration Vivek Gopalakrishnan †, David-Dimitris Chlorogiannis, Andrew Abumoussa, Anna M. Larson, Nazim Haouchine, Darren B. Orbach, Sarah Frisken, Neel Dey †, Polina Golland † Abstract \lx@add@cssclass{ltx_runin}\theabstract Abstract Advanced navigation techniques in image-guided interventions and surgical robotics require the rapid and precise alignment of 3D preoperative volumes ( e.g ., CT, MRI) to 2D intraoperative images ( e.g ., X-ray fluoroscopy) [ 1 , 2 ] . However, existing 2D/3D registration methods fail to generalize across the broad spectrum of fluoroscopy-guided procedures: traditional intensity-based optimizers require careful hyperparameter tuning for each subject [ 3 , 4 ] , while deep learning approaches demand extensive manually labeled datasets and remain constrained to the specific anatomy on which they were trained [ 6 , 6 ] . To address these limitations, we present xvr , a self-supervised framework that combines patient-specific neural networks with gradient-based optimization for automatic 2D/3D registration. xvr leverages physics-based simulation to generate training data from a patient’s own preoperative scan, eliminating the need for manual annotation. We present a foundation model pretrained on thousands of whole-body scans, achieving patient-specific adaptation for any anatomical region in only 5 min 5\text{\,}\mathrm{min} of finetuning. In the largest evaluation of 2D/3D registration on real fluoroscopy to date, xvr achieves high accuracy in seconds across diverse anatomical structures, imaging modalities, and hospitals, improving upon the accuracy of existing methods by an order of magnitude. xvr makes pan-anatomical 2D/3D rigid registration accessible to broad clinical and research communities through open-source software at https://xvr.csail.mit.edu . Abstract. \@abstract Figure 1: xvr rapidly trains patient-specific neural networks that achieve accurate intraoperative 2D/3D registration while integrating into existing clinical workflows. a, Preoperative 3D imaging is routinely acquired before image-guided procedures. b, Clinical teams make diagnoses and preoperative plans from these scans, with planning times ranging from minutes to days depending on the intervention ( e.g ., stroke vs . radiotherapy). c, Preoperatively, we render synthetic X-rays from the patient’s 3D imaging using our differentiable renderer, whose geometry conforms with the radiologic nomenclature of commercial C-arms: rotations \alpha and \beta correspond to the left-right anterior oblique (LAO/RAO) and the craniocaudal (CRA/CAU) axes, respectively, and translation y corresponds to the source-to-object distance (SOD). d, We finetune patient-specific networks to regress the ground truth (g.t.) pose of these synthetic X-rays in just 5\text{\,}\mathrm{min} . e, Patient-specific networks perform accurate 2D/3D registration on live 2D X-rays to 3D preoperative scans in seconds. f, This enables numerous applications, including reprojection of 3D preoperative plans or anatomical targets onto intraoperative imaging or identification of shared structures across multiple X-ray views using epipolar geometry. 1 Main Each year, millions of clinical interventions are performed under real-time X-ray guidance [ 7 ] . While C-arm imaging devices enable noninvasive visualization of interventions from virtually any angle, X-rays are projection images that do not provide explicit depth information. This spatial ambiguity confounds the three-dimensional (3D) navigation of medical devices within the body, increasing the risk of suboptimal device deployment and intraoperative complications [ 8 , 9 ] . Although volumetric modalities such as CT and MRI are routinely acquired preoperatively, they are often unavailable during procedures due to their lengthy acquisition times and incompatibility with surgical workflows [ 10 ] . A promising alternative is to emulate volumetric image guidance by aligning two-dimensional (2D) intraoperative X-rays to 3D preoperative scans, enabling the localization of surgical devices relative to 3D anatomy. This capability makes 2D/3D registration critical to the development of next-generation image navigation and robotic surgery technologies [ 11 , 12 , 13 , 14 , 15 , 16 ] . Despite its broad potential, achieving reliable 2D/3D X-ray to volume registration across diverse specialties remains a challenge [ 1 ] . Conventional methods estimate a C-arm’s pose by combining iterative optimization with computational models of X-ray formation, searching for the synthetic X-ray rendered from the preoperative volume that most closely matches the real intraoperative X-ray [ 3 , 4 ] . Although accurate, iterative optimization is sensitive to initialization errors: if started even a few centimeters from the true pose, iterative solvers often converge to incorrect solutions [ 17 , 4 , 19 ] . To this end, numerous deep learning approaches have been developed to produce more accurate initializations, either by using keypoint estimators to identify paired anatomical landmarks in the 2D and 3D images [ 5 , 21 , 22 ] or by directly regressing the C-arm pose from an X-ray [ 23 , 24 , 25 ] . While typically more accurate than direct pose regression, supervised keypoint estimation methods require expert knowledge of reliably visible anatomical landmarks, as well as manual annotation of these structures for every new preoperative 3D scan [ 6 ] . Even semi-supervised approaches that transfer 3D landmarks from previous patients require a large training corpus of manually annotated scans [ 6 ] . We introduce xvr , a self-supervised framework for patient-specific 2D/3D rigid registration that generalizes to any patient, procedure, or pathology ( Fig. 1 ). Preoperatively, a network is trained to regress the poses of synthetic 2D X-rays rendered from randomly simulated virtual C-arms from the patient’s own preoperative scan. Intraoperatively, this network provides an accurate initial C-arm pose that can be rapidly refined by a gradient-based optimizer via differentiable X-ray rendering. Unlike previous (semi-)supervised methods, xvr leverages patient-specific simulation to generate unlimited synthetic X-rays with ground truth C-arm poses. To enable rapid deployment across diverse clinical scenarios, we pretrain a foundation model with synthetic X-rays generated from over 2,000 volumes with whole-body and [formula] C-arm pose coverage. Finetuning this model on a new patient reduces training time from hours [ 8 ] to only 5\text{\,}\mathrm{min} . Analysis across five datasets from different hospitals spanning multiple patient populations, anatomical structures, and interventional specialties establish three key findings. First, xvr outperforms previous (semi-)supervised deep learning and unsupervised iterative methods in both accuracy and robustness. Second, patient-specific finetuning eliminates the out-of-distribution failures that previous methods suffer when encountering patients not well-represented in their training sets, ensuring reliable performance for every individual. Third, xvr operates at intraoperative speeds with consistent accuracy across heterogeneous real-world clinical and surgical procedures. Figure 2: Training a foundation model for 2D/3D registration using physics-based differentiable rendering . a-d, Comparisons of real X-rays to synthetic images rendered from CT scans of the same patients using ground truth C-arm poses demonstrate the fidelity of xvr . e, Given a volume, we render synthetic X-rays \mathbf{I}_{n} and 2D segmentation masks \mathbf{S}_{n} from randomly sampled poses \mathbf{T}_{n} . f, Given an input synthetic X-ray, a pose regression network predicts pose \mathbf{\hat{T}}_{n} , from which we re-render predicted images \mathbf{\hat{I}}_{n} and segmentations \hat{\mathbf{S}}_{n} . A self-supervised pose regression loss compares all predicted quantities to their ground truth counterparts, which are known as they are simulated. g, Optimizing the pose regression loss requires a sufficiently parameterized network ( i.e ., ResNet34) and a continuous representation of \SE 3 ( i.e ., quaternion adjugate). Error bars represent 90% confidence intervals of held-out synthetic data. 2 Differentiable synthetic data synthesis To generate synthetic training data from a patient’s own preoperative imaging, we developed a differentiable model of X-ray image formation. Given a preoperative