Hybrid Foveated Path Tracing with Peripheral Gaussians for Immersive Anatomy

Constantin Kleinbeck , 1, 2, 3, 4, Luisa Theelke 1, 2, 3, 4, Hannah Schieber 1, 2, 3, 4, Ulrich Eck 1, Rüdiger von Eisenhart-Rothe 1, 2, 4, Daniel Roth 1, 2, 3, 4 · 2026

Read Hybrid Foveated Path Tracing with Peripheral Gaussians for Immersive Anatomy by Constantin Kleinbeck , 1, 2, 3, 4, Luisa Theelke 1, 2, 3, 4, Hannah Schieber 1, 2, 3, 4, Ulrich Eck 1, Rüdiger von Eisenhart-Rothe 1, 2, 4, Daniel Roth 1, 2, 3, 4 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22026.txt.

Introduction

Constantin Kleinbeck and co-authors render medical volumes fast enough for immersive use. Three-dimensional scans hold rich information about disease, but flat slices force clinicians to rebuild anatomy mentally. Direct volumetric path tracing in virtual reality gives true spatial understanding, yet it is computationally expensive.

The team's 2026 preprint mixes two techniques. Sharp path tracing covers the fovea, the center of vision where detail matters, while smooth Gaussian representations fill the periphery, where the eye notices less. That hybrid foveated approach targets immersive anatomy viewers that stay fluid on real hardware. Better spatial views could help surgeons plan operations and students learn anatomy from real patient data.

Catalog record

CREATORConstantin Kleinbeck , 1, 2, 3, 4, Luisa Theelke 1, 2, 3, 4, Hannah Schieber 1, 2, 3, 4, Ulrich Eck 1, Rüdiger von Eisenhart-Rothe 1, 2, 4, Daniel Roth 1, 2, 3, 4
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22026

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hybrid Foveated Path Tracing with Peripheral Gaussians for Immersive Anatomy Constantin Kleinbeck , 1, 2, 3, 4, Luisa Theelke 1, 2, 3, 4, Hannah Schieber 1, 2, 3, 4, Ulrich Eck 1, Rüdiger von Eisenhart-Rothe 1, 2, 4, Daniel Roth 1, 2, 3, 4 Abstract Volumetric medical imaging offers great potential for understanding complex pathologies. Yet, traditional 2D slices provide little support for interpreting spatial relationships, forcing users to mentally reconstruct anatomy into three dimensions. Direct volumetric path tracing and VR rendering can improve perception but are computationally expensive, while precomputed representations, like Gaussian Splatting, require planning ahead. Both approaches limit interactive use. We propose a hybrid rendering approach for high-quality, interactive, and immersive anatomical visualization. Our method combines streamed foveated path tracing with a lightweight Gaussian Splatting approximation of the periphery. The peripheral model generation is optimized with volume data and continuously refined using foveal renderings, enabling interactive updates. Depth-guided reprojection further improves robustness to latency and allows users to balance fidelity with refresh rate. We compare our method against direct path tracing and Gaussian Splatting. Our results highlight how their combination can preserve strengths in visual quality while re-generating the peripheral model in under a second, eliminating extensive preprocessing and approximations. This opens new options for interactive medical visualization. \vgtcinsertpkg \teaser Our approach combines path tracing for the foveated region with a rapidly generated Gaussian Splatting model for the periphery. This combination utilizes the strengths of both approaches and enables anatomy rendering at a higher quality than just path tracing while overcoming the limited interactivity of only using Gaussians. The left side shows stylized Gaussians with overlay path tracing, the right close-up views of the foveal region, without and with combined path tracing. \nocopyrightspace AR augmented reality AP average precision API application programming interface BEV bird eye view BRB Bottleneck residual block mBIoU mean Boundary Intersection over Union CAI computer-assisted intervention CE cross entropy CAD computer-aided design CNN convolutional neural network CT computed tomography CRF conditional random fields DPC dense prediction cells DLA deep layer aggregation DNN deep neural network DVR direct volume rendering DA domain adaption DR domain randomization FAT falling things FCN fully convolutional network FoV field of view FV front view FP False Positive FPN feature Pyramid network FN False Negative FMSS fast motion sickness scale GAN generative adversarial network GCN graph convolutional network 3DGS 3D Gaussian Splatting HMI Human-Machine-Interaction HMD Head Mounted Display IoU intersection over union IRB inverted residual bock IPQ igroup presence questionnaire KNN k-nearest-neighbor LiDAR light detection and ranging LSFE large scale feature extractor LLM large language model mAP mean average precision MC mismatch correction module mIoU mean intersection over union MIS Minimally Invasive Surgery MSDL Multi-Scale Dice Loss ML Machine Learning MLP multilayer perception mIoU mean Intersection over Union MRI magnetic resonance imaging NN neural network NDDS NVIDIA Deep Learning Data Synthesizer NOCS Normalized Object Coordiante Space NeRF Neural Radiance Fields NVISII NVIDIA Scene Imaging Interface NGP neural graphics primitives OR Operating Room PBR physically based rendering PSNR peak signal-to-noise ratio PnP Perspective-n-Point RV range view ROI region of interest BB residual basic block RAS robot-assisted surgery SPP samples per pixel SH spherical harmonics SGD stochastic gradient descent SDF signed distance function SfM structure-from-motion SAM Segment-Anything SUS system usability scale SSIM structural similarity index measure SfM structure from motion SLAM simultaneous localization and mapping TP True Positive TN True Negative thor The House Of inteRactions TSDF signed distance function VR Virtual Reality YCB Yale-CMU-Berkeley AR augmented reality ATE absolute trajectory error BVIP blind or visually impaired people CNN convolutional neural network c2f coarse-to-fine FoV field of view GAN generative adversarial network GCN graph convolutional Network GNN Graph Neural Network HMI Human-Machine-Interaction HMD head-mounted display MR mixed reality IoT internet of things LLFF Local Light Field Fusion BLEFF Blender Forward Facing LPIPS learned perceptual image patch similarity NeRF neural radiance fields NVS novel view synthesis MLP multilayer perceptron MRS Mixed Region Sampling OR Operating Room PBR physically based rendering PSNR peak signal-to-noise ratio PnP Perspective-n-Point SUS system usability scale SSIM similarity index measure SfM structure from motion SLAM simultaneous localization and mapping TP True Positive TN True Negative thor The House Of inteRactions UEQ User Experience Questionnaire VR virtual reality WHO World Health Organization XR extended reality YCB Yale-CMU-Berkeley YOLO you only look once Introduction keywords Virtual reality, Visualization, Reconstruction, Path tracing, Gaussian splatting, Computed tomography. Visualizing medical anatomy holds significant potential for education, diagnosis, treatment planning, and intervention [ 64 ] . Although clinical imaging modalities such as ct ( ct ) or mri ( mri ) inherently capture the three-dimensional structure of the body, most visualization workflows for diagnosis and planning still present these data as 2D sequential cross-sectional slices. Consequently, users are required to mentally reconstruct these 2D images into a 3D understanding of human anatomy. This cognitive transformation is time-consuming and challenging, especially for students, patients, and other non-experts [ 4 ] , and could be significantly alleviated by techniques that allow the 3D anatomy to be directly visualized and explored. Advances in rendering have pushed the boundaries of medical visualizations for many years, bringing more advanced ways to present anatomical data [ 6 ] . In particular, 3D visualization techniques have been adapted for anatomical data, aiming to present complex structures in a more intuitive and accessible way [ 64 ] . Approaches range from direct volume rendering [ 13 ] to volumetric path tracing [ 8 ] , improving visual quality, especially in rendering of lighting and material interactions. These, in turn, aid in understanding complex spatial relationships and have shown success in medical education [ 5 , 7 ] , as well as clinical analysis, and planning [ 50 , 12 ] . Recently, intermediate point-based representations like gs ( gs ) [ 27 ] have been used to improve rendering speed while sacrificing interactivity, enabling viewers to crop or adjust visibilities to focus on certain areas [ 45 , 29 ] . Each approach brings with it distinct trade-offs between realism, interactivity, and computational costs. Although 3D visualization techniques make anatomical structures perceptible, most medical applications still rely on 2D visualization, limiting depth perception and interactivity. Immersive displays such as vr ( vr ) headsets overcome these limitations, providing stereoscopic cues and head-tracked exploration that improve comprehension of complex anatomical relationships [ 38 ] . High-quality volumetric path tracing in vr , however, remains prohibitively expensive at the refresh rates and resolutions required for comfortable, low-latency immersion. Intermediate representations, such as polygon meshes or point clouds, can accelerate rendering, but they sacrifice fine anatomical detail and limit flexibility in modifying appearance interactively. Perceptual optimization strategies, particularly foveated rendering, exploit the eye’s nonuniform acuity to allocate rendering resources where they matter most [ 59 ] . Recent works further indicate that the improved depth understanding provided by vr is not affected by foveated rendering [ 28 ] . Yet, scaling these techniques to the real-time performance targets of modern, high-resolution vr displays remains a challenge [ 43 ] . To address this problem, we propose a hybrid rendering system for immersive anatomical visualization that balances quality, speed, and interactivity. Our approach combines streamed high-quality, volumetric path tracing for the foveal region with a rapidly

Read the complete text