Supercharging Thermal Gaussian Splatting with Depth Estimation
Manoj Biswanath 1, Chenxin Cai 2, Hannah Schieber 3, Daniel Roth 3, Benjamin Busam 1 · 2026
Read Supercharging Thermal Gaussian Splatting with Depth Estimation by Manoj Biswanath 1, Chenxin Cai 2, Hannah Schieber 3, Daniel Roth 3, Benjamin Busam 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30328.txt.
Introduction
Manoj Biswanath and four co-authors propose Thermal-to-Depth Gaussian Splatting, deriving radiance fields from thermal images and depth estimation. Fusing multimodal data slows synthesis, while a single thermal modality promises speed. Their method uses only thermal images plus depth estimation. It beats the MSMG baseline on test datasets, improving quality metrics by about one percent and by tiny fractions of a percent on the other two. It cuts training time by nearly 13 minutes, more than half. Thermal fields suit surveillance, search and rescue, and industrial inspection.
Catalog record
| CREATOR | Manoj Biswanath 1, Chenxin Cai 2, Hannah Schieber 3, Daniel Roth 3, Benjamin Busam 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36544 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30328
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Supercharging Thermal Gaussian Splatting with Depth Estimation Manoj Biswanath 1, Chenxin Cai 2, Hannah Schieber 3, Daniel Roth 3, Benjamin Busam 1 Abstract Efficient and robust 3D scene representation is crucial in autonomous driving, robotics, and related fields. While RGB images provide valuable content for 3D reconstruction, other modalities like thermal or depth can enable additional information on the environment. Lately, novel view synthesis methods like 3D Gaussian Splatting have started using multiple modalities to further boost their performance. But fusing or combining multimodal data can make the process slower and can bring in additional challenges. Therefore, our project aims to use single modality based on thermal infrared domain, by removing the reliance on visible light as much as possible. This single modality can be expected to be faster as it does not rely on multimodal data. We propose a method, tdg ( tdg ) , that uses only thermal images and depth estimation in its architecture to derive the radiance fields. Our tdg method outperforms the MSMG (Multiple Single-Modal Gaussians) baseline in most cases on our test datasets, RGBT-Scenes and ThermalMix. On average, the rendering quality metrics such as lpips ( lpips ), ssim ( ssim ), and psnr ( psnr ) of tdg are 1.12%, 0.034%, and 0.01% better than the baseline MSMG values. It also reduces the training time significantly, by 12 mins 47 secs (55% improvement). Overall, our method is successful in deriving these thermal radiance fields, which can ultimately have several applications, such as identifying heat sources critical in surveillance, search or rescue operations, and industrial inspections where temperature is widely used to monitor machines. AR augmented reality AP average precision API application programming interface TDg Thermal-to-Depth Gaussian Splatting BEV bird eye view BRB Bottleneck residual block mBIoU mean Boundary Intersection over Union POI Point of Interest CAI computer-assisted intervention CE cross entropy CAD computer-aided design CNN convolutional neural network CRF conditional random fields DPC dense prediction cells DLA deep layer aggregation DNN deep neural network DA domain adaption DR domain randomization FAT falling things FCN fully convolutional network FoV field of view FV front view FP False Positive FPN feature Pyramid network FN False Negative FMSS fast motion sickness scale GAN generative adversarial network GCN graph convolutional network GS Gaussian Splatting GG Gaussian Grouping HMI Human-Machine-Interaction HMD Head Mounted Display IoU intersection over union IRB inverted residual bock IPQ igroup presence questionnaire KNN k-nearest-neighbor LiDAR light detection and ranging LSFE large scale feature extractor LLM large language model mAP mean average precision MC mismatch correction module mIoU mean intersection over union MIS Minimally Invasive Surgery MSDL Multi-Scale Dice Loss ML Machine Learning MLP multilayer perception mIoU mean Intersection over Union NN neural network NDDS NVIDIA Deep Learning Data Synthesizer NOCS Normalized Object Coordiante Space NeRF Neural Radiance Fields NVISII NVIDIA Scene Imaging Interface NGP Neural Graphics Primitives OR Operating Room PBR physically based rendering PSNR peak signal-to-noise ratio PnP Perspective-n-Point RV range view RoI region of interest BB residual basic block RAS robot-assisted surgery SPP spatial pyramid pooling SH spherical harmonics SGD stochastic gradient descent SDF signed distance field SfM Structure-from-Motion SAM Segment-Anything SUS system usability scale SSIM structural similarity index measure SLAM simultaneous localization and mapping TP True Positive TN True Negative thor The House Of inteRactions TSDF truncated signed distance function VR Virtual Reality YCB Yale-CMU-Berkeley AR augmented reality ATE absolute trajectory error BVIP blind or visually impaired people CNN convolutional neural network c2f coarse-to-fine FoV field of view GAN generative adversarial network GCN graph convolutional Network GNN Graph Neural Network HMI Human-Machine-Interaction HMD head-mounted display MR mixed reality IoT internet of things LLFF Local Light Field Fusion BLEFF Blender Forward Facing LPIPS learned perceptual image patch similarity NeRF neural radiance fields NVS novel view synthesis MLP multilayer perceptron MRS Mixed Region Sampling OR Operating Room PBR physically based rendering PSNR peak signal-to-noise ratio PnP Perspective-n-Point SUS system usability scale SLAM simultaneous localization and mapping TP True Positive TN True Negative thor The House Of inteRactions UEQ User Experience Questionnaire VR virtual reality WHO World Health Organization XR extended reality YCB Yale-CMU-Berkeley YOLO you only look once keywords Thermal imaging, Gaussian Splatting, Depth Estimation, Radiance fields, Novel view synthesis. 1 Introduction Robotics ( 7 ; 14 ) , autonomous driving ( 28 ) , virtual reality ( 22 ; 10 ) , construction industry ( 20 ) and similar fields demand highly efficient and robust 3D scene representation. In recent years, explicit rendering techniques, particularly nvs ( nvs ) methods such as gs ( gs ) ( 9 ) , received significant attention due to their exceptional rendering speed and visual quality ( 13 ) . Mainstream methods heavily rely on RGB images, whose performance deteriorates significantly in visually degraded environments, such as low-light, fog, smoke, or extreme weather conditions. This can make the direct application of RGB-designed gs frameworks highly challenging. Introducing other modalities alongside RGB, such as thermal, depth, or LiDAR data, can provide additional information about the environment, or can especially help in cases where drones and ground robots are often equipped with additional sensors, or only contain thermal and depth cameras, rather than RGB. Figure 1: Thermal images rendered from the gs model of our tdg method, compared against Ground Truth (GT) and baseline (MSMG). Infrared imaging cameras, which detect the inherent thermal radiation of objects, can provide a robust perception capability, independent of visible light. It is suitable in challenging environments, regardless of lighting and weather conditions. Thermal imaging converts temperature information into interpretable images. One common application of thermography is energy inspection of objects. Classically, this has been done by visually evaluating the images. But a direct 3D spatial reference is not established for the measured values. This deficiency becomes obvious for complex structures, when their images taken from different angles need to be fused, or the measured thermal radiations need to be stored in an object-related manner, for further processing. In such cases, creation of a thermal radiance field (as a 3D representation) can be helpful. It can provide us a 3D spatial reference of the temperature measurements. For example, one specific application could be energy inspection in buildings, where knowledge of both the inside and outside wall temperatures would be necessary to provide insights about the geometry and material properties of the walls. This knowledge could be acquired from the 3D representations. Further, these continuous scene representations can provide correct view-dependent effects such as reflections, translucency, etc., and can serve as either digital twins or an enrichment of already available digital twins. Novel views are required to get the correct heat signatures of objects, avoiding parallax errors. Thermal patterns of objects can be very complex. For example, heat leakages from building walls mostly look like blobs, elongated structures, or irregular shapes ( 11 ) . To obtain a more accurate picture of these complex patterns, many different viewpoints would be necessary to have the ability to rotate, view, and inspect them. Novel views can provide realistic images from viewpoints that were never actually captured, because taking photos from all angles is not feasible. Also, continuous viewpoint changes can be useful to study space/time-related heat transfer behavior. And predict heat radiation from new viewpoints. Recent