MultiBARF: Integrating Imagery of Different Wavelength Regions by Using Neural Radiance Fields
Kana Kurata, Hitoshi Niigaki, Xiaojun Wu, and Ryuichi Tanida · 2025
Read MultiBARF: Integrating Imagery of Different Wavelength Regions by Using Neural Radiance Fields by Kana Kurata, Hitoshi Niigaki, Xiaojun Wu, and Ryuichi Tanida (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15070.txt.
Introduction
Kana Kurata, Hitoshi Niigaki, Xiaojun Wu, and Ryuichi Tanida present a way to fuse images from unlike sensors, in work from twenty twenty-five. Real tools must tie sensor data to true places in the world. They must also blend views from cameras that see in unlike ways. Getting that blend right takes deep skill in sensing and image work. Many users lack that skill, so prep work becomes a bottleneck.
Their method, called MultiBARF, builds paired sensor views and depth views at chosen angles. It extends a neural scene method that learns a full three-dimensional view from flat shots. The new version handles two imagers at once instead of one. Tests pair normal colour views with heat views of the same scene. The fused output stacks the colour channels cleanly on the learned scene. That result shows the blend lines up well in space.
Catalog record
| CREATOR | Kana Kurata, Hitoshi Niigaki, Xiaojun Wu, and Ryuichi Tanida |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15070
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MultiBARF: Integrating Imagery of Different Wavelength Regions by Using Neural Radiance Fields Kana Kurata, Hitoshi Niigaki, Xiaojun Wu, and Ryuichi Tanida Abstract Optical sensor applications have become popular through digital transformation. Linking observed data to real-world locations and combining different image sensors is essential to make the applications practical and efficient. However, data preparation to try different sensor combinations requires high sensing and image processing expertise. To make data preparation easier for users unfamiliar with sensing and image processing, we have developed MultiBARF. This method replaces the co-registration and geometric calibration by synthesizing pairs of two different sensor images and depth images at assigned viewpoints. Our method extends Bundle Adjusting Neural Radiance Fields(BARF), a deep neural network-based novel view synthesis method, for the two imagers. Through experiments on visible light and thermographic images, we demonstrate that our method superimposes two color channels of those sensor images on NeRF. I Introduction Sensor applications, consisting of sensing, data processing, and visualization or machine automation techniques, have become standardly used daily through digital transformation. Optical imagers are especially popular for recognizing the environments around us: visible light cameras for driving assistance, near-infrared cameras for non-contact pill inspection, and thermal infrared cameras for body temperature measurement at building entrances. These applications often link observed data to real-world locations for display to humans and automatic machine control. Also, combining multiple different sensors is a popular technique [ 1 ] . For example, combinations of optical imagers of different wavelength regions are effective remote sensing for environmental monitoring [ 2 ] . For another example, driving assist systems use visible light cameras combined with infrared cameras, also, according to the monitoring target or the surrounding conditions, with active sensors such as radars and LiDARs [ 3 , 4 ] . These examples show the importance of selecting appropriate sensor combinations for targets and observation environments. Data acquisition and Preprocesses are necessary tasks for sensor applications. Preprocesses have to make the dataset comprehensively analyzable by post-processes of applications. Sometimes, dataset providers distribute co-registered products of multiple sensor images like [ 5 ] . Also, providing applications bounded with the equipment sets, like driving support systems, decreases users’ efforts in data preparation. Such simplifications in data preparation support the recent popularization of sensor applications. On the other hand, trying several sensor combinations without the limitation of existing equipment combo still requires high sensing and image processing skills. To prepare custom datasets consisting of multiple sensor imagery and linked with location information, the preprocess must include radiometric/geometric corrections for each sensor image, co-registration between different sensors, and geometric calibration of all dataset images. While radiometric/geometric corrections depend on sensor properties, co-registration and geometric calibration are often communalized for different types of sensors. We have therefore focused on making the co-registration and geometric calibration processes not require high expertise and be adaptive to various combinations of sensors. This paper proposes MultiBARF, a method to synthesize pairs of two different sensor images and depth images (2D rendered 3D shape) at assigned viewpoints by only inputting multiview images of two sensors into the model. Neural Radiance Fields (NeRF) [ 6 ] is a successful deep learning-based 3D representation for photorealistic scenes. NeRF records 3D models as color and density distributions by a Deep Neural Network(DNN). It is optimized with multiview images and visualized through volume rendering. MultiBARF is based on BARF [ 7 ] , an expansion of NeRF having a function of bundle adjustment. We enabled BARF to work for two different sensor images and automatically register them. Our method does not directly solve the co-registration and geometric calibration but can replace these processes. The characteristics of our method are that it extends the DNN structure of NeRF to be used for two different imagers and does not require camera calibration between the two imagers by utilizing the camera pose estimation process of Bundle Adjusting Neural Radiance Fields(BARF) [ 7 ] , a kind of NeRF, for the two imagers. We evaluated our model using a combination of visible light and thermal infrared images since thermal infrared images are a challenging target for 3D reconstruction and registration in daily scenes. We demonstrate that MultiBARF synthesizes an image pair of two image sensors, observing different wavelength regions at a single viewpoint. In summary, our contributions are: • We propose MultiBARF, a new architecture of NeRF that has two color channels for two different image sensors. • We provide a method to optimize MultiBARF depending on two sensor images without complicated camera calibration. • We demonstrate the synthesizing results of MultiBARF on visible light and thermographic images. II Related Work This study approaches the sensing problem of preparing multispectral datasets using a computer vision technique called NeRF. Therefore, we introduce related methods from the following perspectives: applications in multiple wavelength regions, preparation of multispectral datasets, and NeRF with its extensions. Optical imager applications. There are many applications in the visible and infrared regions, as these areas contain abundant information and are observable in non-special environments. As an object’s condition appears as a reflection or emission feature at the corresponding wavelength, choosing the optimal observation wavelengths is essential. Ultraviolet (UV), visible (VIS), and near-infrared (NIR) to short-wavelength infrared (SWIR) mainly relate to chemical composition or surface microstructure. The UV region can watch the characteristics relating to the attractiveness of a flower to insects [ 8 ] . Conditions of vegetations mainly appear in NIR [ 9 ] , but combined with VIS and SWIR [ 10 ] improved the estimation accuracy for leaf water content. For another example, combining VIS to SWIR [ 11 ] discriminates minerals that seem to be just whitish rocks. On the other hand, the long-wavelength regions called long-wavelength infrared (LWIR) or thermal infrared (TIR), relating to the temperature of daily scenes, sometimes show objects’ internal conditions. Facilities’ internal deteriorations make temperature differences on wall surfaces over them and captured by can be revealed by thermography (TIR region) [ 12 , 13 ] . For another example, Maierhofer et al. [ 14 ] estimate internal void and honeycombing of concrete. The above examples show that multispectral datasets can help us acquire independent information from different wavelength regions and estimate the targets’ comprehensive conditions. Preparation of multispectral datasets. Co-registration between different sensors is necessary to use multiple wavelength regions because a single imager supports a limited wavelength range. As previously reviewed, many effective methods have been developed for datasets with different characteristics [ 15 , 16 ] . Also, linking locational information (e.g., relative locations among the dataset or on 3D models, absolute locations on GIS) through geometric camera calibration is essential to identify the analyzed targets in real space. For the VIS region, there are many convenient calibration tools like COLMAP( [ 17 ] , [ 18 ] ), generally working near VIS (UV, NIR, SWIR). However, these methods require hardware-level customization for long-wavelength regions [ 19 ] . Although we can measure camera parameters alternatively when shooting, it requires additional equipment and expertise. Neural Radiance Fields (NeRF). Mildenhall et al. [ 6 ] proposed the novel view synthesis method that represents photorealistic continuous scenes by “Neural Radiance Fields” (NeRF). Unlike conventional 3D representations such as voxel or mesh, NeRF