Infrared and Visible Image Fusion Based on Implicit Neural Representations
Shuchen Sun, Ligen Shi, Chang Liu, Lina Wu, Jun Qiu · 2025
Read Infrared and Visible Image Fusion Based on Implicit Neural Representations by Shuchen Sun, Ligen Shi, Chang Liu, Lina Wu, Jun Qiu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16773.txt.
Introduction
Shuchen Sun and four co-authors present a method for merging heat-sensor and normal camera images, in this 2025 paper. Heat images show warm objects clearly but lack fine detail. Normal photos hold detail but miss heat. A merged image can serve night driving, rescue work, and inspection tasks better than either source alone.
Their method, INRFuse, stores the merged picture inside a neural network rather than as a grid of pixels. The network takes each point's coordinates and outputs the blended value, learning from both source images at once. Loss rules push it to keep heat data from one image and texture from the other. Because the picture is a smooth function, it can be drawn at any size, even larger than the inputs. Tests beat rival methods on both human ratings and standard scores, with no training data set needed.
Catalog record
| CREATOR | Shuchen Sun, Ligen Shi, Chang Liu, Lina Wu, Jun Qiu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-16773
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Infrared and Visible Image Fusion Based on Implicit Neural Representations Shuchen Sun, Ligen Shi, Chang Liu, Lina Wu, Jun Qiu Abstract Infrared and visible light image fusion aims to combine the strengths of both modalities to generate images that are rich in information and fulfill visual or computational requirements. This paper proposes an image fusion method based on Implicit Neural Representations (INR), referred to as INRFuse. This method parameterizes a continuous function through a neural network to implicitly represent the multimodal information of the image, breaking through the traditional reliance on discrete pixels or explicit features. The normalized spatial coordinates of the infrared and visible light images serve as inputs, and multi-layer perceptrons is utilized to adaptively fuse the features of both modalities, resulting in the output of the fused image. By designing multiple loss functions, the method jointly optimizes the similarity between the fused image and the original images, effectively preserving the thermal radiation information of the infrared image while maintaining the texture details of the visible light image. Furthermore, the resolution-independent characteristic of INR allows for the direct fusion of images with varying resolutions and achieves super-resolution reconstruction through high-density coordinate queries. Experimental results indicate that INRFuse outperforms existing methods in both subjective visual quality and objective evaluation metrics, producing fused images with clear structures, natural details, and rich information without the necessity for a training dataset. I Introduction Traditional image processing mainly relies on information from a single modality and struggles to effectively handle complex real-world scenarios. As a result, image fusion technology has become a key research direction for enhancing perception and decision-making capabilities.Infrared and visible light image fusion represents a fundamental aspect of image fusion technology. Researchers both domestically and internationally have undertaken extensive studies in this field, encompassing both traditional fusion techniques and those based on deep learning. Traditional methods [ 1 ] depend on manually designed feature extraction rules, which often fail to adequately account for the differences between image modalities, thereby limiting the ability to harness their complementary information. Furthermore, these approaches exhibit limited adaptability and lack the capability to be adjusted according to varying scenarios or image characteristics. The advantages of deep learning in feature extraction and data representation have led to the development of a growing number of deep learning-based fusion methods [ 2 ] . These methods excel in feature extraction and detail enhancement. However, due to their reliance on pixel-level explicit operations, they often face challenges in delicately processing image details, particularly in high-frequency regions under complex scenarios. Additionally, deep learning approaches typically rely on fixed network architectures and training strategies, which may not adaptively accommodate modality differences. Moreover, as image resolution increases, there is a corresponding rise in storage and computational overhead. To address the limitations of existing methods, this paper proposes an end-to-end fusion method based on Implicit Neural Representation (INR). INR is a technique that utilizes neural networks to learn continuous representations of data [ 3 ] , treating images as continuous implicit functions and generating their representations through neural network parameterization to achieve high visual quality.In the proposed method, the neural network learns a continuous mapping function to implicitly model the multimodal information of images, efficiently fusing the complementary characteristics of visible light and infrared images through a joint optimization strategy. Compared to traditional methods, INRFuse eliminates the need for manually designed fusion rules, automatically constructing the optimal fusion strategy through end-to-end learning. Additionally, its continuous representation avoid interpolation errors caused by discrete representations. Unlike deep learning methods that require large training datasets, INRFuse demonstrates high robustness with fewer network parameters and does not rely on extensive training data. The main contributions of this paper can be summarized as follows: • A novel image fusion method for infrared and visible light images based on INR is proposed. This method effectively processes data from different sensors through implicit neural representations, avoiding the detail loss caused by traditional discrete pixel representations and facilitating the precise capture of fine image structures. • A multi-constraint loss function incorporating pixel consistency, structural preservation, and sparsity regularization is designed to achieve unsupervised high-quality fusion. • This method offers a general framework for multimodal fusion, supporting the fusion of images with different resolutions as well as super-resolution reconstruction of the fused image. II Method Fig. 1: Infrared-Visible Light Fusion Framework Based on Implicit Neural Representations. II-A Parameterized Representation of Image Fusion This paper proposes an infrared and visible image fusion method based on Implicit Neural Representation (INR). The method constructs a continuous mapping function that maps spatial coordinates \mathbf{X}=(x,y) to the pixel values of the fused image, as defined by: F_{\theta}:\mathbf{X}\mapsto I_{\text{F}}(\mathbf{X}), (1) where I_{\text{F}} denotes the implicit function parameterized by a neural network F_{\theta} . The input is the image coordinate \mathbf{X} , and the output is the corresponding pixel value in the fused image. By parameterizing this continuous function with a neural network, the model learns a joint implicit representation that simultaneously encodes information from the infrared image I_{\text{IR}}(\mathbf{X}) and the visible image I_{\text{VIS}}(\mathbf{X}) . During the optimization process, the network adaptively learns the correspondence between the two modalities and coherently maps them into a unified representational space, thus enabling effective multi-modal fusion. The overall framework of the proposed fusion method is illustrated in Fig. 1 . II-B Neural Network Model Sinusoidal Representation Networks (SIREN) [ 4 ] enhance the modeling capacity for high-frequency information by incorporating the sine function as the activation function within Multilayer Perceptrons (MLPs). From a physical perspective, both visible and infrared light are electromagnetic waves whose intrinsic oscillatory nature can be naturally represented by sinusoidal functions. This inherent alignment between the physical characteristics of light and the mathematical form of sine functions makes SIREN particularly well-suited for modeling multi-modal image signals. To address the common issue of gradient vanishing associated with sine activations, researchers have developed a specialized parameter initialization scheme and demonstrated SIREN stability and reliability in fitting complex signals and their derivatives. In this paper, we construct MLPs network consisting of 5 layers, each containing 256 neurons, with the sine function applied as the activation in all hidden layers: \phi_{i}(\mathbf{X}_{i})=\sin\left(\mathbf{W}_{i}\mathbf{X}_{i}+\mathbf{b}_{i}\right), (2) where \phi_{i}:\mathbb{R}^{M_{i}}\mapsto\mathbb{R}^{N_{i}} is the i^{th} layer of the network. It consists of the affine transform defined by the weight matrix \mathbf{W}_{i}\in\mathbb{R}^{N_{i}\times M_{i}} and the biases \mathbf{b}_{i}\in\mathbb{R}^{N_{i}} applied on the input \mathbf{X}_{i}\in\mathbb{R}^{M_{i}} , followed by the sine nonlinearity applied to each component of the resulting vector. II-C Loss Function The loss function design should consider both the salient targets of the infrared image and the texture details of the visible image. To achieve this, this paper proposes a loss function with three components: pixel consistency loss, gradient consistency loss, and regularization loss. The pixel consistency loss calculates the L_{1} norm difference between the fused image and the source images to reduce distortion during the fusion process. The gradient consistency loss constrains the difference between the fused image and the source images in the gradient domain to preserve edge and structural information. The regularization term is used to suppress noise and enhance the smoothness of the fused image.The formulation is as follows: \displaystyle\mathcal{L}= \displaystyle\|I_{\text{IR}}(\mathbf{X})-I_{\text{F}}(\mathbf{X})\|_{2}^{2}+\|I_{\text{VIS}}(\mathbf{X})-I_{\text{F}}(\mathbf{X})\|_{2}^{2} (3) \displaystyle+\|\nabla I_{\text{IR}}(\mathbf{X})-\nabla I_{\text{F}}(\mathbf{X})\|_{2}^{2}+\|\nabla I_{\text{VIS}}(\mathbf{X})-\nabla I_{\text{F}}(\mathbf{X})\|_{2}^{2} \displaystyle+\lambda\|\nabla I_{\text{F}}(\mathbf{X})\|_{1}, where \lambda is the hyperparameter for sparse regularization loss, \nabla represents