Language-Depth Navigated Thermal and Visible Image Fusion
Jinchang Zhang, Zijun Li, Guoyu Lu · 2025
Read Language-Depth Navigated Thermal and Visible Image Fusion by Jinchang Zhang, Zijun Li, Guoyu Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08676.txt.
Introduction
Jinchang Zhang and co-authors merge heat and light images into a single picture that also knows depth. Thermal cameras see in darkness but lose texture, while normal cameras fail in the dark; fused views serve robots and rescue teams better than either alone. Most fusion work serves detection tasks and throws depth away.
Their network leans on written prompts: a vision-language model reads the scene description and steers a diffusion process that blends the two feeds into a color image. Two side branches then estimate depth from the fused result, and depth errors flow back to sharpen the fusion itself. The loop is meant to give machines richer scene sense for driving, navigation, and search missions in murky settings.
Catalog record
| CREATOR | Jinchang Zhang, Zijun Li, Guoyu Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08676
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Language-Depth Navigated Thermal and Visible Image Fusion Jinchang Zhang, Zijun Li, Guoyu Lu Abstract Depth-guided multimodal fusion combines depth information from visible and infrared images, significantly enhancing the performance of 3D reconstruction and robotics applications. Existing thermal-visible image fusion mainly focuses on detection tasks, ignoring other critical information such as depth. By addressing the limitations of single modalities in low-light and complex environments, the depth information from fused images not only generates more accurate point cloud data, improving the completeness and precision of 3D reconstruction, but also provides comprehensive scene understanding for robot navigation, localization, and environmental perception. This supports precise recognition and efficient operations in applications such as autonomous driving and rescue missions. We introduce a text-guided and depth-driven infrared and visible image fusion network. The model consists of an image fusion branch for extracting multi-channel complementary information through a diffusion model, equipped with a text-guided module, and two auxiliary depth estimation branches. The fusion branch uses CLIP to extract semantic information and parameters from depth-enriched image descriptions to guide the diffusion model in extracting multi-channel features and generating fused images. These fused images are then input into the depth estimation branches to calculate depth-driven loss, optimizing the image fusion network. This framework aims to integrate vision-language and depth to directly generate color-fused images from multimodal inputs. I Introduction Multimodal fusion integrates multi-source information to achieve comprehensive data representation, addressing the limitations of single-modal data [ 40 ] . Infrared and visible image fusion is a typical task [ 20 ] , where infrared imaging is robust in low-light conditions but lacks texture details [ 14 ] , while visible images contain rich structural and texture information but are affected by lighting variations [ 15 ] . The complementarity of these modalities allows fused images to retain both thermal targets and texture details, enhancing visual perception [ 52 ] . However, existing methods primarily focus on detection-based image fusion, overlooking the role of depth guidance. Depth-guided multimodal fusion combines depth information from visible and infrared images to improve 3D reconstruction and robotic perception. However, modality differences can cause information loss and feature mismatches. Fusing these modalities enables capturing complete geometric structures in complex environments, improving 3D point cloud quality and model adaptability. Depth-guided fusion is crucial for navigation, localization, and environmental perception, yet achieving comprehensive scene understanding in low-light or obstacle-dense environments remains a significant challenge. For example, in autonomous driving, roads and obstacles must be identified under extreme conditions; in rescue missions, reliable 3D reconstruction is required in environments obscured by smoke or dust. To address these challenges, we propose a depth-driven visible-infrared image fusion framework based on vision-language models. This framework includes a multi-channel feature extraction module based on diffusion models, a language-guided fusion module, and depth-driven fusion branches. In the depth-driven module, we design two depth estimation branches for supervised depth estimation of infrared and visible images. The fused image is fed into the trained depth estimation networks, where depth-driven loss is computed by comparing the predicted depth with ground truth, optimizing the image fusion network using depth information. In the diffusion-based feature extraction module, visible and infrared images are concatenated along the channel dimension to form a four-channel source image, serving as the multi-channel input for the diffusion model and the ground truth for training a self-supervised diffusion model. In the forward process, Gaussian noise is gradually added to the multi-channel data until it becomes nearly pure noise. In the reverse process, a denoising network predicts and removes the noise. After training the noise prediction network, we use it to extract multi-channel features. Given the lack of constraints from real fused images, we introduce a language-guided fusion process. By generating text descriptions that integrate image and depth information, we comprehensively capture scene details and structure. In the language-driven fusion module, we employ a Depth-Informed Image Captioning Network, which takes infrared and visible images along with their predicted depth as inputs and generates depth-enriched text descriptions. These descriptions are then processed by CLIP’s text encoder to obtain text embedding features. From the text embeddings, we extract semantic parameters and use an MLP to analyze semantic information. The network outputs semantic information and parameters, which interact with the fusion features through scale adjustment and bias control. Under the guidance of semantic parameters, the fusion process dynamically adjusts feature weights and priorities, producing results that better align with scene requirements. Fig. 1: Overview of the framework. The visible and infrared images are combined as a four-channel input, serving as ground-truth for self-supervised training of the noise prediction network to extract multi-channel features. In the forward diffusion process, I_{0} and I_{t} represent the multi-channel input and the data at timestep t , respectively. P(\cdot|\cdot) and Q(\cdot|\cdot) denote the forward and reverse diffusion processes. The multi-channel fusion loss includes intensity loss and gradient loss. Two depth estimation networks are trained separately for visible and infrared images, generating corresponding depth, which are then input into the depth information description module to produce image-text descriptions containing depth information. Using the CLIP text encoder, text features are extracted, and an MLP predicts semantic information and parameters to guide multi-channel feature reconstruction of fused image. The fused image is processed through the two depth estimation networks, generating depth that are compared with ground-truth to calculate depth-driven loss, optimizing the fusion process. Overall, our contributions are summarized as follows: 1. We propose a depth-driven, vision-language-based framework for infrared and visible image fusion. 2. We are the first to introduce a depth-driven infrared and visible image fusion method, using depth-driven loss from the fused images to provide spatial information guidance for optimizing the multimodal image fusion network. 3. We introduce a semantic interaction guidance module, which extracts textual semantic information from image content descriptions containing depth information, guiding the multi-channel features extracted by the diffusion model to generate the fused images. Our framework is shown in Fig 1 . II RELATED WORK II-A Conventional Image Fusion Methods Sparse representation theory represents image signals as a linear combination of the fewest atoms or transformation primitives from an overcomplete dictionary [ 8 ] . In image fusion, it captures data-driven image representations by learning a complete dictionary. Multi-scale transformation decomposes the original image into sub-images at different scales, mimicking the human visual process and enhancing the visual quality of the fused images [ 20 ] . These methods include nonlinear methods, pixel-level weighted averaging, estimation-based methods, and color composite fusion [ 6 ] Subspace representation-based methods project high-dimensional features into a low-dimensional subspace, capturing the intrinsic structure of the input images while saving processing time and memory [ 31 ] . Common methods include PCA [ 50 ] , NMF [ 37 ] Saliency detection models simulate human behavior by capturing the most salient regions or objects in an image and have significant applications in computer vision and pattern recognition [ 39 ] . II-B Deep learning-based Image Fusion Methods CNN-based fusion methods, such as PMGI [ 48 ] , focus on fast image fusion while preserving gradient and intensity. SDNet [ 47 ] enhances this by adaptively determining the proportion of gradient information to retain, aiming to preserve more detailed textures. Diffusion models, a powerful class of