OCCO: LVM-guided Infrared and Visible Image Fusion Framework based on Object-aware and Contextual COntrastive Learning

Hui Li, Congcong Bian, Zeyang Zhang, Xiaoning Song, Xi Li, Xiao-Jun Wu · 2025

Read OCCO: LVM-guided Infrared and Visible Image Fusion Framework based on Object-aware and Contextual COntrastive Learning by Hui Li, Congcong Bian, Zeyang Zhang, Xiaoning Song, Xi Li, Xiao-Jun Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18635.txt.

Introduction

Merging an infrared image with a normal photo can reveal people and objects hidden by darkness or glare. Li and five co-authors propose OCCO, a fusion method guided by a large pretrained vision model. The big model supplies meaning-level guidance, so the fusion network can focus on blending while still learning the features that matter.

A new feature-mixing network settles clashes between the two image types. Training with contrastive learning, comparing matched and mismatched samples in a hidden feature space, keeps target details intact in the merged image. The team compares OCCO with eight leading methods on four datasets. It produced high-quality fused images and also improved the visual tasks that use them, a balance earlier methods struggled to hold. The authors pitch the design for surveillance, driving, and other fields that fuse sensor views.

Catalog record

CREATORHui Li, Congcong Bian, Zeyang Zhang, Xiaoning Song, Xi Li, Xiao-Jun Wu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18635

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OCCO: LVM-guided Infrared and Visible Image Fusion Framework based on Object-aware and Contextual COntrastive Learning Hui Li, Congcong Bian, Zeyang Zhang, Xiaoning Song, Xi Li, Xiao-Jun Wu Abstract Image fusion is a crucial technique in the field of computer vision, and its goal is to generate high-quality fused images and improve the performance of downstream tasks. However, existing fusion methods struggle to balance these two factors. Achieving high quality in fused images may result in lower performance in downstream visual tasks, and vice versa. To address this drawback, a novel LVM (large vision model)-guided fusion framework with O bject-aware and C ontextual CO ntrastive learning is proposed, termed as OCCO . The pre-trained LVM is utilized to provide semantic guidance, allowing the network to focus solely on fusion tasks while emphasizing learning salient semantic features in form of contrastive learning. Additionally, a novel feature interaction fusion network is also designed to resolve information conflicts in fusion images caused by modality differences. By learning the distinction between positive samples and negative samples in the latent feature space (contextual space), the integrity of target information in fused image is improved, thereby benefiting downstream performance. Finally, compared with eight state-of-the-art methods on four datasets, the effectiveness of the proposed method is validated, and exceptional performance is also demonstrated on downstream visual task. keywords Image fusion, Large vision models, Semantic segmentation, Contrastive learning, Contextual loss. 1 Introduction The multi-modal image fusion task aims to merge different modal images captured by various sensors in the same scene ( Liu et al, 2020 ) , and fully leveraging complementary features to generate a fused image containing effective information. In infrared and visible image fusion, visible images offer abundant texture details and rich contextual scene information. However, under adverse conditions, such as darkness or object occlusion, the utility of visible information may be substantially reduced ( Liu et al, 2024a ) . Conversely, infrared imaging, which senses thermal radiation, possesses the intrinsic advantage of being considerably less susceptible to environmental factors. By leveraging the complementary characteristics with a well-designed fusion strategy, the fused image will contain the rich scene information and highlight salient objects. Theoretically, this will lead to better performance in advanced visual tasks such as tracking ( Zhang et al, 2021 ) , object detection ( Jain et al, 2023 ; Zheng et al, 2024 ) , semantic segmentation ( Zhang et al, 2023a ) , etc. Infrared and visible image fusion can be categorized into three types based on fusion orientation: (1) visual-driven methods ( Pajares and Manuel de la Cruz, 2004 ; Li and Wu, 2019 ; Ma et al, 2019b ) , (2) utilizing multi-modal complementarity to directly impact downstream tasks ( Zhang et al, 2023a ; Zhang et al, 2023b ) , and (3) task-driven approaches ( Tang et al, 2022 ; Liu et al, 2023b ) . Over time, a multitude of non-deep learning ( Li et al, 2013 ; Liu et al, 2015 ) and deep learning ( Li and Wu, 2019 ; Ma et al, 2019b ) methods have been created and applied in image fusion, leading to improved image quality. However, infrared and visible image fusion aims to preserve source information as much as possible, while downstream tasks require clear boundaries. This disparity restricts the further advancement of image fusion. Figure 1: From (a) to (d) are visual-driven method, semantic-driven method, feature-level semantic-driven method and object-aware method, respectively. The visual quality in (a) is outstanding with preserved information integrity, albeit showing slightly weaker performance in downstream tasks. Combining segmentation networks in both (b) and (c) enhances downstream performance but has an impact on the quality of the fused images. Focusing exclusively on the fusion task in (d) improves downstream performance without disrupting the fusion results. Therefore, task-driven methods are crucial for image fusion. Visual-driven methods focus on visual perception and make decisions on information retention or elimination based on algorithms, potentially reducing the importance of essential objects due to low intensity. In Fig. 1 (a) ( Zhao et al, 2023b ) , for optimal visual outcomes, uniform information retention without highlighting key scene elements lead to false negative and low confidence in object detection. To address this drawback, some scholars propose training fusion networks with segmentation networks in a cascaded manner to enhance performance in downstream tasks, as shown in Fig. 1 (b) ( Tang et al, 2022 ) . Although detection results are outstanding, the substantial differences between the two tasks can disrupt fusion outcomes, resulting in fragmented information retention. Moreover, researchers explores methods to effectively interact the features between two tasks to achieve synergistic development. However, for converging with the segmentation loss, the fusion results in Fig. 1 (c) ( Liu et al, 2023b ) exhibit forced information diffusion to enhance edge textures. To this end, we choose to only utilize the LVM to provide prior semantic information to guide the training, without introducing the segmentation network with segmentation loss, as depicted in Fig. 1 (d). In this way, it not only avoids the gradient disturbance caused by the display constraint, but also retains the semantic perception ability to the greatest extent. Contrastive learning optimizes fusion results based on extracted semantic information and enhances semantic content without interfering with the fusion network. Proposed the LVM-guided O bject-aware C ontextual CO ntrastive Learning for infrared and visible image fusion, termed as OCCO, which tackles incomplete semantic information on salient targets due to modality discrepancies. With advancements in LVM, more effective guiding information can be captured. The Segment Anything Model (SAM) ( Kirillov et al, 2023 ) , as a state-of-the-art zero-shot segment large model, it exhibits robust segmentation and generalization capabilities, outperforming other segmentation models. This allows us to thoroughly analyze which information in each modality is more effective for the fused image. Thus, SAM will guide the overall training process as a high-level visual task adviser. Contrastive learning, as a powerful training strategy in both supervised and unsupervised learning, is particularly well-suited for multi-modal image fusion tasks that lack ground truth. Many methods have successfully introduce contrastive learning into multi-modal image fusion tasks ( Liu et al, 2023a ; Liu et al, 2023c ) . Due to the requirement for manual annotation of salient objects, CoCoNet ( Liu et al, 2023a ) is limited to training on the few sample dataset, such as TNO, which lacks scene diversity and texture information in visible images. Thus, in this paper, we propose a novel SAM-guided mask generator, which utilizes a text-driven object detection model Grounding DINO ( Liu et al, 2023d ) to acquire complete detection boxes and used as box prompts in SAM. This processing facilitates the extraction of numerous precisely annotated samples. The differences in two modalities are visually reflected in the masks. When visible information is rich and infrared information is missing at the same location, the corresponding part will only be displayed in the visible mask. This forms a natural pair of positive samples (rich semantic information) and negative samples (sparse semantic context). We expect the anchor samples (fused image) to contain as much rich information as possible, while staying away from modalities with sparse information. In addition, for contrastive learning in multi-modal

Read the complete text