Bi-directional Self-Registration for Misaligned Infrared-Visible Image Fusion
Timing Li, Bing Cao, Pengfei Zhu, Bin Xiao, Qinghua Hu · 2025
Read Bi-directional Self-Registration for Misaligned Infrared-Visible Image Fusion by Timing Li, Bing Cao, Pengfei Zhu, Bin Xiao, Qinghua Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06920.txt.
Introduction
Timing Li and four co-authors present a self-supervised method for aligning infrared and visible images before fusing them, in a 2025 paper. Fusion works well only when the two views line up, and misaligned pairs are common. Worse, there is no ground truth alignment to train on, since perfectly registered pairs are exactly what is missing.
Their framework creates its own training signal in both directions. It registers real pairs to find global differences, and it also distorts images on purpose, through crops, flips, and stitching, to create known local differences to learn from. A special loss aligns edges across the two modalities despite their different look. Tests on misaligned image sets showed better alignment and fusion than rival methods.
Catalog record
| CREATOR | Timing Li, Bing Cao, Pengfei Zhu, Bin Xiao, Qinghua Hu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06920
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bi-directional Self-Registration for Misaligned Infrared-Visible Image Fusion Timing Li, Bing Cao, Pengfei Zhu, Bin Xiao, Qinghua Hu Abstract Acquiring accurately aligned multi-modal image pairs is fundamental for achieving high-quality multi-modal image fusion. To address the lack of ground truth in current multi-modal image registration and fusion methods, we propose a novel self-supervised B i-directional S elf- R egistration framework ( B-SR ). Specifically, B-SR utilizes a proxy data generator (PDG) and an inverse proxy data generator (IPDG) to achieve self-supervised global-local registration. Visible-infrared image pairs with spatially misaligned differences are aligned to obtain global differences through the registration module. The same image pairs are processed by PDG such as cropping, flipping, stitching, etc., and then aligned to obtain local differences. IPDG converts the obtained local differences into pseudo-global differences, which are used to perform global-local difference consistency with the global differences. Furthermore, aiming at eliminating the effect of modal gaps on the registration module, we design a neighborhood dynamic alignment loss to achieve cross-modal image edge alignment. Extensive experiments on misaligned multi-modal images demonstrate the effectiveness of the proposed method in multi-modal image alignment and fusion against the competing methods. Our code will be publicly available. August 24, 2026 1 Introduction Multi-modal image fusion has become a crucial technology in various fields, including surveillance, medical imaging, and remote sensing [ 1 , 2 , 3 , 4 , 5 ] . The infrared images excel at capturing thermal information and identifying objects in low light conditions, while the visible images provide detailed texture and color information. Through the exploitation of inter-modal information complementarity, multi-modal image fusion achieves more accurate and comprehensive scene information [ 6 , 7 , 8 , 9 ] . Figure 1: (a) Self-registration paradigm in B-SR. (b) The first column shows infrared and visible image pairs with 5 pixels difference in the horizontal direction, where the yellow dashed line denotes the horizontal difference between different modalities. Compared with existing alignment-based fusion methods, B-SR effectively addresses the issue of misalignment between modalities. The effectiveness of multi-modal image fusion is heavily based on precise image alignment [ 10 , 11 , 12 , 13 , 4 , 14 ] . Spatial misalignment between different modes can lead to degradation of the quality of the fused image, and severe spatial misalignment can result in ghosting in the fused image. [ 15 ] . In the acquisition of multi-modal image data, the differences in sensors and camera jitter lead to inevitable changes in the position and viewing angle of different sensors during the image capture process [ 16 ] . This results in spatial differences in the acquired images, and fusion using unaligned multi-modal images can lead to degradation in the quality of the fused image and, in severe cases it can lead to mismatch in the fused image. Unfortunately, current multi-modal fusion methods rarely consider this problem. Therefore, multi-modal image alignment and fusion is an urgent problem to be solved in practical applications. Most of the existing image fusion methods fail to consider the effect of data misalignment on multi-modal image fusion, which leads to significant differences in model performance on different datasets. As shown in Figure 1 , the performance of the model fusion is affected by the quality of the data alignment, i.e., there are significant artifacts in the fusion results after artificially creating non-aligned offsets in the aligned RoadScene dataset. The same problem exists when the model uses the non-aligned DroneVehicle and TNO datasets. In particular, the misalignment problem is especially severe with the drone data due to having a larger viewing angle and more severe jitter problems. Several related studies have emerged in recent years to mitigate the effects of misaligned data on infrared-visible image fusion. These frameworks align multi-modal images using either singer or multiple alignment modules. However, these methodologies invariably rely on image translation, converting one type of multi-modal image into another to reduce the impact of modality discrepancies. These alignment methods based on image translation are straightforward, but inevitably introduce new noise into the images, while making the effectiveness of modal alignment heavily dependent on the performance of image translation. However, the lack of specialized sensors capable of acquiring aligned infrared-visible image pairs represents a significant challenge. Although manual annotation can result in precisely aligned image pairs, the high costs associated with this approach make it difficult to obtain enough aligned data. Therefore, alternative strategies are necessary to supervise the alignment process without ground-truth data. To address the aforementioned challenges, we propose a novel self-supervised B idirectional S elf- R egistrationtion framework, named B-SR , as shown in Figure 2 . We designed the data proxy generator (PDG) and inverse data proxy generator (IPDG) to achieve global-local self-supervised inter-branch registration, as well as self-supervised intra-branch registration using bi-directional deformation fields. Unlike other methods that rely on image translation models, we propose the neighborhood dynamic alignment loss to reduce the effect of modal differences on the registration module, thus enabling multi-modal image registration without introducing additional noise. Furthermore, the reconstruction module is introduced to bridge the registration and fusion modules, while jointly optimizing the registration and reconstruction modules. Finally, a fused image is generated by fusing the predicted registered image with the original input image. The main contributions are summarized as follows: • We propose a bi-directional self-registration framework that utilizes PDG and IPDG to provide self-supervised signals to the registration network, solving the problem of lacking constraints for the alignment task. • We propose a neighborhood dynamic alignment loss to constrain regions with significant gradient changes, effectively overcoming the modality gap without introducing additional noise typically in translation-based techniques. • We introduce a self-reconstruction module, jointly optimized with the registration to improve alignment and fusion performance. Extensive experiments demonstrate our superiority against the competing methods for misaligned infrared and visible image fusion. 2 Related Work 2.1 Multi-modal Image Alignment Image alignment is divided into traditional alignment methods and deep learning alignment methods. Traditional alignment methods primarily include feature-based matching algorithms, mutual information-based registration, and frequency domain alignment techniques [ 17 , 18 ] . Arar [ 19 ] proposed a multi-modal alignment method based on modal translation, which translates different modal images into the same modality before image alignment, reducing the effect of modal differences on multi-modal image alignment. Building upon this foundation, Wang et al. [ 20 ] proposed an unsupervised alignment network based on modal style transformation, and Han et al. [ 21 , 22 ] further refined this methodology, advancing the research in multi-modal image alignment. While image translation can mitigate modality differences, it inevitably introduces additional random noise [ 23 ] . Moreover, the dynamic nature of multi-modal images can degrade the performance of image translation. Therefore, we propose a dedicated loss function and jointly optimize image alignment and reconstruction tasks without introducing additional noise. This approach aims to achieve image alignment while maximally preserving information from different modalities. Figure 2: Architecture of the B-SR method. (a) The training process of the bi-directional self-registration framework is designed to align multi-modal images by leveraging both intra-branch bi-directional alignment constraints and inter-branch alignment results simultaneously. The inter-branch alignment is achieved through PDG and IPDG, while the intra-branch alignment is achieved