Beyond Strict Pairing: Arbitrarily Paired Training for High-Performance Infrared and Visible Image Fusion
Yanglin Deng, Tianyang Xu, Chunyang Cheng, Hui Li, Xiao-jun Wu, Josef Kittler, School of Artificial Intelligence, Computer Science, Jiangnan University · 2026
Read Beyond Strict Pairing: Arbitrarily Paired Training for High-Performance Infrared and Visible Image Fusion by Yanglin Deng, Tianyang Xu, Chunyang Cheng, Hui Li, Xiao-jun Wu, Josef Kittler, School of Artificial Intelligence, Computer Science, Jiangnan University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21820.txt.
Introduction
Deng and co-authors question whether infrared and visible image fusion needs perfectly aligned training pairs. Fusion blends heat signatures with natural textures, but aligned pairs are costly to collect. They compare strict pairing with unpaired and arbitrarily paired training. A framework built for loose data enriches the cross-modal links the model can learn. Three light baselines cover CNN, Transformer, and GAN designs, each with new loss functions. Models trained on a tiny unaligned set matched those trained on a set one hundred times larger with strict pairing.
Catalog record
| CREATOR | Yanglin Deng, Tianyang Xu, Chunyang Cheng, Hui Li, Xiao-jun Wu, Josef Kittler, School of Artificial Intelligence, Computer Science, Jiangnan University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40693 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.21820
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Strict Pairing: Arbitrarily Paired Training for High-Performance Infrared and Visible Image Fusion Yanglin Deng, Tianyang Xu, Chunyang Cheng, Hui Li, Xiao-jun Wu, Josef Kittler, School of Artificial Intelligence, Computer Science, Jiangnan University Abstract Infrared and visible image fusion(IVIF) combines complementary modalities while preserving natural textures and salient thermal signatures. Existing solutions predominantly rely on extensive sets of rigidly aligned image pairs for training. However, acquiring such data is often impractical due to the costly and labour-intensive alignment process. Besides, maintaining a rigid pairing setting during training restricts the volume of cross-modal relationships, thereby limiting generalisation performance. To this end, this work challenges the necessity of Strictly Paired Training Paradigm (SPTP) by systematically investigating UnPaired and Arbitrarily Paired Training Paradigms (UPTP and APTP) for high-performance IVIF. We establish a theoretical objective of APTP, reflecting the complementary nature between UPTP and SPTP. More importantly, we develop a practical framework capable of significantly enriching cross-modal relationships even with severely limited and unaligned training data. To validate our propositions, three end-to-end lightweight baselines, alongside a set of innovative loss functions, are designed to cover three classic frameworks (CNN, Transformer, GAN). Comprehensive experiments demonstrate that the proposed APTP and UPTP are feasible and capable of training models on a severely limited and content-inconsistent infrared and visible dataset, achieving performance comparable to that of a dataset 100 × \times larger in SPTP. This finding fundamentally alleviates the cost and difficulty of data collection while enhancing model robustness from the data perspective, delivering a feasible solution for IVIF studies. The code is available at https://github.com/yanglinDeng/IVIF_unpair . 1 Introduction Figure 1 : Motivation of proposed two training paradigms. (a) Compared with the strictly-paired paradigm, the two proposed paradigms lift the requirement for spatiotemporally aligned pairs, expanding the trainable pairs to N\times(N-1) and N^{2} , respectively, reducing data-collection costs and improving generalisation. (b) Performance improvement pattern: greater enhancement from N^{2} for Smaller N . Infrared and visible image fusion (IVIF) aims to generate a comprehensive fused image O that contains both consistent and complementary information from infrared source I_{ir} and visible source I_{vis} 48 , O=\mathcal{F}(I_{ir},I_{vis}), (1) where \mathcal{F} represents the fusion model. Specifically, visible modality excels at capturing colourful visual information aligned with human perception. However, it is sensitive to abnormal lighting conditions, occlusion, and poor weather 47 . In contrast, infrared modality is robust to illumination variations and environmental challenges ( e.g., darkness, smoke, haze). Nevertheless, it lacks colour and fine-grained textural details inherent in the visible modality 54 . Thanks to the complementarity of the two, IVIF has been studied in recent decades with practical applications demands, such as semantic segmentation 44 ; 9 ; 56 , object detection 11 ; 10 ; 7 , and crowd counting 23 ; 22 . In general, an IVIF model is trained to extract the complementary content and consistent scene information across the modalities 18 ; 41 ; 20 ; 30 ; 6 ; 51 ; 40 ; 31 ; 5 . In principle, fusion performance is determined by four factors, i.e. , paired training data, model architecture, supervision loss, and training strategies 14 . Despite the encouraging performance, existing solutions require a large quantity of strictly aligned infrared and visible image pairs, which poses significant challenges for data collection 48 . For instance, when collecting strictly paired data in diverse weather, managing spatial and temporal synchronisation and device settings is not trivial. Besides, considering sensor calibration errors, target motion, and thermal deformations, image registration requires additional effort 43 ; 34 ; 1 ; 35 . Given the inherent differences among modalities, achieving perfect registration is challenging and time-consuming. Notwithstanding the issues raised by collecting paired image data, the fusion performance is also fragile due to discrepancies in data distributions. In particular, within a certain dataset, the observed relationships, which are somewhat limited in paired image datasets due to the dataset size, impose an upper bound on the fusion capability that a model can absorb. This is a consequence of the well-known result in machine learning that increasing the amount of training data always helps to improve the generalisation performance. In conclusion, the conventional strictly-paired training paradigm (SPTP) suffers from critical issues in both cost-effectiveness and model generalisation. On the contrary, given the availability of unimodal datasets, it is very easy to create arbitrarily paired infrared and visible pairs. This convenience motivates us to explore a novel arbitrarily-paired training paradigm (APTP). In fact, the idea of APTP is quite common in the field of image-to-image generation 57 ; 25 ; 15 ; 45 ; 29 ; 55 . In this particular task, the indispensable conducive factor is the existence of the desired image output. Unfortunately, the absence of the ground truth fusion output precludes the application of methods like 57 to the IVIF task. To remedy this situation, we pioneer the APTP of IVIF from a brand-new perspective. The pixel-level self-supervised nature of image fusion tasks allows us to relax the learning objective of traditional SPTP, which is premised on strictly consistent content across modalities. Specifically, we explore supervision signals to facilitate arbitrary pixel-level combinations of cross-modal source images, thereby avoiding reliance on strictly consistent content and enabling arbitrary training. This novel working mechanism motivates the expanded independent relationship between the multi-modal inputs. According to its independence, we expand the maximum log-likelihood estimation optimisation objective of fusion system parameters trained from SPTP to APTP. Both the independent condition and the improved optimisation formulation of APTP deliver a key message: APTP is the union of the two complementary sets, UPTP (Unpaired Training Paradigm) and SPTP, which inherently poses a broader range of applications. Among them, UPTP is the most challenging subtask. The unrestricted pairing among source images significantly enhances the trainable cross-modal diversity within a volume-limited base dataset, leading to a more robust optimisation formulation. This not only fundamentally alleviates the difficulties of data collection and reduces costs, but also enriches the diversity of training data, thereby enhancing model robustness. To systematically investigate the merits of the proposed APTP and UPTP, we build three concise fusion baselines, including CNN, Transformer, and GAN. In particular, CNN aims at extracting context features by focusing on local patterns. In contrast, the Transformer architecture is expert in long-distance modelling, while ignoring the local details to some extent 37 . The generative adversarial network (GAN) baseline follows the generation-and-verification mode to perform network training. Our experimental comparison of the traditional SPTP, novel UPTP, and APTP, provides convincing support for the proposed innovation. Our contributions are as follows: • Two new training paradigms for the IVIF task, which are not constrained to rely exclusively on strictly paired inputs, are presented. • A detailed analysis of the data relationships and training requirements across the three paradigms. • A thorough investigation of the merits of the new training paradigms, which reveals their ability to reduce data-collection cost, to expand the effective training set, and to enhance model robustness. • The experimental results providing supporting evidence for the validity of the proposed APTP and UPTP, and the superiority of their performance compared to SOTA, with high efficiency and robust generalisation. 2 Related Work 2.1 Strictly Paired IVIF Most IVIF approaches are trained using strictly paired datasets. Depending on