On Modality Incomplete Infrared-Visible Object Detection: An Architecture Compatibility Perspective

Shuo Yang, Yinghui Xing, Shizhou Zhang, Zhilong Niu · 2025

Read On Modality Incomplete Infrared-Visible Object Detection: An Architecture Compatibility Perspective by Shuo Yang, Yinghui Xing, Shizhou Zhang, Zhilong Niu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06406.txt.

Introduction

Shuo Yang, Yinghui Xing, Shizhou Zhang, and Zhilong Niu keep infrared-visible object detectors working when one camera type fails. Pairing heat-sensing infrared with ordinary visible images serves round-the-clock surveillance and driving, but accuracy collapses when a modality goes missing, especially the dominant one. The authors study the failure through the lens of architecture compatibility.

Their Scarf Neck plugs into DETR-style detectors and adds modality-blind deformable attention, so one detector flexibly accepts either input alone or both together. A pretend-dropout training trick rehearses missing inputs in advance. The resulting detector excelled in missing-modality tests on a new benchmark and also topped standard benchmarks where both inputs are present.

Catalog record

CREATORShuo Yang, Yinghui Xing, Shizhou Zhang, Zhilong Niu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06406

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On Modality Incomplete Infrared-Visible Object Detection: An Architecture Compatibility Perspective Shuo Yang, Yinghui Xing, Shizhou Zhang, Zhilong Niu Abstract Infrared and visible object detection (IVOD) is essential for numerous around-the-clock applications. Despite notable advancements, current IVOD models exhibit notable performance declines when confronted with incomplete modality data, particularly if the dominant modality is missing. In this paper, we take a thorough investigation on modality incomplete IVOD problem from an architecture compatibility perspective. Specifically, we propose a plug-and-play Scarf Neck module for DETR variants, which introduces a modality-agnostic deformable attention mechanism to enable the IVOD detector to flexibly adapt to any single or double modalities during training and inference. When training Scarf-DETR, we design a pseudo modality dropout strategy to fully utilize the multi-modality information, making the detector compatible and robust to both working modes of single and double modalities. Moreover, we introduce a comprehensive benchmark for the modality-incomplete IVOD task aimed at thoroughly assessing situations where the absent modality is either dominant or secondary. Our proposed Scarf-DETR not only performs excellently in missing modality scenarios but also achieves superior performances on the standard IVOD modality complete benchmarks. Our code will be available at https://github.com/YinghuiXing/Scarf-DETR. 1 Introduction Infrared-visible object detection (IVOD) plays a crucial role in multi-modal learning ( Xu et al. 2024 ; Zhao et al. 2023 ; Wang, Li, and Liu 2023 ; Wu et al. 2023 ; Piekenbrinck et al. 2024 ) because infrared images provide supplementary information in challenging environments such as poor lighting, fog, or situations where visible images are difficult to capture targets. Existing methods typically focus on designing specialized infrared-visible object detectors, which often necessitate complex fusion modules, customized loss functions, and tailored optimization strategies ( Li et al. 2025 ; Shin et al. 2024 ; Xing et al. 2024 ; Guo et al. 2024 ) . Despite significant progress, these models usually require all modalities to be complete during training, limiting their application in real-world scenarios. In practical applications, sensor faults, blurriness and overexposure in images may occur, leading to the missing of specific modality. Given that the architectures of traditional IVOD models are often tightly coupled with the complete input of the modality, when a certain modality is missing, such models will exhibit significant deterioration. Modality Model mAP \uparrow FLIR-aligned M 3 FD LLVIP VIS+IR DAMS-DETR 49.3 52.9 69.6 Scarf-DETR 49.6 54.4 69.6 VIS DAMS-DETR 28.4 46.6 15.0 Scarf-DETR 37.8 51.4 55.5 IR DAMS-DETR 47.0 26.9 69.3 Scarf-DETR 47.3 41.0 69.3 Figure 1: Illustration and necessity to deal with the modality-incomplete issue in IVOD. Available multi-modal detectors fail to detect targets with incomplete modality, while Scarf-DETR is compatible with complete and incomplete modality scenarios. As shown in Figure 1 , previous state-of-the-art model DAMS-DETR achieves only 15% mAP in the VIS-only scenario on the LLVIP dataset, which is 54.6% decrease compared to the complete-modality setting. We also observe that DAMS-DETR can achieve comparable results in the IR-only scenario compared to the ideal modality-complete setting (69.3 vs. 69.6). Based on the observation, it can be clearly seen that the IR modality shows dominant while the visible modality shows secondary on the LLVIP dataset because it mainly consists of images captured at night or low-illuminance conditions. Although we admit that whether the missing modality is dominant or secondary is a significant factor in the final performance, it is worth noting that our Scarf-DETR, which is designed to decouple double input modalities and the model architecture, can improve performance on secondary modality input scenarios from 15.0% to 55.5% in the LLVIP dataset when entering the visible modality only. Consequently, having architectures that can seamlessly adjust to handle any individual or combined modalities during both training and inference is essential to ensure the robustness and dependability of IVOD detectors. In this paper, we devote our efforts to a comprehensive study on the modality incomplete IVOD problem from an architecture compatibility perspective, aiming to design a plug-and-play module which can flexibly adapt the detector to any single or double input modalities when equipped with existing detectors. To this end, we develop a compatible neck architecture alongside an effective training strategy to enhance performance and robustness. Specifically, we introduce a versatile neck, termed as Scarf Neck, tailored for DETR-like detectors. This component incorporates a straightforward Modality-Agnostic Deformable Attention (MADA) mechanism, which is designed to efficiently integrate features from multiple modalities or to enhance features from a single modality when the other is absent. In complete-modality scenarios, the method fully considers features from all modalities to dynamically select feature points in each modality and update the features of each modality separately. This approach, on the one hand, alleviates the prevalent spatial misalignment issue in multimodal images; on the other hand, it simultaneously achieves intra-modal feature enhancement and cross-modal feature fusion. In modality-missing scenarios, instead of simply duplicating the available modality to fit the subsequent model architecture, we align the model’s data flow with an additional set of reference points to enhance the feature representations of the remaining modality. To improve the robustness of the detector, we introduce a modality dropout mechanism to mimic the modality-missing scenarios during training. However, the naive modality dropout strategy reduces the diversity of training samples, thereby hindering model convergence. Therefore, we propose a pseudo modality dropout strategy that selectively disconnects modality connections for some sample pairs, enhancing both information utilization and performance in both complete and incomplete modality scenarios. Moreover, to fully evaluate the performance of modality incomplete IVOD task, we construct a comprehensive benchmark based on existing three IVOD datasets, dubbed FLIR-MI, M 3 FD-MI and LLVIP-MI. While the visible and the IR modality images alone on each dataset can serve as significant benchmarks, we further randomly discard 30%, 50% and 70% visible images and 70%, 50% and 30% IR images of all the paired images respectively to constitute FLIR-MI, M 3 FD-MI and LLVIP-MI with all single modality images to mimic the practical scenarios where the absent modality is either dominant or secondary. Furthermore, we conduct thorough experiments on the constructed three modality incomplete benchmarks with existing state-of-the-art IVOD detectors. Integrating the Scarf Neck and pseudo-modality dropout strategy, our proposed Scarf-DETR achieves significant performance gains in modality-incomplete scenarios including both the VIS/IR only extreme cases and the constructed three modality incomplete settings. Notably, our method also achieves superior or competitive performance compared with existing state-of-the-art IVOD detectors on three IVOD datasets. Figure 1 demonstrates that Scarf-DETR substantially outperforms DAMS-DETR ( Guo et al. 2024 ) across all testing settings, with a remarkable 40.5% mAP improvement in the VIS-only scenario on the LLVIP dataset. The main contributions of this paper are as follows: • We aim to conduct a comprehensive investigation into the modality-incomplete IVOD problem, with key research efforts including exploring this issue from the perspective of architecture compatibility and establishing a comprehensive evaluation benchmark. • We develop a Scarf Neck module which utilizes modality-agnostic deformable attention to accommodate the input of any single or double modalities. Combined with our pseudo-modality dropout training strategy, Scarf Neck can serve as a plug and play module with DETR-series detectors. • Our proposed approach yields significant performance gains in modality-incomplete scenarios across the three constructed modality-incomplete benchmarks, namely FLIR-MI, M 3 FD-MI and LLVIP-MI. Figure

Read the complete text