DualVision : RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning
Abrar Majeedi, Zhiyuan Ruan, Ziyi Zhao, Hongcheng Wang, Jianglin Lu, Yin Li · 2026
Read DualVision : RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning by Abrar Majeedi, Zhiyuan Ruan, Ziyi Zhao, Hongcheng Wang, Jianglin Lu, Yin Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18829.txt.
Introduction
Majeedi and five colleagues propose DUALVISION, a lightweight fusion module that brings infrared information into multimodal large language models through patch level localized cross attention. Such models are fragile under fog, blur, and low light, while infrared imaging stays robust in these conditions. The team also introduces DV-204K, about 25,000 aligned infrared-RGB image pairs with 204,000 modality specific question answer annotations, and DV-500, a 500 pair benchmark for cross modal reasoning. DUALVISION performs strongly under a wide range of visual degradations.
Catalog record
| CREATOR | Abrar Majeedi, Zhiyuan Ruan, Ziyi Zhao, Hongcheng Wang, Jianglin Lu, Yin Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47014 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18829
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DualVision : RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning Abrar Majeedi, Zhiyuan Ruan, Ziyi Zhao, Hongcheng Wang, Jianglin Lu, Yin Li Abstract Multimodal large language models (MLLMs) have achieved impressive performance on visual perception and reasoning tasks with RGB imagery, yet they remain fragile under common degradations, such as fog, blur, or low-light conditions. Infrared (IR) imaging, a well-established complement to RGB, offers inherent robustness in these conditions, but its integration into MLLMs remains underexplored. To bridge this gap, we propose DualVision , a lightweight fusion module that efficiently incorporates IR–RGB information into MLLMs via patch-level localized cross-attention. To support training and evaluation and to facilitate future research, we also introduce DV-204K , a dataset of ∼ \sim 25K publicly available aligned IR–RGB image pairs with 204 K 204K modality-specific QA annotations, and DV-500 , a benchmark of 500 500 IR–RGB image pairs with 500 QA pairs designed for evaluating cross-modal reasoning. Leveraging these datasets, we benchmark both open- and closed-source MLLMs and demonstrate that DualVision delivers strong empirical performance under a wide range of visual degradations. Our code and dataset are available at https://abrarmajeedi.github.io/dualvision . Figure 1 : When visibility degrades in RGB images ( e.g . , at night), MLLMs struggle to “see and reason”, limiting their reliability in many real-world applications such as autonomous driving. By complementing RGB with infrared data, DualVision enables robust visual perception and reasoning while reducing computation by \sim 75% compared to naïve fusion. 1 Introduction Multimodal large language models (MLLMs) 42 represent an increasingly important class of vision language models (VLMs) that connect visual and textual data through large language models (LLMs). MLLMs have achieved strong performance across many visual recognition tasks such as object recognition, visual grounding, and visual question answering, enabling broad applications in robotics 15 , autonomous driving 7 , and digital health 2 . Most existing MLLMs, however, rely exclusively on RGB imagery as their visual input, drawing from large-scale web datasets composed primarily of well-lit, natural scenes. Although RGB imagery provides rich color and texture information and has driven impressive generalization in standard benchmarks, it exposes a critical weakness: their reliability drops sharply when inputs are degraded by adverse visual conditions 35 . This vulnerability stems from RGB’s dependence on visible light and its susceptibility to optical distortions. Common examples include low-light environments, motion- or defocus-induced blur, and non-ideal weather such as rain or fog. These degradations are not rare anomalies but frequent realities in practical deployment, particularly in domains such as transportation, surveillance, and health, where robustness is paramount. For example, autonomous vehicles must sustain robust perception at night and in adverse weather, while home-based health monitoring systems must function effectively under poor lighting and motion blur. Infrared (IR) imaging offers a valuable complement: by capturing electromagnetic radiation beyond the visible spectrum, IR can remain effective in darkness, fog, and other challenging environments faced by RGB imagery 29 ; 10 . However, IR imagery lacks the fine-grained appearance details and semantic richness that RGB captures under favorable conditions. Fusing RGB and IR signals thus provides a promising pathway towards more robust visual perception and reasoning by leveraging their complementary strengths. This fusion has been widely explored in traditional vision tasks such as recognition, detection, and segmentation 41 ; 43 ; 40 , and as a strategy for mitigating degradations through complementary sensing 33 ; 28 . However, its integration into MLLMs, particularly to overcome the limitations of RGB under degraded visual conditions , remains largely underexplored to date. Developing MLLMs that can jointly comprehend RGB and IR data faces three key challenges. First , there is no principled design for a fusion mechanism that maintains spatial alignment between RGB and IR modalities while adaptively prioritizing the informative signals for MLLMs. Second , progress in IR-RGB perception is hindered by the scarcity of large-scale, semantically rich datasets. Existing datasets, often developed for detection or segmentation, tend to be narrow in scope, lack linguistic annotations, and are not aligned with the instruction-tuning paradigm that drives recent advances in MLLMs. Third , the absence of standardized IR-RGB benchmarks for vision language tasks leads to inconsistent evaluation, particularly under visual degradations, making it difficult to rigorously assess robustness across varying visual conditions. In this paper, we address key challenges in developing IR-RGB MLLMs for robust visual reasoning. As illustrated in Fig. 1 , we propose DualVision , a lightweight fusion approach that leverages multi-scale localized cross-attention to selectively route information between aligned IR and RGB tokens. Our design exploits the inherent spatial structure of visual data by employing progressively expanding local attention radii across different fusion levels. This hierarchical approach enables precise local correspondence at fine scales while capturing broader contextual relationships at coarser ones. The result is a unified IR-RGB representation that avoids the quadratic overhead of naïve concatenation, while the localized cross-attention mechanism provides inductive biases that strengthen cross-modal alignment, particularly under degraded RGB conditions. Complementing our approach, we introduce DV-204K and DV-500 , a new dataset suite designed to facilitate both instruction tuning and systematic evaluation of MLLM’s robustness under visual degradations. Our datasets provide diverse, well-aligned IR-RGB image pairs with modality-aware question-answer pairs, allowing the study of both general reasoning and degradation-specific performance. Our contributions are summarized as follows. 1. Our work is among the first to develop MLLMs that integrate RGB and IR modalities for robust visual reasoning under visual degradations ( e.g . , blur, low-light, and fog). 2. We introduce DualVision , a lightweight IR-RGB fusion module with enhanced robustness to degradations, while remaining compatible with existing MLLMs. 3. To support training and evaluation of IR-RGB MLLMs, we create and release two datasets: 1) DV-204K : A dataset of \sim 25K aligned IR–RGB image pairs with \sim 204K modality-aware question-answer annotations designed for instruction tuning and 2) DV-500 : A carefully curated evaluation benchmark featuring 500 IR-RGB image pairs with 500 associated QA pairs. 4. Leveraging our datasets and through extensive experiments, we demonstrate strong empirical results of DualVision under various degradations. 2 Related Work VLMs and MLLMs. Modern VLMs align visual and textual modalities through contrastive and generative pretraining. CLIP 27 pioneers large-scale contrastive learning, enabling open-vocabulary recognition. Recent efforts, including ImageBind 9 , LanguageBind 44 extend this paradigm to additional modalities including IR, depth and audio. However, these works focus on broad modality binding 9 ; 44 , rather than more complex reasoning. More recent MLLMs such as LLaVA 19 , BLIP-2 17 , and Flamingo 1 integrate vision encoders with LLMs to support open-ended visual reasoning. Instruction tuning further improved alignment with human intent. InstructBLIP 8 , MiniGPT-4 45 , and LLaVA-1.5 18 demonstrated that curated instruction datasets enhance reasoning and zero-shot generalization. While most MLLMs rely solely on RGB imagery, a few very recent works, including IR-LLaVA 14 and IRGPT 6 , have begun exploring IR-based MLLMs. However, these approaches operate on IR-only inputs and discard RGB, overlooking the complementary sensing capabilities of the two modalities. In contrast, we aim to develop IR–RGB MLLMs that fully exploit cross-modal synergy for robust visual reasoning. Figure 2 : Overview of DualVision . (a) shows how DualVision integrates into a MLLM to fuse RGB and IR image tokens for