Thermo-VL : Extending Vision–Language Models to Thermal Infrared Perception

Rusiru Thushara, Yasiru Ranasinghe, Jay Paranjape, Vishal M. Patel · 2026

Read Thermo-VL : Extending Vision–Language Models to Thermal Infrared Perception by Rusiru Thushara, Yasiru Ranasinghe, Jay Paranjape, Vishal M. Patel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21882.txt.

Introduction

Thushara and co-authors build Thermo-VL, a vision-language model that can read heat images as well as ordinary photos. Normal models fail in dim light because they learn mostly from visible-light pictures, while thermal infrared still shows scene shape. Their design adds a trainable heat encoder to a frozen Molmo-7B base, fused by text-guided dual attention. Tests show strong gains on heat-only and combined reasoning tasks, and the team also releases a paired instruction dataset and a screened benchmark.

Catalog record

CREATORRusiru Thushara, Yasiru Ranasinghe, Jay Paranjape, Vishal M. Patel
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45268
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21882

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Thermo-VL : Extending Vision–Language Models to Thermal Infrared Perception Rusiru Thushara, Yasiru Ranasinghe, Jay Paranjape, Vishal M. Patel Abstract Vision-language models (VLMs) often fail under low illumination because their visual grounding is learned predominantly from RGB imagery, whereas thermal infrared preserves complementary scene structure when visible cues degrade. We present Thermo-VL , a wavelength-aware VLM that augments a frozen Molmo-7B backbone with a trainable thermal encoder and a text-guided dual-attention fusion module. Given aligned RGB tokens, thermal tokens, and prompt embeddings, the fusion module conditions thermal features on both language and RGB context, then injects a gated residual into the frozen RGB stream so thermal evidence can be incorporated without disrupting Molmo’s pretrained RGB–language interface. We train the model with the standard language-modeling objective together with auxiliary alignment and regularization losses that improve cross-modal grounding and reduce over-reliance on RGB. We also introduce a pixel-aligned RGB–thermal instruction-tuning dataset and Thermo-VL-Bench, a manually screened RGB–thermal VQA benchmark for low-light and cross-spectrum reasoning. Experiments show strong gains on challenging thermal-only and RGB+thermal reasoning tasks, highlighting the value of prompt-conditioned multispectral fusion. Our dataset and code are publicly available at: thusharakart.github.io/Thermo-VL 1 Introduction Recent advances in vision-language models (VLMs) have dramatically improved open-ended image understanding, captioning, and visual question answering by combining large-scale multimodal pretraining with instruction tuning 8 ; 19 ; 12 . However, most current VLMs are trained almost exclusively on visible-spectrum RGB imagery and therefore inherit the failure modes of RGB sensing. Under low illumination, glare, haze, smoke, or other adverse conditions, RGB observations can lose critical scene evidence even when reliable reasoning is most needed. Thermal infrared provides complementary information in precisely these settings. Because it captures radiometric structure rather than reflected visible light, thermal imagery can preserve salient cues such as humans, animals, and warm vehicles when visible features degrade 31 . This mismatch makes RGB-only VLMs brittle in safety-critical scenarios such as nighttime driving, search-and-rescue, and low-visibility inspection. A common workaround is to translate infrared imagery into pseudo-RGB and then apply an RGB-trained VLM 25 . However, translation-based pipelines introduce a generative bottleneck, may hallucinate visible-style details, and can suppress modality-specific information that is not naturally represented in RGB space. A more principled alternative is to integrate thermal observations directly inside the VLM. The challenge is to do so without destabilizing the pretrained RGB–language interface or requiring expensive retraining of the backbone. Progress is further limited by the lack of language-supervised RGB–thermal data: widely used paired datasets such as KAIST, FLIR-ADAS, and OSU/OTCBVS largely target detection or tracking and provide little to no VQA-style supervision. In this work, we present Thermo-VL , a wavelength-aware VLM built on Molmo-7B-D 8 . Our model combines a frozen RGB vision tower with a partially trainable thermal encoder and a text-guided dual-attention fusion module. Given aligned RGB tokens, thermal tokens, and prompt embeddings, the fusion module first refines thermal features using both the question and RGB context, then injects the resulting information into the frozen RGB stream through a token-wise gated residual. This design makes fusion question-aware, preserves compatibility with Molmo’s pretrained language stack, and supports parameter-efficient adaptation by updating only the last N_{\ell} blocks of the thermal encoder together with the fusion parameters. The overall architecture is shown in Fig. 1 (A), and the detailed fusion block is shown in Fig. 1 (B). To support instruction tuning and standardized evaluation in this setting, we also introduce a unified RGB–thermal data suite. We construct a pixel-aligned RGB–thermal instruction-tuning dataset with question–answer pairs spanning scene understanding, counting, spatial relations, and safety-relevant cues. We further introduce Thermo-VL-Bench , a human-verified RGB–thermal VQA benchmark designed for low-light and cross-spectrum reasoning. Experiments on Thermo-VL-Bench and RGB-Th Bench show that explicit prompt-conditioned RGB–thermal fusion is especially helpful when both modalities are available. Relative to unmodified VLM baselines evaluated with paired inputs, Thermo-VL preserves RGB-only performance while improving thermal-only and RGB+thermal settings. These results suggest that thermal cues can complement RGB when fused inside the model rather than appended at the input level. Contributions. Our main contributions are summarized as follows: • Wavelength-aware VLM with question-aware fusion: We introduce Thermo-VL , a Molmo-based RGB–thermal VLM that integrates paired RGB and thermal inputs through a text-guided dual-attention fusion module with token-wise gated residual injection, while keeping the RGB tower and language model frozen. • Parameter-efficient multispectral adaptation: We fine-tune only the last N_{\ell} blocks of the thermal encoder together with the thermal LayerNorm and fusion module, and regularize training with token-level RGB–thermal alignment, visual–thermal–text contrastive supervision, gate-entropy regularization, and block-wise RGB masking. • Unified RGB–thermal language supervision: We introduce a pixel-aligned RGB–thermal instruction-tuning dataset and Thermo-VL-Bench, a manually screened binary benchmark built from real aligned image pairs, to support training and initial evaluation in low-light cross-spectrum settings. Figure 1: Thermo-VL overview. (A) Overall pipeline: a frozen RGB vision tower and a finetuned thermal encoder produce aligned patch tokens from paired RGB and thermal inputs. Prompt embeddings guide a text-aware fusion module, and the frozen language model processes the fused visual tokens to generate the final output. (B) Text-guided dual-attention fusion block: thermal tokens attend to both prompt embeddings and RGB tokens; the refined thermal representation predicts residual and token-wise gates, and the gated residual is injected into RGB tokens to form fused visual tokens \mathbf{R}^{\dagger} . 2 Related Work Vision-language models have advanced rapidly, enabling open-ended captioning and visual question answering (VQA) on RGB images through instruction tuning on large-scale datasets such as PixMo 8 . Models like Molmo-7B, which Thermo-VL directly extends, fuse a frozen ViT vision tower with a language model connected via a projector, trained on detailed captions and synthetic QA pairs 8 . These models perform well in natural lighting but typically overlook wavelength-specific inputs, limiting performance in low-light or adverse weather. Thermal and Infrared Language Understanding. Thermal and infrared language understanding remains underexplored, with limited prior work on direct thermal input processing for captioning and VQA 15 . Existing approaches often translate thermal images to pseudo-RGB before applying RGB-trained VLMs, or use language models to mine captions from infrared imagery without native cross-spectrum fusion 15 . These strategies do not directly align thermal observations with language and therefore remain limited in settings where visible-spectrum cues degrade. RGB–Thermal Fusion Mechanisms. RGB–thermal fusion has been widely studied in detection and segmentation, where attention and gating modules dynamically combine visible and thermal cues 10 ; 26 ; 30 ; 28 ; 16 ; 24 . Most of this literature focuses on closed-set perception tasks such as object detection or pedestrian recognition. In contrast, our setting requires open-ended language generation and question answering within a pretrained VLM, where fusion must preserve the existing RGB–language interface. Visible-to-Thermal Translation. Visible-to-thermal translation has emerged as a useful tool for data augmentation and cross-spectrum synthesis 22 ; 29 ; 21 ; 2 ; 17 . While such methods can expand training data, they are not a substitute for native multispectral reasoning because the downstream model still needs to ground thermal evidence in language. In this work, we use synthetic thermal generation as augmentation only, while the proposed model reasons directly over paired RGB and thermal inputs. RGB–Thermal Datasets. Public paired datasets such as LLVIP 14 , KAIST 13 , FLIR 27

Read the complete text