Bringing Multimodal Large Language Models to Infrared-Visible Image Fusion Quality Assessment
Yuchen Guo, Junli Gong, Yao Lu, Xintong Xu, Yiuming Cheung, Weifeng Su · 2026
Read Bringing Multimodal Large Language Models to Infrared-Visible Image Fusion Quality Assessment by Yuchen Guo, Junli Gong, Yao Lu, Xintong Xu, Yiuming Cheung, Weifeng Su (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06969.txt.
Introduction
Yuchen Guo and five co-authors tackle how to judge fused infrared and visible images. Fusion combines heat information with visible detail, but common scoring methods can reward the wrong qualities. Human ratings help, yet reducing them to one averaged score loses disagreement between viewers.
Their system, FuScore, uses a multimodal language model to give a continuous quality score, closer to how people perceive images. Training labels reflect how much raters agreed on each image, and extra losses preserve ordering between methods and scenes. In experiments, FuScore matches human preferences better than earlier approaches, reaching leading results on the measures reported.
Catalog record
| CREATOR | Yuchen Guo, Junli Gong, Yao Lu, Xintong Xu, Yiuming Cheung, Weifeng Su |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06969
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bringing Multimodal Large Language Models to Infrared-Visible Image Fusion Quality Assessment Yuchen Guo, Junli Gong, Yao Lu, Xintong Xu, Yiuming Cheung, Weifeng Su Abstract Infrared-Visible image fusion (IVIF) aims to integrate thermal information and detailed spatial structures into a single fused image to enhance perception. However, existing evaluation approaches tend to over-optimize both hand-crafted no-reference statistics and full-reference metrics that treat the source images as pseudo ground truths. Recent IVIF reward-modelling efforts learn from human ratings but use scalar regression on aggregated scores, neither leveraging the reasoning of Multimodal Large Language Models (MLLMs) nor encoding per-image perceptual ambiguity in their supervision, but naively introducing MLLMs with discrete one-hot supervision likewise collapses fused images of similar quality into different rating levels. To address this, we introduce FuScore , which utilizes an MLLM to mimic human visual perception by producing continuous quality score, rather than discrete level predictions, enabling fine-grained discrimination among fused images of similar quality. We exploit the agreement among four IVIF-specific sub-dimensions to construct a per-image soft label whose sharpness reflects how consensual the overall judgment is. We further introduce a tripartite objective combining per-image distributional supervision, within-source-pair Thurstone fidelity for method-level ordering, and cross-source-pair Thurstone fidelity for scene-level ordering across scenes. Extensive experiments demonstrate that FuScore achieves state-of-the-art correlation with human visual preferences. 1 Introduction Infrared-Visible image fusion (IVIF) is an important technique in image processing, which aims to combine both thermal and structural information from two source images into a single fused image [ 28 , 22 , 11 ] . By integrating complementary thermal and structural cues into a unified image representation, IVIF has been widely applied in a variety of real-world scenarios, including security surveillance, military target detection, scene understanding, and assisted driving. Existing IVIF models can be broadly categorized into two approaches: discriminative methods such as carefully designed CNN-based and Transformer-based models [ 29 , 16 ] ; generative approaches, including autoencoder-based methods [ 6 , 30 ] , GAN-based [ 12 , 17 ] and diffusion-based methods [ 19 , 31 ] . With the rapid advancement of deep learning, above IVIF models have achieved substantial progress in improving fusion quality. However, a fundamental limitation remains: these existing methods are still predominantly optimized for hand-crafted traditional no-reference statistics such as Entropy (EN) [ 18 ] and Mutual Information (MI) [ 4 ] , or toward full-reference objectives that treat source image pairs as pseudo ground truths, such as Structural Similarity Index (SSIM) [ 23 ] and Peak Signal-to-Noise Ratio (PSNR) [ 7 ] , due to the absence of authentic ground-truth fused images in IVIF. As a result, these objectives often fail to faithfully reflect human experts perceptual preference, and may even assign higher scores to a visually inferior fused image (Fig. 1 ). Figure 1: Two limitations of current IVIF quality assessment. (a) Misranking within the same scene: for a single source pair, traditional no-reference metrics (VIF, EN, SD, SF) rank the human-disfavoured fused image B above the human-preferred A (MOS \mu_{A}{=}3.50>\mu_{B}{=}2.75 ). (b) One-hot MLLM-as-Judge lacks fidelity: two fused images with near-tie human MOS ( 3.75 vs. 3.50 ) are forced into distinct discrete levels ( 4 vs. 3 ) by an MLLM trained with one-hot supervision over five quality levels. A recent IVIF reward-modelling effort, EVAFusion [ 13 ] , addresses the human-alignment gap by training a ViT-based regressor with MSE on aggregated overall scores. Yet it neither leverages the linguistic reasoning of MLLMs nor encodes how consensual each image’s mean score is across the underlying multi-criterion judgments. Another naive remedy of introducing MLLMs with the discrete one-hot supervision used in prior MLLM-based IQA [ 25 ] is unable to capture subtle perceptual differences and carries no signal about per-image annotation consensus either. This leaves a fundamental question largely unanswered: how can MLLMs produce continuous IVIF quality scores under supervision that encodes per-image annotation consensus? We answer this question with FuScore , an MLLM-based scorer that yields a continuous quality score for each fused image rather than forcing it into a single discrete level. Specifically, instead of training the MLLM to emit a single quality token, we read out the decoding probabilities over the predefined quality-level tokens and supervise this distribution with a soft label, so that the final score, computed as the expectation under the predicted distribution, can take any real value within the rating range and capture subtle quality differences that one-hot supervision collapses. A key challenge, however, lies in how to construct the soft label itself: naively smoothing a discrete overall score with a fixed-width Gaussian ignores the fact that fused images vary substantially in how consensual their perceptual quality is. To address this, we exploit the structure of existing IVIF annotations, where each fused image is labeled along four sub-dimensions: thermal retention, texture preservation, artifacts, and sharpness, and use the agreement among these sub-dimensions as a per-image consensus signal. When the four criteria agree, the soft label is sharply concentrated around the overall score; when they disagree, the label is broadened to reflect that the overall judgment is less consensual across reasonable aggregation rules. We further observe that subjective fusion quality exhibits both within-source-pair variation, which captures how different fusion methods perform on the same scene, and cross-source-pair variation, which reflects scene-dependent difficulty. Since neither per-image supervision nor within-scene comparisons alone can cover both variation components, FuScore adopts a tripartite objective that jointly enforces per-image distributional supervision, within-source-pair Thurstone fidelity for method-level ordering, and cross-source-pair Thurstone fidelity for scene-level ordering, with the per-pair margin reusing the per-image label widths derived from sub-dimension agreement. Our core contributions are summarized as: • We propose FuScore, the first MLLM-based IVIF quality scorer trained with distribution-level supervision, generating continuous score of fused images. • We introduce a sub-dimension-aware soft label construction that modulates label sharpness by the agreement among four IVIF-specific sub-dimensions, allowing the supervision target to faithfully reflect how consensual the overall perceptual judgment is on each fused image. • We design a tripartite training objective that jointly enforces per-image distributional supervision, within-source-pair Thurstone fidelity for method-level ordering, and cross-source-pair Thurstone fidelity for scene-level ordering, with the per-pair margin reusing the per-image label widths derived from sub-dimension agreement. • On the EVAFusion benchmark and a 5-expert Semantic RT pilot, FuScore achieves state-of-the-art correlation with human visual preferences across all reported rank metrics, and empirically tracks both data-side ranking difficulty and human-side inter-rater disagreement. 2 Related Works Infrared-Visible Image Fusion. Infrared-visible image fusion (IVIF) is a crucial field of low-level computer vision. Early CNN-based encoder-decoder architectures such as IFCNN [ 29 ] rely on hand-designed fusion rules, while Transformer-based models like SwinFusion [ 16 ] capture long-range cross-modality dependencies via self-attention. Generative approaches instead model the data distribution of fused images: autoencoders [ 30 , 6 , 5 ] learn modality-specific representations, GANs [ 17 , 12 ] encourage perceptual realism via adversarial training, and diffusion [ 31 , 19 ] formulates fusion as conditional generation. Despite diverse architectures, all optimize either no-reference statistics or pseudo-reference objectives derived from source pairs, both poorly correlated with human perception, motivating a perceptually aligned quality scorer. Image Quality Assessment. IQA methods [ 23 , 24 , 26 , 21 , 10 ] predict