TriFusion-SR: Joint Tri-Modal Medical Image Fusion and SR
Fayaz Ali Dharejo, Sharif S. M. A, Aiman Khalil, Nachiket Chaudhary, Rizwan Ali Naqvi, Radu Timofte · 2026
Read TriFusion-SR: Joint Tri-Modal Medical Image Fusion and SR by Fayaz Ali Dharejo, Sharif S. M. A, Aiman Khalil, Nachiket Chaudhary, Rizwan Ali Naqvi, Radu Timofte (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09702.txt.
Introduction
Fayaz Ali Dharejo and colleagues propose TriFusion-SR, a diffusion system that fuses three medical image types and super-resolves them in one joint step. Splitting the stages breeds artifacts, and tri-modal mixes of anatomical and functional scans suffer strong frequency imbalances. A 2D discrete wavelet transform splits features into frequency bands, letting cross-modal interaction be tuned per band. Rectified wavelet terms calibrate the latent coefficients, and an adaptive fusion module then refines the result across space and frequency.
Catalog record
| CREATOR | Fayaz Ali Dharejo, Sharif S. M. A, Aiman Khalil, Nachiket Chaudhary, Rizwan Ali Naqvi, Radu Timofte |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20934 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.09702
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TriFusion-SR: Joint Tri-Modal Medical Image Fusion and SR Fayaz Ali Dharejo, Sharif S. M. A, Aiman Khalil, Nachiket Chaudhary, Rizwan Ali Naqvi, Radu Timofte Abstract Multimodal medical image fusion facilitates comprehensive diagnosis by aggregating complementary structural and functional information, but its effectiveness is limited by resolution degradation and modality discrepancies. Existing approaches typically perform image fusion and super-resolution (SR) in separate stages, leading to artifacts and degraded perceptual quality. These limitations are further amplified in tri-modal settings that combine anatomical modalities (e.g., MRI, CT) with functional scans (e.g., PET, SPECT) due to pronounced frequency-domain imbalances. We propose TriFusionSR, a wavelet-guided conditional diffusion framework for joint tri-modal fusion and SR. The framework explicitly decomposes multimodal features into frequency bands using the 2D Discrete Wavelet Transform, enabling frequency-aware cross-modal interaction. We further introduce a Rectified Wavelet Features (RWF) strategy for latent coefficient calibration, followed by an Adaptive Spatial-Frequency Fusion (ASFF) module with gated channel–spatial attention to enable structure-driven multimodal refinement. Extensive experiments demonstrate state-of-the-art performance, achieving 4.8–12.4% PSNR improvement and substantial reductions in RMSE and LPIPS across multiple upsampling scales. 1 Introduction Medical image fusion combines complementary multi-modal information to improve diagnostic reliability, providing the high-quality, detail-preserving images essential for clinical analysis [ 31 ] . Although integrating modalities such as PET, CT, MRI, and SPECT [ 23 ] yields rich anatomical and functional insights, their inherent disparities in resolution, contrast, and noise complicate the fusion process [ 21 ] . To counter this, fusion techniques have been extensively explored recently [ 10 ] . These methods could be broadly categorized into traditional [ 8 ] and deep learning (DL) approaches [ 11 ] . Traditional methods, such as sparse representation and transform-domain techniques, are efficient but often fail to preserve fine structures [ 4 , 26 , 2 ] . DL methods learn hierarchical cross-modal representations [ 30 ] ; however, the instability and mode collapse of GANs have motivated a shift toward diffusion models [ 16 , 3 ] . These models generate images via progressive denoising, exemplified by Zhao et al.’s DDPM-based hierarchical Bayesian framework [ 29 ] . Despite this progress, tri-modal medical image fusion remains underexplored, largely confined to traditional techniques like adaptive sparse representation [ 6 ] and BitonicX filtering [ 5 ] . Figure 1: Low–High Frequency Disparity Across MR and SPECT Modalities. Early DL-based SR methods have demonstrated strong capacity for fine-detail recovery with high fidelity [ 27 , 20 ] . In particular, Mao et al. adapted decoupled conditional diffusion frameworks for multi-contrast MRI SR, enabling uncertainty modeling and robust optimization [ 12 , 14 ] . However, performing SR and fusion in separate stages can propagate artifacts and degrade final image quality, motivating end-to-end approaches that jointly perform fusion and SR on low-resolution inputs [ 25 , 1 , 22 , 8 ] . Xiao et al. [ 22 ] introduced a multi-layer attention-based knowledge distillation framework for infrared–visible fusion and super-resolution, but it is neither designed for medical imaging nor applicable to tri-modal fusion. Tri-modal medical imaging, particularly combining MRI modalities (e.g., MR-T1 and MR-T2) with SPECT/PET further increases complexity due to modality-specific frequency-domain characteristics. For example, as shown in Fig. 1 , MRI retains broad high-frequency structural information, while functional signals like SPECT rapidly decay at higher frequencies. This imbalance necessitates frequency-aware handling of low-frequency structural components and high-frequency textures during fusion, which is often overlooked by conventional methods [ 9 ] . To address this, we formulate tri-modal fusion and SR by explicitly decomposing features into frequency bands prior to fusion, enabling modality-consistent integration of structural and functional information across diverse imaging combinations. Building on this formulation, we propose TriFusion-SR, a wavelet-guided diffusion model that leverages 2D-DWT for frequency-aware conditioning, enabling simultaneous tri-modal fusion and super-resolution in an end-to-end manner. Our main contributions are summarized as follows: 1. We introduce a novel wavelet diffusion framework that takes advantage of the low-frequency structure and high-frequency details prior to cross-modal fusion. To our knowledge, this is the first end-to-end model to incorporate 2D-DWT into joint tri-modal fusion and SR. 2. We propose a RWF strategy to calibrate wavelet coefficients in latent space followed by ASFF module with gated channel–spatial attention to enable structure-driven multimodal fusion. 3. We achieve state-of-the-art performance across three challenging upsampling scales ( 2\times , 4\times , 8\times ), with 4.8–12.4% PSNR improvement , 11–33% RMSE reduction , and 52–65% LPIPS reduction over existing methods on Joint tri-modal fusion-SR. 2 Methodology 2.1 Framework Overview We propose TriFusion-SR, a multimodal image SR framework based on conditional denoising diffusion probabilistic models. The method incorporates a 2DWT to decompose input images into high- and low-frequency components, enabling effective extraction of structural details and texture information. By leveraging frequency-aware fusion within the diffusion process, the model generates high-resolution fused images with enhanced edge preservation and richer fine-grained textures, which are critical for multimodal SR tasks. As illustrated in Fig. 2 , TriFusion-SR achieves accurate tri-modal medical image fusion and SR through a forward and reverse Markov chain diffusion process. Figure 2: Overview of the proposed TriFusion-SR Framework. Taking the tri-modal fusion of MR-T1, MR-T2, and SPECT images as an example, the low-resolution inputs from the three modalities are denoted as \mathbf{x}\in\mathbb{R}^{H\times W} , \mathbf{y}\in\mathbb{R}^{H\times W} , and \mathbf{s}\in\mathbb{R}^{H\times W} , respectively. The generated high-resolution fused image is denoted by \mathbf{I}_{0}\in\mathbb{R}^{3\times H\times W} . The three modal images \mathbf{x} , \mathbf{y} , and \mathbf{s} are first upsampled to the target resolution via bicubic interpolation, producing \tilde{\mathbf{x}} , \tilde{\mathbf{y}} , and \tilde{\mathbf{s}} . Subsequently, each modality is decomposed using a two-dimensional discrete wavelet transform (2D-DWT) to extract low-frequency and high-frequency components: \left\{\mathbf{x}_{L},\mathbf{x}_{H}\right\}=\mathcal{X}(\tilde{\mathbf{x}}),\quad\left\{\mathbf{y}_{L},\mathbf{y}_{H}\right\}=\mathcal{X}(\tilde{\mathbf{y}}),\quad\left\{\mathbf{s}_{L},\mathbf{s}_{H}\right\}=\mathcal{X}(\tilde{\mathbf{s}}), (1) where \mathcal{X}(\cdot) denotes the 2D-DWT operator. The decomposed frequency components are then fed into the ASFF block for feature extraction, yielding. \mathbf{z}_{t}=\varepsilon(\mathbf{x}_{L},\mathbf{x}_{H},\mathbf{y}_{L},\mathbf{y}_{H},\mathbf{s}_{L},\mathbf{s}_{H}). (2) The extracted feature \mathbf{z}_{t} is concatenated along the channel dimension with the noisy sample \mathbf{I}_{t}\sim\mathcal{N}(0,\mathbf{I}) : \mathbf{h}_{t}=\text{Concat}(\mathbf{I}_{t},\mathbf{z}_{t}). (3) The objective function optimized by Wave-TriFusion is defined as: \mathcal{L}_{\text{(TFS)}}=\mathbb{E}_{\mathbf{x},\mathbf{y},\mathbf{s},\boldsymbol{\epsilon}\sim\mathcal{N}(0,\mathbf{I}),t}\left[\left\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\theta}\left(\mathbf{z}_{t},\mathbf{I}_{t},\gamma_{t}\right)\right\|_{2}^{2}\right], (4) where \gamma_{t} controls the noise variance in the diffusion process, and \epsilon_{\theta} denotes the U-Net [ 17 ] parameterized denoising network. The conditioning feature \mathbf{z}_{t} is produced by the Wavelet-ASFF hybrid module. Our framework adopts the diffusion backbone of TMFS [ 23 ] , which is based on a U-Net architecture SR3 [ 18 ] . We retain the core diffusion architecture and enrich the conditioning branch with wavelet decomposition, RWF and ASFF, to achieve frequency-aware multimodal fusion. 2.2 Wavelet Decomposition The discrete wavelet transform (DWT) is widely used to decompose an image into low-frequency (LF) and high-frequency (HF) sub-bands. In this work, we adopt the Haar wavelet [ 7 ] . Given an input image \mathbf{I}\in\mathbb{R}^{H\times W\times 3} , we apply the DWT operation, denoted as \mathrm{DWT}(\cdot) . This produces one low-frequency sub-band \mathbf{x}_{L}^{(1)}\in\mathbb{R}^{\frac{H}{2}\times\frac{W}{2}\times 3} and three high-frequency sub-bands \{\mathbf{x}_{V}^{(1)},\mathbf{x}_{H}^{(1)},\mathbf{x}_{D}^{(1)}\}\in\mathbb{R}^{\frac{H}{2}\times\frac{W}{2}\times 3} : \mathbf{x}_{L}^{(1)},\mathbf{x}_{V}^{(1)},\mathbf{x}_{H}^{(1)},\mathbf{x}_{D}^{(1)}=\mathrm{DWT}(\mathbf{I}). (5) The decomposition can be recursively applied to the low-frequency sub-band \mathbf{x}_{L}^{(1)} , resulting in: \mathbf{x}_{L}^{(2)},\mathbf{x}_{V}^{(2)},\mathbf{x}_{H}^{(2)},\mathbf{x}_{D}^{(2)}=\mathrm{DWT}(\mathbf{x}_{L}^{(1)}). (6) After performing the DWT up to the J -th level, we obtain: \{\mathbf{x}_{L}^{(J)},\mathbf{x}_{V}^{(J)},\mathbf{x}_{H}^{(J)},\mathbf{x}_{D}^{(J)}\}\in\mathbb{R}^{\frac{H}{2^{J}}\times\frac{W}{2^{J}}\times 3}. (7) At each level, the low-frequency sub-band is recursively replaced by its decomposed components [ 13 ] . Therefore, after J levels, the complete set of sub-bands is: \left\{\mathbf{x}_{L}^{(J)},\ \mathbf{x}_{V}^{(J)},\ \mathbf{x}_{H}^{(J)},\ \mathbf{x}_{D}^{(J)},\ \mathbf{x}_{V}^{(J-1)},\ \ldots,\