SWIFT: A General Sensitive Weight Identification Framework for Fast Sensor-Transfer Pansharpening

Zeyu Xia, Chenxi Sun, Tianyu Xin, Yubo Zeng, Haoyu Chen, Liang-Jian Deng · 2025

Read SWIFT: A General Sensitive Weight Identification Framework for Fast Sensor-Transfer Pansharpening by Zeyu Xia, Chenxi Sun, Tianyu Xin, Yubo Zeng, Haoyu Chen, Liang-Jian Deng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20311.txt.

Introduction

Zeyu Xia and co-authors present SWIFT, a 2025 framework that adapts image fusion models to new satellite sensors in about a minute. Pansharpening merges a sharp gray image with a soft color one to make sharp color maps. Models trained on one sensor fade badly on another, and retraining is slow.

SWIFT picks just three percent of the new sensor's data, chosen to span its shape well. Probing the old model with that subset flags the weights most sensitive to the shift. Only those weights get updated. On a single graphics card, adaptation falls from hours to near a minute. The plug-in frame fits many fusion models, and adapted models beat their stale versions by clear margins.

Catalog record

CREATORZeyu Xia, Chenxi Sun, Tianyu Xin, Yubo Zeng, Haoyu Chen, Liang-Jian Deng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-20311

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SWIFT: A General Sensitive Weight Identification Framework for Fast Sensor-Transfer Pansharpening Zeyu Xia, Chenxi Sun, Tianyu Xin, Yubo Zeng, Haoyu Chen, Liang-Jian Deng Abstract Pansharpening aims to fuse high-resolution panchromatic (PAN) images with low-resolution multispectral (LRMS) images to generate high-resolution multispectral (HRMS) images. Although deep learning-based methods have achieved promising performance, they generally suffer from severe performance degradation when applied to data from unseen sensors. Adapting these models through full-scale retraining or designing more complex architectures is often prohibitively expensive and impractical for real-world deployment. To address this critical challenge, we propose a fast and general-purpose framework for cross-sensor adaptation, SWIFT (Sensitive Weight Identification for Fast Transfer). Specifically, SWIFT employs an unsupervised sampling strategy based on data manifold structures to balance sample selection while mitigating the bias of traditional Farthest Point Sampling, efficiently selecting only 3% of the most informative samples from the target domain. This subset is then used to probe a source-domain pre-trained model by analyzing the gradient behavior of its parameters, allowing for the quick identification and subsequent update of only the weight subset most sensitive to the domain shift. As a plug-and-play framework, SWIFT can be applied to various existing pansharpening models. Extensive experiments demonstrate that SWIFT reduces the adaptation time from hours to approximately one minute on a single NVIDIA RTX 4090 GPU. The adapted models not only substantially outperform direct-transfer baselines but also achieve performance competitive with, and in some cases superior to, full retraining, establishing a new state-of-the-art on cross-sensor pansharpening tasks for the WorldView-2 and QuickBird datasets. Figure 1: The SWIFT framework and its performance impact. Top: The two-step strategy of SWIFT. Bottom: HQNR comparison of representative models with and without SWIFT enhancement. Introduction High-resolution multispectral (HRMS) images are essential for applications like urban planning and environmental monitoring ( Crampton et al. 2013 ; Fitzner et al. 2013 ) . However, due to hardware limitations, satellites such as WorldView-2 (WV2) and Gaofen-2 (GF2) cannot capture images with both high spatial resolution and rich spectral information. To resolve this trade-off, these systems acquire two complementary data types: high-resolution panchromatic (PAN) and low-resolution multispectral (LRMS) images. The process of fusing these two kinds of images to generate high-resolution multispectral (HRMS) images is known as pansharpening. Over the past few decades, pansharpening methods have evolved considerably, transitioning from traditional approaches to modern deep learning-based methods. Traditional approaches can be roughly categorized into three primary classes: component substitution (CS) ( Choi et al. 2005 ; Vivone et al. 2015a ) , multiresolution analysis (MRA) ( Otazu et al. 2005 ; Vivone, Restaino, and Chanussot 2018 ) , and variational optimization (VO) ( Wu et al. 2025 ; Tian et al. 2022 ) . More recently, deep learning techniques have leveraged the powerful feature extraction capabilities of neural architectures. Models based on Convolutional Neural Networks (CNN) ( Masi et al. 2016 ; He, Zhong, and Ma 2019 ; Yang et al. 2023 ) , Transformer ( Li et al. 2024 ; Wu et al. 2025 ; Zhang and Ma 2021 ; Zhou, Liu, and Wang 2022 ) , and diffusion ( Meng et al. 2023 ; Zhong et al. 2024 ; Rui et al. 2024 ) have already produced superior fusion results, significantly outperforming traditional methods. Although deep learning methods excel at pansharpening, they generally suffer from severe performance degradation when applied to target domain data that exhibit a distribution shift from the source domain. As satellite technology continues to diversify remote sensing data, this problem becomes increasingly critical. To address this cross-domain generalization challenge, current strategies typically fall into a fundamental performance-cost dilemma. One line of work attempts to improve inherent robustness by designing more complex architectures built upon the original models; however, this strategy not only significantly increases model parameters and computational load, but also prevents simple, weight-only updates on deployed hardware, necessitating a full re-deployment cycle, and offers no guarantee of improved generalization, as demonstrated by the results of models like FusionNet ( Quan, Hildebrand, and Jeong 2021 ) and ADWM ( Huang et al. 2025a ) in our experiments. Another line of work relies on expensive retraining on the target domain or zero-shot methods. While effective, the substantial cost of time or computational resource is impractical in many application scenarios. Based on the preceding analysis, two key challenges remain in enhancing generalization: (1) how to reduce the cost of computational resources and time, and (2) how to preserve the original model architecture. To our knowledge, no existing method can effectively address these two mutually constraining challenges simultaneously. Therefore, this paper introduces a novel task:how to improve a model’s generalization capability at an extremely low computational and time cost, while preserving the original model architecture. To address this task, we propose a model-agnostic enhancement framework —SWIFT. The core of this framework lies in a two-step “targeted identification” strategy as is shown in Figure 1 , encompassing data selection, followed by parameter selection and model adaptation. Specifically, in the data selection step, we propose a Density-Aware Farthest Point Sampling (DA-FPS) strategy, which analyzes the data manifold to select a high-fidelity small subset of the target domain by balancing sample diversity from sparse regions with representativeness from dense regions. Subsequently, in the parameter selection step, we use this subset to identify the most sensitive parameters in the cross-domain task . Inspired by network pruning ( Cun, Denker, and Solla 1989 ) , our framework quantifies key gradient behaviors such as magnitude, consistency, and variance to calculate a composite sensitivity score for each parameter. We then select the most sensitive subset of parameters and update them, thereby achieving efficient and precise model adaptation. In summary, the contributions of this study are as follows: • We are the first to define and address a novel task in the pansharpening field that enhances model generalization while jointly considering model architecture invariance and adaptation economy, offering a more practical research perspective for the community. • We propose SWIFT, a plug-and-play and general-purpose framework. It innovatively integrates: (a) a data density-aware sample selection strategy, and (b) an efficient, gradient-based parameter sensitivity identification method, providing the first effective solution to the newly defined task. • Extensive experiments demonstrate that SWIFT significantly improves cross-sensor performance of various mainstream models—achieving results comparable to or even surpassing full retraining on the target domain while requiring only minimal adaptation cost (approximately one minute, 3% training data, and 30% tunable parameters), and achieves state-of-the-art performance on multiple datasets, validating the effectiveness of our proposed framework. Related work Deep Learning-based Pansharpening Deep learning has emerged as the dominant paradigm in pansharpening, based on network architecture, these methods can be broadly classified into three categories. Early CNN-based models, from the pioneering PNN ( Masi et al. 2016 ) to advanced architectures like PanNet ( Yang et al. 2017 ) and FusionNet ( Deng et al. 2021 ) , excel at capturing local spatial-spectral patterns, but their limited receptive fields restrict the modeling of global dependencies, causing performance to degrade sharply under cross-sensor distribution shifts. To address this limitation, Transformer-based methods such as PanFormer ( Zhou, Liu, and Wang 2022 ) incorporating self-attention to model long-range dependencies. Although this improves the accuracy of the fusion, the high computational cost limits

Read the complete text