Training and Inference within 1 Second – Tackle Cross-Sensor Degradation of Real-World Pansharpening with Efficient Residual Feature Tailoring
Tianyu Xin, Jin-Liang Xiao, Zeyu Xia, Shan Yin, Liang-Jian Deng · 2025
Read Training and Inference within 1 Second – Tackle Cross-Sensor Degradation of Real-World Pansharpening with Efficient Residual Feature Tailoring by Tianyu Xin, Jin-Liang Xiao, Zeyu Xia, Shan Yin, Liang-Jian Deng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07369.txt.
Introduction
Tianyu Xin, Jin-Liang Xiao, Zeyu Xia, Shan Yin, and Liang-Jian Deng tackle a practical problem in satellite imaging, in a 2025 paper. Merging a sharp black-and-white image with a softer color one gives a clear color picture, but a model trained on one satellite often fails on another's data. Older fixes are slow or need extra training data.
Their method adds a small tailoring part at a key point inside existing models, where rich features turn into the final image. Trained with rules drawn from physics, and working patch by patch, it adapts in under a second for training and use together. On real images from several sources, it matched the best quality while running far faster than rival approaches.
Catalog record
| CREATOR | Tianyu Xin, Jin-Liang Xiao, Zeyu Xia, Shan Yin, Liang-Jian Deng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-07369
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training and Inference within 1 Second – Tackle Cross-Sensor Degradation of Real-World Pansharpening with Efficient Residual Feature Tailoring Tianyu Xin, Jin-Liang Xiao, Zeyu Xia, Shan Yin, Liang-Jian Deng Abstract Deep learning methods for pansharpening have advanced rapidly, yet models pretrained on data from a specific sensor often generalize poorly to data from other sensors. Existing methods to tackle such cross-sensor degradation include retraining model or zero-shot methods, but they are highly time-consuming or even need extra training data. To address these challenges, our method first performs modular decomposition on deep learning-based pansharpening models, revealing a general yet critical interface where high-dimensional fused features begin mapping to the channel space of the final image. A Feature Tailor is then integrated at this interface to address cross-sensor degradation at the feature level, and is trained efficiently with physics-aware unsupervised losses. Moreover, our method operates in a patch-wise manner, training on partial patches and performing parallel inference on all patches to boost efficiency. Our method offers two key advantages: (1) Improved Generalization Ability : it significantly enhance performance in cross-sensor cases. (2) Low Generalization Cost : it achieves sub-second training and inference, requiring only partial test inputs and no external data, whereas prior methods often take minutes or even hours. Experiments on the real-world data from multiple datasets demonstrate that our method achieves state-of-the-art quality and efficiency in tackling cross-sensor degradation. For example, training and inference of 512 × 512 × 8 512\times 512\times 8 image within 0.2 seconds and 4000 × 4000 × 8 4000\times 4000\times 8 image within 3 seconds at the fastest setting on a commonly used RTX 3090 GPU, which is over 100 times faster than zero-shot methods. Introduction Remote sensing images are widely utilized in various fields. However, due to hardware limitations, satellite sensors cannot capture images that are both spectrally rich and of high spatial resolution. Typically, satellites capture low-resolution multispectral (LRMS) images and high-resolution panchromatic (PAN) images separately. And the target of pansharpening is to fuse an LRMS image and a PAN image to a high-resolution multispectral (HRMS) image that preserves both spectral and spatial information. Figure 1: Comparison of different cross-sensor pansharpening methods. Existing approaches (upper panel) struggle to balance efficiency, quality, and data requirement. In contrast, our method (lower panel) achieves state-of-the-art performance with sub-second efficiency, requires only test-time inputs, and fully leverages pretrained model capabilities. Pansharpening methods are broadly classified into four categories: component substitution, multi-resolution analysis, variational optimization, and deep learning (DL) approaches. The first three categories are traditional methods that process inputs via mathematical transformations without model training, but their inability to capture nonlinear PAN–HRMS relationships limits fusion quality ( Xiao et al. 2022 ; Xiao et al. 2023 ) . In contrast, deep learning methods leverage excellent feature extraction capability of neural networks, such as convolutional neural network (CNN)-based models ( He et al. 2019 ; Cao et al. 2021 ; Wang et al. 2024 ) , Transformer-based models ( Li et al. 2024 ; Zhang and Ma 2021 ) , and Diffusion-based methods ( Cao et al. 2024b ; Zhong et al. 2024 ) , enabling them to produce superior fusion results. While DL methods have achieved impressive results, the growing diversity of satellite sensors makes cross-sensor generalization increasingly important for existing pansharpening models. However, current DL approaches often suffer from cross-sensor degradation , where models pretrained on data from a specific sensor generalize poorly to data from others. As shown in Figure 1 (upper panel), typical strategies addressing the issue include model retraining or zero-shot methods, but they are highly time-consuming or even need extra training data. For example, zero-shot methods ( Rui et al. 2024 ; Cao et al. 2024a ) usually take minutes to process only one LRMS-PAN input ( 512\times 512 PAN), let alone model retraining that takes hours to accomplish. Furthermore, although traditional methods are more efficient, their fusion quality remains significantly lower than that of DL methods. More importantly, most existing methods often disrupt the pretrained model weights, failing to fully leverage the rich representational capacity already embedded in them. In sum, current methods for addressing cross-sensor degradation struggle to balance the trade-off between quality, efficiency, and data requirement, while also underutilizing the capabilities of existing pretrained models. Therefore, two critical challenges in tackling cross-sensor degradation for real-world pansharpening remain underexplored: 1) minimizing generalization cost by reducing processing time and avoiding the need for extra training data ; 2) preserving the pretrained model’s existing capabilities while ensuring improved generalization to unseen sensors . To address the challenges, we propose a plug-and-play solution to tackle cross-sensor degradation with sub-second efficiency: the Efficient Residual Feature Tailoring (ERFT) pipeline. It begins by modularly decomposing pansharpening models into two general components: Feature Extractor (FE) and Channel Mapper (CM). The FE generates high-dimensional fused feature from the LRMS-PAN input with various neural network architectures. And the CM maps the feature from feature space to channel space to generate the high resolution output image, mostly with a shallow CNN structure. Then we integrates a Feature Tailor (FT) between frozen FE and CM to address cross-sensor degradation of pretrained model at feature level while leveraging capabilities of pretrained models. To maximize efficiency and ensure structural compatibility, the FT also employs a CNN structure, which is trained with physics-aware unsupervised losses to preserve spectral and spatial fidelity. To further improve runtime efficiency, our method operates in a patch-wise manner during both training and inference. Specifically, the LRMS-PAN input is first partitioned into multiple patches. A randomly selected subset is used to train the FT, while keeping the pretrained FE and CM frozen. Then all modules (FE, FT, and CM) are fixed to perform parallel inference on all patches. Finally, the predicted HRMS patches are stitched together to form the final high-resolution output. Experiments on real-world data from sensors with different band setting again prove that our method tackles cross-sensor degradation with state-of-the-art (SOTA) quality and efficiency . Moreover, our method could process input images scaled up to megapixel level (images with over one million pixels). In detail, with a commonly used RTX 3090 GPU, our method can complete pansharpening of 512\times 512\times 8 image within 0.2 seconds and 4000\times 4000\times 8 within 3 seconds at the fastest setting, which is over 100 times faster than zero-shot methods and outperform latest pansharpening results of 2025. To the best of our knowledge, our method is the first to tackle cross-sensor degradation with the sub-second efficiency while achieving SOTA fusion quality. The main contributions are summarized as follows: • We propose the Efficient Residual Feature Tailoring (ERFT) to address the efficiency challenge of cross-sensor generalization. It avoids the need for extra training data and achieves training and inference within a sub-second runtime that has not been attained in this domain. • Our method offers a novel plug-and-play solution for cross-sensor degradation in real-world pansharpening. It avoids modifying the pretrained model parameters, thereby preserving their existing capabilities while enabling improved generalization to unseen sensors. • Extensive experiments demonstrate that our method achieves state-of-the-art results on both in-sensor and cross-sensor datasets with sub-second efficiency. Its ability to process megapixel-scale inputs within 3 seconds further showcases its scalability and practical potential. Related Works Deep Learning-based Pansharpening Deep