Parameter-Efficient Fine-Tuning of Multispectral Foundation Models for Hyperspectral Image Classification

Bernardin LIGAN, Khalide Jbilou, Fahd Kalloubi, Ahmed Ratnani · 2025

Read Parameter-Efficient Fine-Tuning of Multispectral Foundation Models for Hyperspectral Image Classification by Bernardin LIGAN, Khalide Jbilou, Fahd Kalloubi, Ahmed Ratnani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15334.txt.

Introduction

Bernardin Ligan and co-authors adapt a multispectral foundation model to hyperspectral images on a budget. Their twenty twenty-five paper notes that big remote sensing models learn from a few color bands, while hyperspectral shots carry hundreds. Full fine-tuning is heavy on memory and storage.

The target is SpectralGPT, retrained only in small adapter parts. Several cheap tuning styles are tested, from low-rank adapters to a variant with split learning rates. The goal is scene labeling accuracy close to full tuning at a sliver of the cost. Earth watchers get a route to reuse giant models on the richest satellite data without giant hardware.

Catalog record

CREATORBernardin LIGAN, Khalide Jbilou, Fahd Kalloubi, Ahmed Ratnani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15334

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Parameter-Efficient Fine-Tuning of Multispectral Foundation Models for Hyperspectral Image Classification Bernardin LIGAN, Khalide Jbilou, Fahd Kalloubi, Ahmed Ratnani Abstract Foundation models have experienced tremendous success across diverse fields, including remote sensing (RS), due to their versatility and strong generalization capabilities. However, most foundation models in RS are designed for multispectral imaging, with limited efforts devoted to hyperspectral imagery, which presents unique challenges due to its hundreds of spectral bands. Moreover, the efficient fine-tuning of foundation models for downstream tasks remains a challenging issue, as it often requires substantial memory and storage resources. In this paper, we present an effective framework to fine-tune SpectralGPT, a multispectral foundation model, for hyperspectral image classification (HSIC) tasks. To improve its efficiency, we explore various well-established Parameter-Efficient Fine-Tuning (PEFT) methods, including Low-Rank Adaptation (LoRA), Kronecker product-based adaptation (KronA), Low-Rank Kronecker product adaptation (LoKr), and the recent LoRA+, an extension of LoRA that employs distinct learning rates, linked by a scaling factor λ \lambda , for each low-rank adapter matrix during training. Inspired by LoRA+, we propose KronA+, an extension of KronA that applies a similar mechanism to the matrices used in the Kronecker product. We evaluate our methods on five datasets from different sensors, demonstrating competitive performance with state-of-the-art backbones in the hyperspectral domain. Our full fine-tuning (FFT) framework for SpectralGPT even surpasses a dedicated hyperspectral foundation model on certain datasets while requiring only about a quarter of the training epochs. Under identical training epochs and with significantly fewer trainable parameters compared to FFT of SpectralGPT, KronA+ achieves comparable performance, demonstrating its effectiveness as the best-performing PEFT method. It utilizes only 0.056% of trainable parameters and adds merely ∼ \sim 0.2 MB of storage overhead. This work is currently under review. keywords Remote sensing, foundation model , fine-tuning , Hyperspectral Image , Parameter Efficient Fine-Tuning , LoRA † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction Hyperspectral imaging (HSI) has emerged as a powerful technology for applications requiring precise spectral characterization of materials, enabling advancements in environmental monitoring, agriculture, defense, and mineral exploration. Unlike traditional RGB or multispectral data, which capture a limited number of spectral bands, HSI provides a continuous spectrum for each pixel across hundreds of narrow wavelengths. The unique ability of HSI to capture subtle spectral differences is critical for applications where traditional imaging methods lack the necessary depth. However, harnessing the full potential of HSI data presents significant challenges. Its high dimensionality and complex structure require advanced machine learning and deep learning models capable of extracting meaningful patterns. Despite progress in this area, performance gaps remain, particularly in classification tasks where distinguishing fine spectral variations is essential. Foundation models have revolutionized natural language processing (NLP) by leveraging large-scale datasets to generalize across diverse tasks. Models such as BERT ( Devlin et al., 2019 ) and GPT ( Radford et al., 2018 ) are pre-trained on vast corpora and fine-tuned to adapt to a variety of downstream applications. This paradigm shift has led to significant advancements in NLP and has been successfully extended to other domains, notably computer vision. In computer vision, foundation models like Segment Anything Model (SAM) ( Kirillov et al., 2023 ) , and CLIP ( Radford et al., 2021 ) are pre-trained on large-scale RGB image datasets and achieve impressive performance across a range of tasks, including image classification, object detection, and segmentation. Encouraged by the success of foundation models in NLP and computer vision, researchers have begun to develop such models for remote sensing tasks ( Lu et al., 2025 ) . Most of these models, however, are trained primarily on multispectral data, which has historically been more accessible and widely utilized. To our knowledge, only a few foundation models have been specifically designed for hyperspectral data. These hyperspectral foundation models, while promising, are typically more complex and computationally expensive, as they must process the high-dimensional spatial-spectral information inherent to hyperspectral imagery. Fine-tuning large-scale models for specific tasks presents significant challenges, particularly due to the high computational and storage demands. As foundation models continue to grow in scale, the development of efficient fine-tuning methods becomes increasingly critical, especially in resource-constrained environments. To address these challenges, Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA ( Hu et al., 2022 ) , have emerged. LoRA (Low-Rank Adaptation) introduces low-rank decomposition into the model’s weight updates during fine-tuning, enabling the adaptation of large models to downstream tasks with only a small subset of parameters. Alternative techniques to LoRA such as KronA ( Edalati et al., 2022 ) and Lokr ( YEH et al., 2024 ) , have been proposed to address the limitations of low-rank constraints in LoRA, enhancing expressiveness while maintaining efficiency. Recent work on LoRA+ ( Hayou et al., 2024 ) has demonstrated that using distinct learning rates for low-rank matrices can further improve fine-tuning performance. Inspired by the advancements in PEFT methods and the limited availability of hyperspectral-specific foundation models, two key questions arise: 1. Rather than developing a new hyperspectral-specific foundation model, why not fine-tune an existing multispectral foundation model for hyperspectral image classification? 2. How can parameter-efficient fine-tuning (PEFT) methods be leveraged to reduce the number of trainable parameters during this adaptation while maintaining performance comparable to full fine-tuning? Building on these questions, our contributions can be summarized as follows: • We introduce the first attempt to fine-tune a multispectral foundation model (SpectralGPT) for hyperspectral image classification, demonstrating that, with appropriate dataset transformations and fine-tuning strategies, it can achieve performance comparable to dedicated hyperspectral architectures. • We explore a range of PEFT techniques, including LoRA, LoRA+, KronA, Lokr, and introduce KronA+—a novel extension of KronA inspired by LoRA+ that applies differentiated learning rates to the matrices involved in the Kronecker product. This strategy significantly reduces the number of trainable parameters while maintaining performance competitive with full fine-tuning. To the best of our knowledge, this is the first systematic study on PEFT methods in hyperspectral image classification, offering new insights into the trade-offs between efficiency and performance in adapting foundation models to hyperspectral data. • We conduct a comprehensive evaluation of our framework on five diverse hyperspectral datasets, demonstrating its effectiveness across different sensors and imaging environments. 1 1 1 This work is currently under review. 2 Related Work 2.1 Deep Learning for Hyperspectral Image Classification Hyperspectral images (HSI) are represented as 3D tensors \textit{X}\in\mathbb{R}^{H\times W\times B} , where H and W denote spatial dimensions, and B is the number of spectral bands. The classification task involves assigning a label y\in Y to each pixel based on its spectral signature. Beyond spectral features, spatial information is crucial in HSIC. To integrate spectral-spatial context, hyperspectral images are generally divided into patches X_{p}\in\mathbb{R}^{P\times P\times B} , with classification performed on the center pixel. Deep learning models, particularly convolutional neural networks (CNNs), have significantly improved HSIC by jointly learning spectral and spatial representations ( Leng et al., 2016 ; Şakacı and Urhan, 2020 ; Quan et al., 2020 ; He et al., 2018 ) . More recently, transformer-based models ( Hong et al., 2022 ) have leveraged self-attention ( Vaswani et al., 2017 ) to model long-range

Read the complete text