HyperspectralMAE: The Hyperspectral Imagery Classification Model using Fourier-Encoded Dual-Branch Masked Autoencoder
Wooyoung Jeong, Hyun Jae Park, Seonghun Jeong, Jong Wook Jang, Tae Hoon Lim, Dae Seoung Kim * · 2025
Read HyperspectralMAE: The Hyperspectral Imagery Classification Model using Fourier-Encoded Dual-Branch Masked Autoencoder by Wooyoung Jeong, Hyun Jae Park, Seonghun Jeong, Jong Wook Jang, Tae Hoon Lim, Dae Seoung Kim * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05710.txt.
Introduction
Jeong and five co-authors built a foundation model for hyperspectral satellite imagery in twenty twenty-five. Hyperspectral images record hundreds of wavelength bands per pixel, giving rich material fingerprints but posing a heavy data challenge in both space and spectrum. Their model, HyperspectralMAE, learns by reconstruction: hide parts of the image and predict them.
During pre-training, half the spatial patches and half the spectral bands are masked at random, forcing the network to infer missing content in both dimensions. Learnable Fourier-based embeddings encode each band's wavelength so spectral order is respected. The loss balances pixel accuracy with spectral shape fidelity. Pre-training used two large satellite corpora, and fine-tuning on the Indian Pines land-cover benchmark set a new state-of-the-art transfer accuracy. The authors say dual masking yields robust spectral-spatial representations.
Catalog record
| CREATOR | Wooyoung Jeong, Hyun Jae Park, Seonghun Jeong, Jong Wook Jang, Tae Hoon Lim, Dae Seoung Kim * |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-05710
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HyperspectralMAE: The Hyperspectral Imagery Classification Model using Fourier-Encoded Dual-Branch Masked Autoencoder * Corresponding author. Email: [email protected] Wooyoung Jeong, Hyun Jae Park, Seonghun Jeong, Jong Wook Jang, Tae Hoon Lim, Dae Seoung Kim * Abstract Hyperspectral imagery provides rich spectral detail but poses unique challenges due to its high dimensionality in both spatial and spectral domains. Therefore, we propose HyperspectralMAE , a transformer - based foundation model for hyperspectral data featuring a dual - masking strategy that randomly occludes 50% of the spatial patches and 50% of the spectral bands during pre - training. This forces the model to learn meaningful representations by reconstructing missing information across both dimensions. A positional embedding with spectral wavelength based on learnable harmonic Fourier components is introduced to encode the identity of each spectral band, ensuring that the model is sensitive to spectral order and spacing. The reconstruction objective employs a composite loss combining mean - squared error (MSE) and spectral angle mapper (SAM) to balance pixel - level accuracy and spectral - shape fidelity. The resulting model is of foundation scale ( ≈ \approx 0.18 B parameters, 768 - dimensional embeddings), indicating a high - capacity architecture suitable for transfer learning. We evaluated HyperspectralMAE on two large - scale hyperspectral corpora— NASA EO - 1 Hyperion (1,600 scenes, about 300 billion pixel spectra) and DLR EnMAP Level - 0 (1,300 scenes, about 300 billion pixel spectra)—and fine - tuned it for land - cover classification on the Indian Pines benchmark. Transfer - learning results on Indian Pines demonstrate state - of - the - art performance, confirming that our masked pre - training yields robust spectral–spatial representations. The proposed approach highlights how dual masking and spectral embeddings can advance hyperspectral image reconstruction. I INTRODUCTION Hyperspectral images (HSIs) capture dozens to hundreds of contiguous spectral bands per pixel, enabling fine - grained material discrimination in fields such as remote sensing, agriculture, and mineral exploration. However, this wealth of spectral information leads to high dimensionality and large data volume, making representation learning for HSIs challenging [ 1 , 2 ] . Self - supervised foundation models—especially masked autoencoders (MAE)—have recently shown promise in learning robust representations from unlabeled data in computer vision [ 3 ] . In the hyperspectral domain, such models offer the potential to exploit rich spectral–spatial redundancies, yet existing work remains limited to small - scale tasks. Therefore, we propose HyperspectralMAE , a dedicated masked - modeling framework that scales MAE principles to foundation - model size for hyperspectral imagery. I-A Motivation and Contributions Our main contributions are: • Dual - Masking Strategy: We simultaneously mask spatial patches and spectral bands, forcing the model to capture cross - dimensional dependencies [ 4 ] . • Fourier - Based Spectral Encoding: Spectral wavelength embeddings are introduced via harmonic Fourier components, encouraging the model to learn spectral locality and periodicity [ 5 ] . • Spectral–Spatial Reconstruction Loss: A composite loss combining mean - squared error (MSE) and spectral angle mapper (SAM) [ 6 ] ensures both numerical accuracy and high - fidelity spectral profiles [ 7 ] . • Foundation - Scale Model: Our transformer architecture scales to hundreds of millions of parameters, setting a new size benchmark for hyperspectral foundation models. I-B Organization The remainder of the paper is organized as follows. Section II reviews related work on masked autoencoders and hyperspectral representation learning. Section III details the proposed model and training methodology. Section IV describes the experimental setup and datasets. Section V presents empirical results, and Section VI concludes the paper. Fig. 1: HyperspectralMAE overall pipeline. II RELATED WORK II-A Masked Autoencoders (MAE) Masked image modeling has gained popularity after the success of Vision MAE by He et al. [ 3 ] , which showed that reconstructing masked patches is an effective pretext task for learning image representations. The original MAE applies random masking to input image patches and uses a ViT [ 8 ] encoder to encode visible patches and a lightweight decoder to reconstruct the missing patches. This approach has been extended to various domains. In hyperspectral imaging, a straightforward application is to mask spatial patches of the HSI cube and reconstruct them. Zhuang et al. [ 9 ] applied an MAE - like augmentation network for target detection in HSI, using a 1 - D positional encoding for spectra because 2 - D positional encoding was deemed unsuitable for spectral data. Similarly, Lin et al. [ 10 ] introduced SS - MAE, a spatial–spectral masked autoencoder with two branches: one masks and reconstructs random spatial patches, while the other masks and reconstructs random spectral bands. Their results confirm that incorporating spectral masking in addition to spatial masking leads to better feature learning. Our dual - masking strategy is inspired by this idea, but we integrate both aspects within a single unified architecture and additionally introduce cross - branch interactions. II-B Transformers for HSI Vision Transformers (ViT) [ 8 ] have been applied to hyperspectral images in recent studies. A naïve approach treats each hyperspectral patch (with all bands) as a token sequence and applies ViT for classification. While ViTs can model long - range spatial relationships, early attempts often ignored the explicit spectral structure. For example, some works reduced HSI to RGB (three bands) to apply pretrained ViTs, losing most spectral information. Others processed each pixel’s spectrum with a spectral transformer, treating each pixel’s spectral vector as a sequence of tokens. Scheibenreif et al. [ 11 ] proposed a Spectral Transformer that attends to bands for each pixel, significantly improving accuracy by leveraging full spectral information. They further introduced a factorized Spatial–Spectral Transformer (SST), which applies self - attention sequentially along spatial and spectral dimensions to handle the large token count in HSI. Factorizing attention in this way proved efficient and boosted performance. Our dual - branch transformer shares a similar motivation of decoupling spectral and spatial attention, but instead of sequential factorization, we employ parallel branches with cross - attention fusion to jointly learn spectral–spatial features. II-C HSI - Specific Networks Recently, various specialized networks have been designed explicitly for hyperspectral image (HSI) classification tasks. Traditionally, methods such as 3-D convolutional neural networks (3-D CNNs) and hybrid spectral-spatial CNNs have been widely adopted due to their ability to directly extract spatial-spectral features from HSI cubes via volumetric convolutions. Emerging advancements have focused on incorporating transformer architectures into HSI-specific contexts to leverage self-attention mechanisms for improved representation learning. For example, Zhu et al. [ 1 ] developed SpectralMAE, a spectral-focused masked autoencoder explicitly designed for reconstructing spectral bands from partially masked inputs. SpectralMAE employs a specialized positional encoding tailored to spectral data, achieving superior reconstruction quality for various band combinations. Similarly, Wang et al. [ 4 ] introduced HSI-MAE, which utilizes separate spatial and spectral encoders with fusion layers. HSI - MAE explicitly learns spatial context and spectral correlation via two encoder branches, somewhat similar to our dual - branch design. Our work, HyperspectralMAE goes further by introducing learnable Fourier - based positional features and a cross - attention - based embedding mechanism with MSE + SAM Loss function. In summary, whereas previous works have addressed spectral–spatial feature learning in parts (e.g., 1 - D positional embeddings or dual