Towards Scalable Foundation Model for Multi-modal and Hyperspectral Geospatial Data
Haozhe Si, Yuxuan Wan, Minh Do, Deepak Vasisht, Han Zhao, Hendrik F. Hamann · 2025
Read Towards Scalable Foundation Model for Multi-modal and Hyperspectral Geospatial Data by Haozhe Si, Yuxuan Wan, Minh Do, Deepak Vasisht, Han Zhao, Hendrik F. Hamann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12843.txt.
Introduction
Haozhe Si and five co-authors lay groundwork for a scalable base model for geospatial data with many modes and fine color bands. Satellite raster data hold rich place and time context across channels. Their work came out in twenty twenty-five.
Past self taught methods adapt poorly as channels and modes grow, which wastes computing. The new vision transformer splits its focus step into low rank space and spectra parts. A place and channel code layer keeps the physical sense of each patch. A mask limits focus to near patches, which uses local links. The team also builds a broad test bench to check these ideas.
Catalog record
| CREATOR | Haozhe Si, Yuxuan Wan, Minh Do, Deepak Vasisht, Han Zhao, Hendrik F. Hamann |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-12843
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Scalable Foundation Model for Multi-modal and Hyperspectral Geospatial Data Haozhe Si, Yuxuan Wan, Minh Do, Deepak Vasisht, Han Zhao, Hendrik F. Hamann Abstract Geospatial raster (imagery) data, such as that collected by satellite-based imaging systems at different times and spectral bands, hold immense potential for enabling a wide range of high-impact applications. This potential stems from the rich information that is spatially and temporally contextualized across multiple channels (e.g., spectral bands, polarizations) and sensing modalities. Recent work has adapted existing self-supervised learning approaches for such geospatial data. However, they fall short of scalable model architectures, leading to inflexibility and computational inefficiencies when faced with an increasing number of channels and modalities. To address these limitations, we introduce Low-rank Efficient Spatial-Spectral Vision Transformer (LESS ViT) with three key innovations: i) the LESS Attention Block that approximates high-dimensional spatial-spectral attention through Kronecker’s product of the low-dimensional spatial and spectral attention components; ii) the Continuous Positional-Channel Embedding Layer that preserves both the continuity and physical characteristics of each spatial-spectral patch; and iii) the Perception Field Mask that exploits local spatial dependencies by constraining attention to neighboring patches. To evaluate the proposed innovations, we construct GFM-Bench, which serves as a comprehensive benchmark for such geospatial raster data. We pretrain LESS ViT using a Hyperspectral Masked Autoencoder framework with integrated positional and channel masking strategies. Experimental results demonstrate that our proposed method achieves competitive performance against state-of-the-art multi-modal geospatial foundation models while outperforming them on cross-satellite generalization tasks with higher computational efficiency. The flexibility and extensibility of our framework make it a promising direction for future geospatial data analysis tasks that involve a wide range of modalities and channels. Code and project page available at https://uiuctml.github.io/GeospatialFM/ . 1 Introduction Figure 1 : Illustration of Multi-modal Geospatial Data. Two satellite systems capture complementary modalities: optical imagery (left) with multiple spectral bands, and Synthetic Aperture Radar (SAR) imagery (right) with different polarization. Both modalities capture information across spatial and spectral dimensions. While our analysis focuses on optical and SAR data in this work, Earth observation systems can incorporate additional modalities such as thermal infrared and atmospheric measurements. Geospatial data provides location-specific, timestamped information about the Earth’s surface. The rapid development and proliferation of satellite-based imaging systems have led to a significant increase in geospatial raster (e.g., imagery) data collection, offering valuable insights into various aspects of our planet. Geospatial raster data 1 1 1 Although we focus on static muti-modal and hyperspectral geospatial raster data in this work, we refer in the following as geospatial data for simplicity. is inherently multi-modal, integrating observations from diverse sensing systems such as optical and radar satellites, as illustrated in Figure 1 . Each modality captures distinct data dimensions through multiple channels (e.g., spectral bands, polarizations 2 2 2 We consider SAR polarization channels analogous to spectral bands in this work. ), while introducing complexities from multi-temporal observations, non-ideal imaging conditions, and varying spatial resolutions. These properties pose significant challenges for interpreting and analyzing geospatial data, requiring most often expert knowledge and specialized tools, which has hampered the utilization of such data. To address these challenges, the remote sensing community has adapted deep-learning methods to process and extract information from geospatial data for specific tasks, including monitoring land use [ 1 , 2 , 3 ] and climate change [ 4 , 5 , 6 ] , forecasting natural disasters [ 7 , 8 , 9 , 10 ] , and managing natural resource [ 11 , 12 , 13 ] . While these task-specific models have demonstrated effectiveness, their reliance on supervised learning approaches requires extensive high-quality labeled data annotated by experts, which is often not available or can be too time-consuming and costly to obtain. Furthermore, the specialized nature of these models limits their applicability to a wider range of problems. Motivated by the success of foundation models in natural language processing and computer vision [ 14 , 15 , 16 , 17 ] , one natural direction is to develop large-scale, self-supervised models for geospatial applications. Self-supervised learning (SSL) allows models to benefit from the vast amounts of unlabeled geospatial data and learning useful representations. These pretrained models can then be quickly adapted to specific downstream tasks using smaller labeled datasets, reducing the need for expensive manual annotations. Recent works on geospatial foundation models have attempted to adapt existing SSL paradigms [ 18 , 19 , 20 , 15 , 21 ] to geospatial datasets using various strategies. For example, CROMA [ 22 ] combines reconstruction and contrastive losses to align features from different modalities, SatMAE [ 23 ] designs specific patch embedding layers to support multi-temporal and hyperspectral data, and Channel-ViT [ 24 ] applies attention mechanisms to individual channels of hyperspectral images. Although these approaches achieved empirical success, their underlying architectures and objectives remain largely the same as those designed for natural images 3 3 3 By natural images, we refer to everyday photographs, e.g., those in conventional computer vision datasets. and thus do not fundamentally suitable to fully capture the spatial, spatial-channel and inter-channel relations of geospatial data. These relationships are critical for downstream geospatial tasks. For instance, spatial features alone may fail to differentiate aged asphalt roads from water bodies, despite their markedly different spectral characteristics. Therefore, developing model architectures that explicitly encode these distinctive relationships in geospatial data and while efficiently scaling to thousands of spectral channels would significantly advance existing approaches that predominantly leverage spatial features. To design a framework specific to geospatial data, we highlight three key characteristics that distinguish it from natural images. First, geospatial data, especially land cover observations, exhibits strong spatial autocorrelation that decays with physical distance, in contrast to natural images where correlations are primarily driven by semantic relationships. In this work, we prioritize this distance-dependent correlation by introducing a Perception Field Mask that constrains the spatial attention map, limiting each patch’s attention to its local neighborhood when generating spatial representations. Second, each pixel in geospatial data corresponds to a specific physical area on Earth with a defined spatial resolution. This intrinsic spatial property makes geospatial data particularly sensitive to resizing operations, as they alter the spatial resolution and complicate model generalization across varying input sizes. To overcome this resolution dependency, we implement a continuous positional embedding that computes inter-patch distances based on physical spatial resolution rather than grid indices. Moreover, the proposed Perception Field Mask facilitates generalization across input resolutions by maintaining a consistent number of attended patches in the attention computation. Finally, geospatial data encompasses often many channels (e.g., multiple spectral bands) but also sensing modalities, such as Multi-Spectral Imagery (MSI) and Synthetic Aperture Radar (SAR). These modalities can be spatially (and temporally) aligned, where for example each channel represents electromagnetic wave reflectance at specific wavelengths and polarizations. To preserve the physical meaning of channels, we implement a continuous channel embedding that encodes channel wavelengths into tokens. To handle varying input channels, we implement a Tied Patch Embedding layer. We further introduce the Low-rank Efficient Spatial-Spectral (LESS) attention block, which replaces standard ViT attention layers to efficiently compute both spatial and spectral attention for geospatial data. The proposed LESS ViT architecture significantly reduces the computational complexity of spatial-spectral attention from O(N^{2}C^{2}) to O(NC) ,