Position Prediction Self-Supervised Learning for Multimodal Satellite Imagery Semantic Segmentation DOI: XXXXXXX.XXXXXXX Journal: JACM Volume: 37 4 111 8 CCS: Computing methodologies Image segmentation
John Waithaka, Moise Busogi · 2025
Read Position Prediction Self-Supervised Learning for Multimodal Satellite Imagery Semantic Segmentation DOI: XXXXXXX.XXXXXXX Journal: JACM Volume: 37 4 111 8 CCS: Computing methodologies Image segmentation by John Waithaka, Moise Busogi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06852.txt.
Introduction
Teaching computers to outline objects in satellite photos is starved by the cost of hand-drawn labels, so models first learn from unlabeled images. John Waithaka and Moise Busogi adapt a position-guessing method for that first stage in a 2025 paper. Instead of rebuilding hidden patches, their model predicts where patches belong, training a sense of place.
Their version stretches across data types, radar and optical together, grouping channels the way multispectral methods do, and masks attention so the model must reason across sensors. On a flood-mapping benchmark, the position-trained model beat reconstruction-trained rivals at finding water in the photos. The authors argue that for satellite work, knowing where things are matters more than repainting them, and pretraining should honor that from the start.
Catalog record
| CREATOR | John Waithaka, Moise Busogi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06852
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Position Prediction Self-Supervised Learning for Multimodal Satellite Imagery Semantic Segmentation DOI: XXXXXXX.XXXXXXX Journal: JACM Volume: 37 4 111 8 CCS: Computing methodologies Image segmentation John Waithaka, Moise Busogi Abstract . Semantic segmentation of satellite imagery is crucial for Earth observation applications, but remains constrained by limited labelled training data. While self-supervised pretraining methods like Masked Autoencoders (MAE) have shown promise, they focus on reconstruction rather than localisation—a fundamental aspect of segmentation tasks. We propose adapting LOCA (Location-aware), a position prediction self-supervised learning method, for multimodal satellite imagery semantic segmentation. Our approach addresses the unique challenges of satellite data by extending SatMAE’s channel grouping from multispectral to multimodal data, enabling effective handling of multiple modalities, and introducing same-group attention masking to encourage cross-modal interaction during pretraining. The method uses relative patch position prediction, encouraging spatial reasoning for localisation rather than reconstruction. We evaluate our approach on the Sen1Floods11 flood mapping dataset, where it significantly outperforms existing reconstruction-based self-supervised learning methods for satellite imagery. Our results demonstrate that position prediction tasks, when properly adapted for multimodal satellite imagery, learn representations more effective for satellite image semantic segmentation than reconstruction-based approaches. Source code is available at https://github.com/johnGachihi/scenic . 1. Introduction Satellite imagery is a fundamental data source for Earth observation research, with semantic segmentation being particularly important for analysing this imagery. Semantic segmentation enables, for example, the extraction of flood extent maps, crop cover maps, and forest cover maps for disaster management, food security analysis, and climate research. While deep learning models have proven effective for semantic segmentation of satellite imagery (for example in ( Zhang et al. 2022 ; Li et al. 2020 ) ), semantic segmentation remains constrained by limited labelled training data. The pixel-level annotation for semantic segmentation is extremely expensive and time-consuming to obtain ( Zhou et al. 2017 ; Caron et al. 2024 ) , and satellite imagery adds to this challenge due to lower spatial resolution, unfamiliar semantic classes, and the need for domain expertise. Pretraining is commonly used to improve model performance when labelled training data is limited. Self-supervised pretraining, which does not require labelled data, particularly fits the satellite imagery domain, where, although there is a scarcity of labelled datasets, there are massive unlabelled satellite imagery datasets. Contrastive learning is a prominent self-supervised pretraining method. It involves matching two different views of the same thing, generated through separate data augmentation draws or temporal displacement ( Ayush et al. 2021 ) . However, Caron et al. ( Caron et al. 2024 ) find that models pretrained with contrastive learning do not transfer well to semantic segmentation tasks. They hypothesise this occurs because contrastive learning encourages global image-level representation with no need for spatial reasoning, whereas semantic segmentation is a pixel-level task that, intuitively, benefits from spatial reasoning. Masked image modelling, particularly the Masked Autoencoder (MAE) pretraining scheme ( He et al. 2022 ) , have been widely explored in the satellite imagery domain ( Cong et al. 2022 ; Reed et al. 2023 ; Tang et al. 2023 ; Ankit et al. 2025 ; Noman et al. 2024 ) . MAE defines a masked patch reconstruction task for self-supervised pretraining. This task encourages spatial reasoning as visible patches in different spatial positions predict masked patches in other positions. MAE-based methods, in some prior works, have outperformed contrastive methods on satellite imagery semantic segmentation ( Reed et al. 2023 ; Ankit et al. 2025 ) . Location prediction is a less prominent self-supervised pretraining method. LOCA (Location-aware) ( Caron et al. 2024 ) , in particular, defines a relative location prediction task for self-supervised learning. More precisely, a query and reference view are sampled from an input image, and each patch in the query view predicts its position in the reference view. This task encourages spatial reasoning for localisation, unlike MAE which encourages spatial reasoning for reconstruction. Since, segmentation is, in part, fundamentally a localisation task, we hypothesise that relative location prediction learns patch representations that are more effective for semantic segmentation. Further, Caron et al. ( Caron et al. 2024 ) , show that LOCA outperforms other self-supervised methods on various semantic segmentation datasets in the natural image domain. However, relative location prediction remains unexplored in the satellite imagery domain. Satellite imagery has significant differences from natural imagery. Whereas natural images typically consist only of RGB bands, satellite images can consist of more bands from a wider range of the electromagnetic spectrum. Further, since satellite images are captured by different kinds of Earth observation sensors, there exist ‘multimodal’ images giving complementary views of the same geolocations. We adopt LOCA to effectively handle the multispectral nature of satellite imagery as well as to exploit its multimodality to improve transfer performance on satellite imagery semantic segmentation. In this work, we adapt LOCA for multimodal satellite imagery semantic segmentation by extending channel grouping to handle multiple modalities (multispectral imagery, SAR, and DEM) and introducing same-group attention masking to encourage cross-modal interaction during pretraining. Evaluation on the Sen1Floods11 ( Bonafilia et al. 2020 ) flood mapping dataset demonstrates that our position prediction approach outperforms existing reconstruction-based self-supervised learning methods for satellite imagery. 2. Related Work 2.1. Position Prediction for SSL and LOCA A relatively unpopular branch of self-supervised learning (SSL) is patch position prediction. Patch position prediction methods exploit the spatial context in images to define a pretext task. These tasks involve predicting the spatial position of patches in an image. Doersch et al. ( Doersch et al. 2015 ) sample two patches from the same image and predict the position of one patch relative to the other. Noroozi and Favaro ( Noroozi and Favaro 2016 ) divide an image into nonoverlapping patches and predict their true positions after they have been shuffled. Zhai et al. ( Zhai et al. 2022 ) , using vision transformers, predict the positions of patches given the patches without positional information (position encoding ( Vaswani et al. 2017 ) ). Our work is based on LOCA ( Caron et al. 2024 ) . LOCA defines a relative patch position prediction task. More precisely, a query view and reference view are sampled from an image, and each patch in the query view predicts its position relative to the reference view. To this end, the query view patches attend to the reference view through a single cross-attention block. To control the difficulty of the task, a fraction of the reference view is made visible to the query view. Caron et al. ( Caron et al. 2024 ) show that LOCA outperforms other SSL pretraining methods on a number of natural image semantic segmentation datasets, however, position prediction methods remain underexplored in the satellite imagery domain. 2.2. Patch Clustering for Dense SSL Ziegler and Asano ( Ziegler and Asano 2022 ) use clustering to generate pseudo-labels for supervising a patch-level classification task. The cluster assignment is done online using a teacher network (and cluster prediction by a student network). LOCA ( Caron et al. 2024 ) uses the same technique in addition to relative position prediction. 2.3. Multimodal Pretraining for Satellite Imagery Multimodal learning attempts to build AI models that can extract and relate information from multiple modalities ( Xu et al. 2023 ; Baltrusaitis et al.