How Does the Spatial Distribution of Pre-training Data Affect Geospatial Foundation Models?
Mirali Purohit, Gedeon Muhawenayo, Esther Rolf, Hannah Kerner · 2025
Read How Does the Spatial Distribution of Pre-training Data Affect Geospatial Foundation Models? by Mirali Purohit, Gedeon Muhawenayo, Esther Rolf, Hannah Kerner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12535.txt.
Introduction
Mirali Purohit and co-authors ask a basic question about Earth observation AI: does it matter where on the planet the training images came from? Geospatial foundation models promise help with climate, farming, and disaster response. Their builders obsessed over model design and training tasks, but not over the map of the data itself.
The team sampled varied regional mixes from a global pool, pretrained two such models on each mix, and scored them on downstream tasks. Balanced world coverage behaved differently from skewed diets. For a technology sold as planetary, proof that the training map matters could redirect how the big shared datasets get built and shared.
Catalog record
| CREATOR | Mirali Purohit, Gedeon Muhawenayo, Esther Rolf, Hannah Kerner |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-12535
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Does the Spatial Distribution of Pre-training Data Affect Geospatial Foundation Models? Mirali Purohit, Gedeon Muhawenayo, Esther Rolf, Hannah Kerner Abstract Foundation models have made rapid advances in many domains including Earth observation, where Geospatial Foundation Models (GFMs) can help address global challenges such as climate change, agriculture, and disaster response. Previous work on GFMs focused on tailoring model architecture and pre-text tasks, and did not investigate the impact of pre-training data selection on model performance. However, recent works from other domains show that the pre-training data distribution is an important factor influencing the performance of the foundation models. With this motivation, our research explores how the geographic distribution of pre-training data affects the performance of GFMs. We evaluated several pre-training data distributions by sampling different compositions from a global data pool. Our experiments with two GFMs on downstream tasks indicate that balanced and globally representative data compositions often outperform region-specific sampling, highlighting the importance of diversity and global coverage in pre-training data. Our results suggest that the most appropriate data sampling technique may depend on the specific GFM architecture. These findings will support the development of robust GFMs by incorporating quality pre-training data distributions, ultimately improving machine learning solutions for Earth observation. 1 Introduction Figure 1: Our experimental pipeline to measure the impact of pre-training data distributions on downstream task performance. Foundation models have proven to be powerful tools across a wide variety of fields. These models have shown remarkable potential by achieving promising results while reducing the dependencies on labeled data. Researchers are increasingly adapting foundation models in specialized domains, such as health, law, physics, and geoscience by leveraging their ability to generalize from large datasets ( Colombo et al. 2024 ; Parmar et al. 2022 ; Cong et al. 2022 ) . In recent years, Geospatial Foundation Models (GFMs) pre-trained on vast quantities of satellite data have been developed for Earth observation (EO) applications. These models can play a critical role in improving the performance of tasks related to climate change, agriculture, oceanography, forests, disaster response, and more ( Jakubik et al. 2023 ; Tseng et al. 2023 ; Klemmer et al. 2023 ) . There is a general perception in AI that more data will help to improve the model’s performance ( Sun et al. 2017 ; Shi et al. 2024 ) . Hence, most current efforts to build foundation models usually pre-train on as much data as available or computationally feasible. However, recent studies show that dataset characteristics like diversity, spatial distribution, and representivity can be more important than data quantity to build robust ML models ( Nguyen et al. 2022 ; Cole et al. 2022 ) . This suggests that understanding the effects of pre-training data distribution on model performance is crucial. Although some studies have explored the relation between pre-training data distribution and downstream task performance in computer vision (CV) for natural images ( Al Kader Hammoud et al. 2024 ; Fang et al. 2022 ; Longpre et al. 2024 ) , there is a lack of studies of this type for GFMs. To build GFMs, researchers use large remote sensing datasets collected by satellites and aerial platforms. Given the massive data volume available from EO satellites (e.g., from the Sentinel-2 satellites), it is essential to select pre-training data in a way that benefits model performance but also reduces the computational cost – e.g., by focusing on data quality instead of quantity ( Rolf et al. 2021 ) . Unlike standard CV datasets, which are generally static, geospatial datasets are dynamic and available from multiple time periods. Satellite data offers the unique advantage of global coverage, allowing data to be sampled from any location on Earth. However, we don’t know the answer to some critical questions, such as “Is it crucial to have data from all continents? Is it crucial to have data evenly represented across biomes?” As a result, existing GFMs employ different strategies to sample pre-training data, showing a lack of consensus on a unified approach to sample pre-training data (details in § 2 ). Roscher et al. 2024 showed that most of the modeling focus in the geospatial domain is currently paid to the pre-training approach or model architecture. At the same time, the composition of data chosen for pre-training is relatively understudied , due to the assumption that training data size is the dominant factor determining performance. In particular, Manas et al. 2021 is the only study that compares model performance with multiple data sampling strategies. Motivated by these limitations, we investigated how different GFMs perform on downstream tasks when different sampling techniques are used to curate pre-training data (pipeline shown in Figure 1 ). We conducted experiments on two GFMs by pre-training models on five different data sampling strategies, referred as data compositions . These strategies include random, stratified, and clustered sampling techniques (see § 3.3 ). We refer to sampling techniques that ensure balanced or global data representation as balanced sampling techniques, and those grouping samples by spatial or environmental characteristics as clustered sampling. We pre-trained the models on five different data compositions and investigated how they affect the downstream task performance disaggregated by continent (Figure 1 ). We compared these results with a baseline of no pre-training. Our main contributions are: • We investigated the effect of different pre-training data compositions on downstream tasks for GFMs. To the best of the authors’ knowledge, this is the first study to explore the impact of pre-training data distribution on downstream task performance for GFMs. • Our experimental results show that all balanced sampling techniques yield approximately equal performance on downstream tasks, and usually outperform models with clustered pre-training data or no pre-training. Overall, our observations highlight the importance of balanced and representative data sampling strategies in pre-training GFMs. We believe our findings will contribute to the development of robust GFMs by incorporating quality pre-training data that benefit downstream task performance. 2 Related Work The relationship between pre-training data distribution and its impact on downstream tasks is well-documented in the CV literature (detailed discussion in Appendix A ). However, this relationship for GFMs has received limited attention. Previous GFMs used heuristic measures of diversity to decide where to sample pre-training data, with large differences from model to model. Presto developed a stratified random sampling strategy across hemispheres, ecological biomes, and land cover types ( Tseng et al. 2023 ) . MMEarth sampled pre-training data uniformly across biomes and from five years ( Nedungadi et al. 2024 ) . SatCLIP used a global uniform at random sampling ( Klemmer et al. 2023 ) . SeCo, SSL4EO, and CROMA used pre-training data within a range of 50km around 10,000 largest cities globally, assuming data will be more diverse around cities ( Manas et al. 2021 ; Wang et al. 2023 ; Fuller, Millard, and Green 2024 ) . Prithvi used pre-training data only from the contiguous United States ( Jakubik et al. 2023 ) . SatMAE, ScaleMAE, and SpectralGPT pre-trained on existing datasets fMoW ( Christie et al. 2018 ) , which is biased towards the Global North, and BigEarthNet ( Sumbul et al. 2019 ) , which includes only Europe ( Cong et al. 2022 ; Reed et al. 2023