MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data
Antoine Labatie, Michael Vaccaro, Nina Lardiere, Anatol Garioud, Nicolas Gonthier, [4pt] Institut national de l’information géographique et forestière (IGN), France, [4pt] Univ Gustave Eiffel, ENSG, IGN, LASTIG, France[4pt] {firstname.lastname}@ign.fr · 2025
Read MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data by Antoine Labatie, Michael Vaccaro, Nina Lardiere, Anatol Garioud, Nicolas Gonthier, [4pt] Institut national de l’information géographique et forestière (IGN), France, [4pt] Univ Gustave Eiffel, ENSG, IGN, LASTIG, France[4pt] {firstname.lastname}@ign.fr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10894.txt.
Introduction
Antoine Labatie and four colleagues present a 2025 paper on teaching AI to read satellite images without labeled examples. Earth observation data mix many wavelengths, revisit the same place over time, and come from different sensors, so standard self-teaching methods need rework. The team ran a broad benchmark of ways to fuse these streams and set learning targets.
From the results they built MAESTRO, a masked autoencoder tuned for satellite data that also uses the spectrum itself as a teaching signal. On four datasets, the model led on tasks that depend on change over time and stayed competitive elsewhere. Tests within and across datasets suggest the design transfers well, which matters for mapping land, crops, and disasters at scale.
Catalog record
| CREATOR | Antoine Labatie, Michael Vaccaro, Nina Lardiere, Anatol Garioud, Nicolas Gonthier, [4pt] Institut national de l’information géographique et forestière (IGN), France, [4pt] Univ Gustave Eiffel, ENSG, IGN, LASTIG, France[4pt] {firstname.lastname}@ign.fr |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-10894
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data Antoine Labatie, Michael Vaccaro, Nina Lardiere, Anatol Garioud, Nicolas Gonthier, [4pt] Institut national de l’information géographique et forestière (IGN), France, [4pt] Univ Gustave Eiffel, ENSG, IGN, LASTIG, France[4pt] {firstname.lastname}@ign.fr Abstract Self-supervised learning holds great promise for remote sensing, but standard self-supervised methods must be adapted to the unique characteristics of Earth observation data. We take a step in this direction by conducting a comprehensive benchmark of fusion strategies and reconstruction target normalization schemes for multimodal, multitemporal, and multispectral Earth observation data. Based on our findings, we propose MAESTRO, a novel adaptation of the Masked Autoencoder, featuring optimized fusion strategies and a tailored target normalization scheme that introduces a spectral prior as a self-supervisory signal. Evaluated on four Earth observation datasets, MAESTRO sets a new state-of-the-art on tasks that strongly rely on multitemporal dynamics, while remaining highly competitive on tasks dominated by a single monotemporal modality. Code to reproduce all our experiments is available at https://github.com/ignf/maestro . Figure 1 : Overview of MAESTRO . MAESTRO extends the Masked Autoencoder to orchestrate the complex interplay of multimodal, multitemporal, and multispectral Earth Observation data. It employs token-based early fusion across time steps and similar modalities, and token-based late fusion across dissimilar modalities. It uses joint-token fusion for multispectrality, but still relies on a novel normalization of reconstruction targets—namely, patch-group-wise within groups of highly correlated bands—to inject a useful spectral prior during pretraining. Best viewed in color. 1 Introduction Self-supervised learning (SSL) has been central to recent breakthroughs in natural language processing [ 63 , 16 ] and computer vision [ 34 , 96 ] . It is an effective pre-training strategy, improving versatility and data efficiency when fine-tuning models on diverse downstream tasks—especially in the case of foundation models pre-trained on large amounts of unlabeled data. The SSL paradigm is promising for Earth Observation (EO) applications, where unlabeled data is abundant, but labeled data remains scarce and costly. However, to fully leverage the potential of SSL, it may be necessary to adapt off-the-shelf approaches to the unique characteristics of EO data. A key characteristic of EO data lies in its heterogeneity across resolutions, scales, modalities, and spectral bands [ 65 ] . This heterogeneity is frequently reflected in multimodal, multitemporal, and multispectral EO datasets, where the integration of various sensors effectively addresses the spatial-temporal-spectral “resolution dilemma” [ 2 ] . Our goal in this work is to efficiently adapt the canonical SSL framework of masked autoencoding to the specific characteristics of EO data to learn useful and versatile representations. We adapt the Masked Autoencoder (MAE) [ 34 ] , which has shown strong performance and computational efficiency on natural images. However, extending the MAE to multimodal, multitemporal, and multispectral EO data introduces several challenges. First, integrating multiple modalities and temporal observations requires an effective fusion strategy. One approach, joint-token fusion , projects all data dimensions into a shared token space. An alternative, token-based fusion , first clusters data dimensions, embeds them separately, and fuses the resulting tokens. Token-based fusion appears more suited for handling multimodality and multitemporality for several reasons: (i) it better captures the heterogeneity across modalities and time steps, (ii) it allows the integration of modality- and temporality-specific encodings, and (iii) it enables the use of cross-modal and cross-temporal self-supervisory signals. Yet, it remains unclear whether fusion should be applied early or late in the model. Second, handling multispectral data adds further complexity. Here, the choice between token-based and joint-token fusion is less obvious. While joint-token fusion offers computational efficiency, it limits the ability to incorporate cross-spectral self-supervisory signals. A key question, then, is whether we can preserve the computational benefits of joint-token fusion while reintroducing an effective prior on the multispectral structure—potentially through a carefully designed normalization of reconstruction targets. We investigate these questions in Section 4.3 and, based on our findings, introduce our proposed approach, MAESTRO , in Section 4.4 . MAESTRO extends the MAE framework by applying token-based early fusion across time steps and similar modalities, and token-based late fusion across dissimilar modalities. For multispectrality, MAESTRO employs joint-token early fusion combined with a novel normalization scheme for reconstruction targets. Our main contributions are as follows: • We extensively benchmark various fusion strategies and target normalization schemes for multimodal, multitemporal, and multispectral SSL in EO. • We introduce a novel patch-group-wise normalization method that normalizes reconstruction targets patch-wise within groups of highly correlated spectral bands. This approach injects a useful spectral prior into the self-supervised signal at negligible computational cost. • Building on the above, we propose MAESTRO, a tailored adaptation of the MAE framework which effectively orchestrates the use of multimodal, multitemporal, and multispectral EO data—both in terms of performance and computational cost. • We validate MAESTRO on four diverse EO benchmarks, demonstrating that it achieves state-of-the-art results on tasks strongly tied to multitemporal dynamics, while remaining highly competitive on others. 2 Related Work SSL Approaches and Models for EO. SSL has emerged as a promising approach for EO, due to the abundance of unlabeled remote sensing data. Initially, the development of SSL approaches for EO followed those from the natural images domain. Early work focused primarily on contrastive SSL [ 54 , 6 , 39 , 88 , 61 , 52 , 91 , 94 , 90 , 73 , 14 , 42 ] , often using convolutional neural networks [ 54 , 6 , 39 , 61 , 52 , 94 , 90 , 73 ] . Positive pairs were typically derived from different modalities, dates, or augmented views at the same location. More recent works have increasingly shifted toward generative SSL using Transformers, well-suited to masked autoencoding. These include approaches inspired by MAE [ 83 , 13 , 49 , 11 , 40 , 64 , 72 , 37 , 77 , 9 , 85 , 86 , 48 , 100 , 35 , 92 , 97 , 84 , 58 , 71 , 93 , 80 ] or SimMIM [ 70 , 55 , 33 , 17 , 36 , 68 ] , or custom generative approaches [ 67 , 89 , 21 , 20 , 87 ] . Some recent works also explore hybrid contrastive/generative approaches [ 57 , 18 ] or latent-space generative SSL [ 45 , 78 , 82 , 4 , 5 , 12 ] . Multimodal and Multitemporal SSL for EO. Recent studies in EO have begun to address multimodal [ 88 , 39 , 37 , 61 , 24 , 9 , 33 , 97 , 90 , 36 , 58 , 62 , 41 , 93 , 14 , 82 , 8 ] or multitemporal SSL [ 13 , 40 , 59 , 21 , 71 ] , but rarely both simultaneously. When multimodality is included, it is typically handled through parameter sharing or late fusion [ 88 , 7 , 37 , 33 , 97 , 58 , 62 , 93 ] . To date, most published EO foundation models support multimodal or multitemporal inputs only via late fusion, which limits representation learning and downstream performance (see Section 4.3 ). Notable exceptions include Presto [ 77 ] , Galileo [ 78 ] , OmniSat [