Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation
Leonard Waldmann, Ando Shah, Yi Wang, Nils Lehmann, Adam J. Stewart, Zhitong Xiong, Xiao Xiang Zhu, Stefan Bauer, John Chuang · 2025
Read Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation by Leonard Waldmann, Ando Shah, Yi Wang, Nils Lehmann, Adam J. Stewart, Zhitong Xiong, Xiao Xiang Zhu, Stefan Bauer, John Chuang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10845.txt.
Introduction
Leonard Waldmann and co-authors build a foundation model for Earth watching in a 2025 paper. Satellites differ in color bands, sharpness, and sensing style, and most models accept only one fixed sensor. A newer class takes any sensor as input. Their model, Panopticon, extends a known self taught vision system in three ways.
Views of one place from different sensors count as natural variations of a single scene. Channel sampling varies the spectral input during training. A cross attention step reads whatever channels arrive, guided by wavelength data. The result is one model that can ingest mixed satellite feeds. Climate, farming, and disaster teams all gain when a single tool reads every eye in orbit.
Catalog record
| CREATOR | Leonard Waldmann, Ando Shah, Yi Wang, Nils Lehmann, Adam J. Stewart, Zhitong Xiong, Xiao Xiang Zhu, Stefan Bauer, John Chuang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10845
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation Leonard Waldmann, Ando Shah, Yi Wang, Nils Lehmann, Adam J. Stewart, Zhitong Xiong, Xiao Xiang Zhu, Stefan Bauer, John Chuang Abstract Earth observation (EO) data features diverse sensing platforms with varying spectral bands, spatial resolutions, and sensing modalities. While most prior work has constrained inputs to fixed sensors, a new class of any-sensor foundation models able to process arbitrary sensors has recently emerged. Contributing to this line of work, we propose Panopticon, an any-sensor foundation model built on the DINOv2 framework. We extend DINOv2 by (1) treating images of the same geolocation across sensors as natural augmentations, (2) subsampling channels to diversify spectral input, and (3) adding a cross attention over channels as a flexible patch embedding mechanism. By encoding the wavelength and modes of optical and synthetic aperture radar sensors, respectively, Panopticon can effectively process any combination of arbitrary channels. In extensive evaluations, we achieve state-of-the-art performance on GEO-Bench, especially on the widely-used Sentinel-1 and Sentinel-2 sensors, while out-competing other any-sensor models, as well as domain adapted fixed-sensor models on unique sensor configurations. Panopticon enables immediate generalization to both existing and future satellite missions, advancing sensor-agnostic EO. 1 University of California, Berkeley; 2 Technical University of Munich; 3 Munich Center for Machine Learning correspondence to [email protected] † † ∗ Denotes co-first authorship, ordered randomly. Co-first authors will prioritize their names on their resumes/websites Code: https://github.com/Panopticon-FM/panopticon 1 Introduction Figure 1 : Architecture of Panopticon. The highlighted blocks indicate modifications to the DINOv2 framework. When generating views, images of the same geographic footprint captured by different sensors at different times, scales, etc., are treated as augmentations. The spectral patch embedding block uses a shared patchifier to tokenize incoming views and a cross attention with a single query to fuse tokens within each spatial patch across channels. Earth observation (EO) satellites provide essential data for monitoring our planet, from detecting illegal mining [ 58 ] and dark vessels [ 47 ] , to field delineation [ 28 ] and yield prediction [ 52 ] . Recent years have witnessed substantial progress towards remote sensing foundation models (RSFMs) that have adapted computer vision methods designed for RGB imagery to EO data [ 51 , 45 , 72 , 12 ] . However, compared to natural images, remote sensing data is remarkably heterogeneous [ 64 , 61 , 53 ] . EO platforms generate tremendously varied data across multiple dimensions, including spatial resolution (centimeters to kilometers), spectral characteristics (multispectral, hyperspectral, radar), revisit times (continuous to static), swath widths, preprocessing levels (raw, top of atmosphere, surface reflectance), and sensing mechanisms (active and passive). While RSFMs began as sensor-specific models [ 36 , 12 , 26 , 56 , 67 ] , the current trend is towards more flexible models that can deal with a variety of EO data [ 75 , 1 , 70 , 17 , 34 , 59 ] . Most recently, an emerging class of any-sensor models has begun addressing the fundamental challenge of sensor agnosticism in EO. With only a few examples at the time of writing [ 74 , 50 ] , these models process data from arbitrary sensor configurations—including unseen combinations of spectral bands, resolutions, and modalities—without requiring sensor-specific adaptations during inference. They achieve this by decoupling spectral and spatial processing while incorporating knowledge of sensor-specific properties such as spectral response functions (SRFs) and ground sampling distances (GSDs). In this work, we propose Panopticon, a new any-sensor model (see Section A for the etymology of this name). Building upon the DINOv2 [ 46 ] framework, we make three key modifications for stronger integration of satellite data as a distinct data modality [ 53 ] . First, instead of distilling knowledge within a single image, we regard a given geolocation or “footprint” as the object of interest and consider snapshots from different sensors of that footprint as augmented views of the same object. This approach enables us to sample a broad spectrum of natural sensor-to-sensor variation by incorporating images from different channel characteristics, modalities, timestamps, and processing levels. Second, in addition to standard DINOv2 spatial augmentations, we apply a spectral augmentation by subsampling channels of multispectral (MS), hyperspectral (HS), and synthetic aperture radar (SAR) training data. Third, to handle views with varying numbers of channels, we implement a cross-attention over channels with added positional embeddings that encode sensor-specific spectral information into a unified representation. To evaluate spectral flexibility, we simulate new sensors from HS datasets using spectral convolution [ 6 ] , and subject models to reduced spectral and scale information to assess their invariance to these properties, in addition to assessing performance on sensors not encountered during pre-training. We demonstrate that Panopticon also achieves state-of-the-art performance on GEO-Bench [ 32 ] which comprises of popular sensors such as Sentinel-1 (S1) and Sentinel-2 (S2). To address sensor distribution shifts in fixed-sensor models, we employ patch embedding re-training as parameter-efficient domain adaptation [ 24 , 5 ] . A surprising finding is that DINOv2 serves as a remarkably strong baseline in these comparisons, hinting at the nature of future work. Through comprehensive experiments across 23 MS, HS, and SAR datasets, we provide valuable insights into the capabilities and limitations of current approaches to sensor-agnostic learning in EO. In summary, our contributions are as follows: • We introduce Panopticon, a RSFM capable of building representations for any MS, HS, or SAR sensor, regardless of GSD or spectral characteristics. • While maintaining competitive-to-state-of-the-art performance on established benchmarks, Panopticon outperforms other any-sensor models on more unique sensor configurations that cover important application domains. • We find that spectral progressive pre-training combined with a wider embedding dimension and the use of spectral embeddings in the channel attention module are crucial for optimal performance. Figure 2 : View Generation. Given a footprint, we randomly sample a snapshot performing an implicit spectral, scale, and temporal augmentation. To increase spectral variance, we apply a spectral crop, and following DINOv2, we apply a spatial crop and resize. We omit additional flips and color jittering for simplicity. Local and global views are created analogously with different augmentation parameters. 2 Related work Fixed-sensor RSFMs Approaches vary on how to process different sensors with the same model. USat [ 25 ] and msGFM [ 21 ] use sensor-specific patch embeddings feeding into a single encoder, with USat adding positional information to the tokens. SatMAE [ 12 ] , Galileo [ 60 ] , and AnySat [ 1 ] employ more granular approaches with patch embeddings for groups of channels. Video-inspired models like Prithvi [ 27 ] and S2MAE [ 35 ] patchify in spatio-temporal or spatio-spectral dimensions. Another line of work uses separate encoders and aligns the resulting sensor-specific representations through contrastive and reconstruction loss, as in CROMA [ 17 ] , modified contrastive loss, as in IAI-SimCLR [ 49 ] , or regularization of covariance matrices, as in DeCUR [ 69 ] . Any-sensor RSFMs To the best of our knowledge, two general-purpose any-sensor models have been proposed, both based on the MAE [ 22 ] architecture. DOFA [ 74 ] proposes dynamic weight generation with a hypernetwork [ 20 ] based on wavelengths of the input channels for the otherwise fixed-shaped input