DOFA-CLIP: Multimodal Vision–Language Foundation Models for Earth Observation

Zhitong Xiong, Yi Wang, Weikang Yu, Adam J Stewart, Jie Zhao, Nils Lehmann, Thomas Dujardin, Zhenghang Yuan, Pedram Ghamisi, Xiao Xiang Zhu · 2025

Read DOFA-CLIP: Multimodal Vision–Language Foundation Models for Earth Observation by Zhitong Xiong, Yi Wang, Weikang Yu, Adam J Stewart, Jie Zhao, Nils Lehmann, Thomas Dujardin, Zhenghang Yuan, Pedram Ghamisi, Xiao Xiang Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06312.txt.

Introduction

Satellite and airborne sensors watch Earth in many modes, from plain photos to radar, and each mode tells a distinct story. Zhitong Xiong and nine co-authors present DOFA-CLIP in 2025, a single model that reads all those modes and ties them to plain language. Past models of this type stuck to one mode at a time.

The team first built a data set of two million image and text pairs that spans six sensor modes. They then trained the model with help from other vision models, which sharpens its sense of space and layout. One shared backbone bends to each mode and to varied band setups. The result aims at a tool that can move across Earth-watch tasks with less rebuild for each new sensor.

Catalog record

CREATORZhitong Xiong, Yi Wang, Weikang Yu, Adam J Stewart, Jie Zhao, Nils Lehmann, Thomas Dujardin, Zhenghang Yuan, Pedram Ghamisi, Xiao Xiang Zhu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06312

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DOFA-CLIP: Multimodal Vision–Language Foundation Models for Earth Observation Zhitong Xiong, Yi Wang, Weikang Yu, Adam J Stewart, Jie Zhao, Nils Lehmann, Thomas Dujardin, Zhenghang Yuan, Pedram Ghamisi, Xiao Xiang Zhu Abstract Earth observation (EO) spans a broad spectrum of modalities, including optical, radar, multispectral, and hyperspectral data, each capturing distinct environmental signals. However, current vision-language models in EO, particularly CLIP-based variants, remain confined to individual modalities, limiting generalization and scalability across diverse tasks. We present DOFA-CLIP (Dynamic-One-For-All CLIP), a unified vision-language foundation model that dynamically adapts to EO modalities with flexible spectral configurations through a single Transformer backbone. Our approach introduces three key contributions: 1) the construction of GeoLangBind-2M, a large-scale EO image–text dataset covering six heterogeneous modalities with rich natural language descriptions; 2) a novel training strategy called VECT (Vision-models Enhanced Contrastive Text-image pretraining), which enhances the spatial awareness of CLIP features with multiple vision foundation models; and 3) a Modality-aware Knowledge Agglomeration (MaKA) module that refines feature distillation with modality-specific awareness. DOFA-CLIP achieves state-of-the-art zero-shot performance across a wide range of EO benchmarks, including unseen modalities and a diverse number of input spectral bands. Together, these contributions establish a scalable foundation for multimodal EO understanding and open new avenues for integrating heterogeneous EO data with large language models. Code and datasets will be released. Introduction Figure 1: DOFA-CLIP employs a single unified vision Transformer to align heterogeneous EO data modalities using language as the grounding signal for cross-modal understanding and zero-shot generalization. Earth observation (EO) technologies have undergone remarkable advancement in recent years, generating vast amounts of multimodal geospatial data spanning different sensors and scales ( Xia et al. 2018 ; Zhu et al. 2017 ; Xiong et al. 2024c ) . In response to the exponential growth of remote sensing data, researchers have proposed foundation models ( Cong et al. 2022 ; Reed et al. 2023 ; Wang et al. 2024a ; Wang et al. 2023a ; Stewart et al. 2023 ; Hong et al. 2024 ) pretrained on large-scale EO data for multiple downstream tasks. However, these models typically focus on individual EO data modality, limiting their ability to leverage complementary characteristics across different sensor types ( Zhu et al. 2024 ) . To address this limitation, recent works such as OFA-Net ( Xiong et al. 2024b ) , DOFA ( Xiong et al. 2024a ) , AnySat ( Astruc et al. 2024a ) , OmniSat ( Astruc et al. 2024b ) , and FoMo-Net ( Bountos, Ouaknine, and Rolnick 2023 ) have introduced foundation models capable of learning from multiple EO modalities within a unified framework. These approaches have demonstrated promising results in multimodal EO representation learning. However, their focus remains on unsupervised visual pretraining, lacking a structured way to align EO data with language for better interpretability and retrieval. Meanwhile, multimodal large language models like GPT ( OpenAI 2023 ) , LLaVA ( Liu et al. 2023 ) , and Pixtral ( Agrawal et al. 2024 ) have shown remarkable capabilities in handling diverse and complex tasks. Vision-language foundation models, such as CLIP ( Radford et al. 2021 ) and SigLIP ( Zhai et al. 2023 ) , have played a crucial role as vision encoders of such multimodal large language models. Inspired by these advances, EO-specific adaptations like RemoteCLIP ( Liu et al. 2024 ) , RS5M ( Zhang et al. 2024b ) , RS-CLIP ( Li et al. 2023 ) , and SkyScript ( Wang et al. 2024c ) have attempted to bring vision-language contrastive learning to remote sensing. However, existing CLIP models for EO have limitations: 1) These models are usually limited to high-resolution RGB imagery and cannot handle modalities like multi/hyper-spectral or SAR, which are vital for many applications. 2) Compared with vision models like DINOv2 ( Oquab et al. 2023 ) , CLIP models are unsuitable for dense understanding tasks due to the lack of spatial awareness. Training a single powerful vision–language model capable of handling heterogeneous EO data modalities raises several core research questions ( RQ ). These include: \small1⃝ RQ1: How can we build a large-scale, semantically rich image–text dataset for diverse heterogeneous EO modalities? \small2⃝ RQ2: How can we efficiently train large CLIP models for EO with better spatial awareness? \small3⃝ RQ3: How can a single CLIP model handle heterogeneous EO data while preserving modality-specific characteristics? To address such challenges, we propose Dynamic-One-For-All CLIP (DOFA-CLIP), a unified multimodal vision–language foundation model that aligns diverse EO modalities to the language space using a single Transformer, as shown in Fig. 1 . Our contributions are as follows: 1. We introduce GeoLangBind-2M, a large-scale EO dataset comprising two million image–text pairs across six heterogeneous modalities, providing high-quality text descriptions for multimodal vision–language training. 2. We propose Vision-models Enhanced Contrastive Text-image pretraining (VECT), a simple yet effective training strategy that enhances the fine-grained spatial awareness of CLIP models with open vision foundation models. 3. We develop the Modality-aware Knowledge Agglomeration (MaKA) module to support modality-adaptive representation distillation within a unified Transformer. 4. We conduct extensive evaluations across zero-shot classification and cross-modal retrieval tasks, demonstrating state-of-the-art performance and better spatial awareness on both seen and unseen EO modalities. Figure 2: Visualization of data samples from GeoLangBind-2M. The dataset includes imagery from six different sensors and modalities: Sentinel-2 multispectral, Sentinel-1 SAR, EnMAP hyperspectral, elevation maps, infrared imagery, and aerial imagery. Each sample is paired with textual descriptions capturing key land cover types, objects, and geographic features. Related work Vision foundation models in Earth observation Existing pretrained models predominantly focus on RGB data, such as GFM ( Mendieta et al. 2023 ) , Scale-MAE ( Reed et al. 2023 ) , and Cross-Scale MAE ( Tang et al. 2024 ) . Others specialize in multispectral imagery, with FG-MAE ( Wang et al. 2023b ) and SatMAE ( Cong et al. 2022 ) tailored for Sentinel-2 data, while SSL4EO-L ( Stewart et al. 2023 ) is trained on Landsat imagery. SpectralGPT ( Hong et al. 2024 ) leverages a 3D generative transformer for spectral data analysis. CROMA ( Fuller, Millard, and Green 2023 ) and DeCUR ( Wang et al. 2024b ) use self-supervised multimodal learning to learn features across modalities. Satlas ( Bastani et al. 2023 ) assembles a large-scale multi-sensor dataset, pretraining separate models for each sensor. DOFA ( Xiong et al. 2024a ) introduces a neural plasticity-inspired hypernetwork that adapts to different sensor wavelengths, enabling joint training on five distinct sensors. OmniSat ( Astruc et al. 2024b ) proposes to fuse features across multiple EO modalities to learn robust multimodal representations without labeled data. SkySense ( Guo et al. 2024 ) utilizes separate visual encoders for different data modalities and designs a dedicated module for multimodal fusion. Vision-language foundation models Zero-shot foundation models like CLIP ( Radford et al. 2021 ) have significantly advanced computer vision. Building upon CLIP, SigLIP ( Zhai et al. 2023 ) introduces a pairwise sigmoid loss function to enlarge the training batch and enhance the model performance. Zero-shot foundation models have also been investigated in EO applications. SkyScript ( Wang et al. 2024c ) constructs a large-scale image–text dataset by linking remote sensing images with OpenStreetMap ( OpenStreetMap contributors 2017 ) semantics. RemoteCLIP ( Liu et

Read the complete text