SenCLIP: Enhancing zero-shot land-use mapping for Sentinel-2 with ground-level prompting
Pallavi Jain, Dino Ienco, Roberto Interdonato, Tristan Berchoux, Diego Marcos · 2024
Read SenCLIP: Enhancing zero-shot land-use mapping for Sentinel-2 with ground-level prompting by Pallavi Jain, Dino Ienco, Roberto Interdonato, Tristan Berchoux, Diego Marcos (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08536.txt.
Introduction
Pallavi Jain and co-authors improve satellite land mapping with a model trained on ground photos, in a 2024 paper. Vision-language models can label images from free-form text prompts, but they were trained mostly on ground-level photos, so they struggle with satellite views. Generic prompts such as a satellite image of a place add little.
Their model, SenCLIP, aligns satellite images from a European mission with geotagged ground photos from across Europe. That teaches it to link the satellite view with the ground-level view a prompt may describe. On two standard land-use data sets, it beat other remote sensing vision models in zero-shot labelling. Gains held for both aerial-style and ground-style prompts. The authors say free-form text can now drive land mapping more reliably, without new labels for each class.
Catalog record
| CREATOR | Pallavi Jain, Dino Ienco, Roberto Interdonato, Tristan Berchoux, Diego Marcos |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08536
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SenCLIP: Enhancing zero-shot land-use mapping for Sentinel-2 with ground-level prompting Pallavi Jain, Dino Ienco, Roberto Interdonato, Tristan Berchoux, Diego Marcos Abstract Pre-trained vision-language models (VLMs), such as CLIP, demonstrate impressive zero-shot classification capabilities with free-form prompts and even show some generalization in specialized domains. However, their performance on satellite imagery is limited due to the underrepresentation of such data in their training sets, which predominantly consist of ground-level images. Existing prompting techniques for satellite imagery are often restricted to generic phrases like “a satellite image of…” , limiting their effectiveness for zero-shot land-use/land-cover (LULC) mapping. To address these challenges, we introduce SenCLIP, which transfers CLIP’s representation to Sentinel-2 imagery by leveraging a large dataset of Sentinel-2 images paired with geotagged ground-level photos from across Europe. We evaluate SenCLIP alongside other state-of-the-art remote sensing VLMs on zero-shot LULC mapping tasks using the EuroSAT and BigEarthNet datasets with both aerial and ground-level prompting styles. Our approach, which aligns ground-level representations with satellite imagery, demonstrates significant improvements in classification accuracy across both prompt styles, opening new possibilities for applying free-form textual descriptions in zero-shot LULC mapping. Code, dataset and pretrained models are available at https://github.com/pallavijain-pj/SenCLIP 1 Introduction Monitoring land-use and land-cover (LULC) is essential for understanding human impacts on the environment and assessing related risks [ 41 ] . Spaceborne sensors have long been used to map LULC, providing key data for policy-relevant indicators, especially in rural areas [ 2 ] . These insights are crucial for sustainable development, land-use planning, habitat preservation, and effective natural resource management [ 9 , 7 ] . During the last decade, the evolution of deep learning has enhanced the reliability of LULC prediction based on remote sensing [ 50 ] . However, these approaches generally depend on large datasets for training the models via supervised learning, requiring a large initial effort and restricting them to a closed set of initial LULC classes. In general, the class labels influence the learning process by guiding the optimisation of the network’s parameters towards minimising the classification error for the known classes [ 11 , 14 ] . Consequently, the network may prioritise learning features that are highly discriminative for these specific set of classes. This limits the usefulness of the representation in recognising unseen classes without additional training or fine-tuning steps. Recent advancements in vision-language models (VLMs) have revolutionised this paradigm. VLMs leverage Web-scale image/caption datasets to learn a representation that allows for zero-shot predictions across various tasks. These models operate by learning a joint semantic space for image/text pairs through contrastive learning strategies. Prominent examples of VLMs include CLIP [ 29 ] , ALIGN [ 13 ] , and BLIP [ 17 , 18 ] , which aim to map both textual descriptions and visual representations into the same latent space, enabling direct semantic comparisons between the two modalities. Despite the significant strides made by VLMs, there are persistent challenges in the practical implementation of zero-shot approaches based on them. One of the obstacles lies in the manual prompting process intrinsic to these approaches, where the choice of vocabulary and specific textual cues for each class play a pivotal role. Even slight variations in prompt formulation, such as the inclusion or exclusion of articles like “a” can wield considerable influence over the model’s accuracy [ 47 ] . Moreover, achieving optimal performance often requires incorporating task-relevant information into the prompts. In specialised domains like remote sensing, specifying details such as “centered satellite photo” or describing specific attributes like “photo of { class } : a type of broadleaf forest” alongside the class name becomes imperative for accurate classification [ 29 , 47 , 1 ] . This nuanced prompt construction ensures that the model captures essential contextual cues, leading to improved classification accuracy [ 44 ] , even if the added context is meaningless [ 31 ] . The sensitivity of prompt engineering and prompt learning [ 46 ] to a specific context underscores the critical role of prompting for harnessing the full potential of VLMs. Another significant challenge arises when attempting to move into specialised domains that are underrepresented in the training sets of VLMs, as is the case in remote sensing tasks. This results in challenges both for the image representation, which suffers due to the differences to the more common image modalities in terms of resolution, perspective and radiometry, and the text representation. Indeed, the CLIP representation of satellite images tends to be aligned with coarse concepts (e.g. this is a satellite image ) that are of little help for understanding LULC [ 4 ] . One potential solution lies in the development of large-scale VLMs specifically tailored for remote sensing applications. However, a significant obstacle arises due to the scarcity of textual descriptions associated with remote sensing data. Unlike other domains with abundant image/text pairs, remote sensing lacks a substantial corpus of annotated satellite imagery with corresponding textual descriptions. Recent advancements like RemoteCLIP [ 22 ] , SkyCLIP [ 42 ] , and GeoRSCLIP [ 45 ] have successfully integrated remote sensing data with captions and class labels to enhance VLMs. These methods use curated satellite imagery, improving the models’ ability to understand spatial contexts and semantic relationships through supervised pre-training, leading to better performance. In contrast, label-free approaches such as Sat2Cap [ 4 ] focus on cross-view learning by integrating geotagged ground-level images with high-resolution satellite imagery. This enables the model to associate local scene details with satellite observations, allowing it to interpret ground-level prompts more effectively. By transferring detailed ground-level knowledge to broader satellite contexts, these approaches improve land-use and land-cover classification accuracy. In this work, we extend the Sat2Cap approach by leveraging the LUCAS dataset, which contains nearly one million geotagged images across the European Union, alongside Sentinel-2 medium-resolution imagery. Our key contributions are: 1) Alignment of Sentinel-2 representations with ground-level images. We advance zero-shot LULC classification by directly aligning Sentinel-2 imagery with co-located ground-level images from LUCAS, which allows the model to understand the satellite imagery via textual descriptions associated with ground-level concepts, as done in Sat2Cap [ 4 ] . This is done without relying on labels or captions, in contrast to competing approaches [ 22 , 42 , 45 ] . Unlike in Sat2Cap [ 4 ] , which focuses on higher-resolution satellite images, our approach bridges the visual representation gap between ground-level perspectives and Sentinel-2’s medium-resolution (10 m) multi-spectral data. This is particularly valuable given Sentinel-2’s widespread use in environmental and agricultural applications, where capturing fine-grained details remains a challenge. 2) New state-of-the-art in zero-shot LULC classification. By coupling our SenCLIP model with a rich, LLM-generated, set of class-specific prompts, we obtain substantial improvements in terms of zero-shot LULC classification performance on both EuroSAT and BigEarthNet, validating the usefulness of the approach for real-world zero-shot LULC mapping. Additionally, we introduce a simple prompt selection method , aiming to optimise performance by identifying the most representative prompts for each class to enhance classification performance. 2 Related work Zero-shot land use and land cover mapping: The large-scale study of LULC with spaceborne sensors [ 30 ] has taken large strides in terms of performance thanks to the application of deep learning-based methods [ 50 ] . Although determining some