GeoVision Labeler: Zero-Shot Geospatial Classification with Vision and Language Models
Gilles Quentin Hacheme, Girmaw Abebe Tadesse, Caleb Robinson, Akram Zaytar, Rahul Dodhia, Juan M. Lavista Ferres · 2025
Read GeoVision Labeler: Zero-Shot Geospatial Classification with Vision and Language Models by Gilles Quentin Hacheme, Girmaw Abebe Tadesse, Caleb Robinson, Akram Zaytar, Rahul Dodhia, Juan M. Lavista Ferres (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24340.txt.
Introduction
Gilles Quentin Hacheme and five co-authors present GeoVision Labeler in two thousand twenty-five. The tool classifies satellite images with no task training at all. Such labels feed disaster response and land use checks. Tagged data are scarce in many regions that need them.
A vision language model first writes rich prose about each image. A plain language model then maps that prose to user set classes. For hard multi class tasks, a recursive grouping step builds meta classes. Scores were strong on a binary building task. Multi class scores stayed competitive on two known sets. Code is shared in the open for real world use.
Catalog record
| CREATOR | Gilles Quentin Hacheme, Girmaw Abebe Tadesse, Caleb Robinson, Akram Zaytar, Rahul Dodhia, Juan M. Lavista Ferres |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24340
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GeoVision Labeler: Zero-Shot Geospatial Classification with Vision and Language Models Gilles Quentin Hacheme, Girmaw Abebe Tadesse, Caleb Robinson, Akram Zaytar, Rahul Dodhia, Juan M. Lavista Ferres Abstract Classifying geospatial imagery remains a major bottleneck for applications such as disaster response and land-use monitoring–particularly in regions where annotated data is scarce or unavailable. Existing tools (e.g., RS-CLIP) that claim zero-shot classification capabilities for satellite imagery nonetheless rely on task-specific pretraining and adaptation to reach competitive performance. We introduce GeoVision Labeler (GVL), a strictly zero-shot classification framework: a vision Large Language Model (vLLM) generates rich, human-readable image descriptions, which are then mapped to user-defined classes by a conventional Large Language Model (LLM). This modular, and interpretable pipeline enables flexible image classification for a large range of use cases. We evaluated GVL across three benchmarks–SpaceNet v7, UC Merced, and RESISC45. It achieves up to 93.2% zero-shot accuracy on the binary Buildings vs. No Buildings task on SpaceNet v7. For complex multi-class classification tasks (UC Merced, RESISC45), we implemented a recursive LLM-driven clustering to form meta-classes at successive depths, followed by hierarchical classification–first resolving coarse groups, then finer distinctions–to deliver competitive zero-shot performance. GVL is open-sourced at https://github.com/microsoft/geo-vision-labeler to catalyze adoption in real-world geospatial workflows. † † footnotetext: * Corresponding author: Gilles Quentin Hacheme ([email protected]) 1 Introduction Geospatial image classification plays a pivotal role in remote sensing, supporting applications such as land use monitoring, urban planning, and disaster response ( Yin et al. 2021 ; Al Shafian & Hu 2024 ; Navin & Agilandeeswari 2020 ) . Traditional supervised learning methods rely on large, labeled datasets, which are often resource-intensive to compile or entirely unavailable, especially in scenarios such as disaster response ( Mehmood et al. 2022 ; Li et al. 2022 ; Song et al. 2019 ; Li et al. 2024a ) . Zero-shot learning presents a compelling alternative, allowing models to classify images without prior exposure to labeled examples of target classes ( Sun et al. 2024a ; Sun et al. 2024b ; Mirza et al. 2023 ) . Moreover, zero-shot classification can greatly accelerate human annotation workflows by providing timely weak labels that guide and streamline manual verification ( Wang et al. 2025 ; Li et al. 2024b ) . While some models offer zero-shot capabilities, they often rely on pretraining with remote sensing data or synthetic pseudo-labels. Indeed, frameworks such as RS-CLIP ( Li et al. 2023 ) and SuperCLIP ( Rambabu et al. 2025 ) , though designed for zero-shot remote sensing scene classification, adopt pretraining strategies thereby deviating from a strict zero-shot learning paradigm. In contrast, GeoVision Labeler (GVL) follows a strict zero-shot approach, requiring no fine-tuning or domain-specific customization. GVL achieves this goal by combining vision Large Language Models (vLLMs) and Large Language models (LLMs) to perform zero-shot classification of geospatial images. GVL leverages the complementary strengths of vLLMs and LLMs to overcome the known limitation that vLLMs excel at generating rich, high-level descriptions but often struggle to assign inputs to a fixed taxonomy of labels ( Snæbjarnarson et al. 2025 ; Zhang et al. 2024 ; Menon & Vondrick 2022 ) . Given a geospatial image and user-specified categories, first, the vLLM generates a detailed textual description. The LLM then classifies the description into one of the predefined categories. This modular architecture allows the tool to adapt seamlessly to various classification tasks by simply updating the list of target classes, making it highly versatile for diverse remote sensing applications. Additionally, because the classification is based on a human-readable description, users can readily understand the classification decisions, making GVL an inherently interpretable tool ( Nguyen et al. 2024 ; Menon & Vondrick 2022 ; Bilal et al. 2025 ) . Classification tasks in geospatial context that involve a large set of semantically related labels can be challenging, as models often confuse subtly different categories ( Sun et al. 2020 ; Ye et al. 2021 ; He & Peng 2017 ; Ge et al. 2015 ) . To this end, We introduced a zero-shot LLM-based semantic clustering pipeline that optionally organizes labels into a handful of coarse meta-classes , then refines within each group. By recursively applying (1) meta-class name generation, (2) meta-class assignment to each class, and (3) hierarchical extension, our approach reduces ambiguity among similar classes and achieves robust classification Overall Accuracy (OA) on more dissimilar meta-classes without any task-specific training. This paper evaluates GVL on three widely recognized remote sensing datasets: Spacenet v7 ( Van Etten & Hogan 2021 ) , UC Merced ( Neumann et al. 2019 ) , and RESISC45 ( Yang & Newsam 2010 ; Neumann et al. 2019 ) . Through zero-shot experiments, we assessed GVL’s performance across different model configurations and compared with UC Merced and RESISC45 results reported in recent studies ( Corley et al. 2024 ; Li et al. 2023 ) . Our findings demonstrate GVL’s strong potential to achieve competitive classification OA, particularly in scenarios where label classes are well-separated and exhibit minimal overlap. For example, on the binary Buildings vs. No Buildings task from SpaceNet v7, our pipeline achieves up to 93.2% zero-shot OA. Moreover, by merging semantically similar classes via our clustering method, GVL attains up to 86.4% and 84.3% OA respectively on meta-classes from UC Merced and RESISC45. GVL offers a fast, interpretable, and adaptable zero-shot classification framework for real-world satellite image analysis. It can be used for rapid tagging and visual search of satellite imagery--crucial when labels are scarce or time-to-action is critical. The source code for GVL is openly accessible from Github 1 1 1 https://github.com/microsoft/geo-vision-labeler , facilitating further research and application. 2 Background The classification of geospatial images has advanced significantly with the advent of deep learning, yet the reliance on labeled data remains a bottleneck ( Tao et al. 2020 ; Li et al. 2020 ) . Recent research has explored the application of pretrained vision models for zero-shot applications in remote sensing ( Li et al. 2023 ; Rambabu et al. 2025 ) . For instance, RS-CLIP achieves impressive accuracies of 95.94% on UC Merced and 85.76% on RESISC45 ( Li et al. 2023 ) . However, its flexibility remains limited, partly due to the demand for an additional image–text pretraining stage, the generation of pseudo-labels from unlabeled imagery, and a multi-stage curriculum learning step to progressively adapt the model. SuperCLIP similarly departs from a pure zero-shot approach: it enhances feature learning with super-resolution modules and semantic attribute-guided transformers, relying on domain-specific adaptations and additional training ( Rambabu et al. 2025 ) . These approaches, while effective, rely on subsequent pretraining. Our work provides a strict zero-shot learning framework by integrating a vLLM and an LLM classifier into a single, end-to-end pipeline that generates detailed image descriptions and directly maps them to user-defined classes. Unlike approaches that depend on additional adaptation, GVL does not require a task-specific training or a subsequent tuning–making it a broader, more efficient solution for rapid deployment across diverse geospatial classification tasks. Existing vLLMs such as Kosmos-2 and Llama 3.2 (Vision-Instruct) are pre-trained on vast amount of images and text data, enabling them to understand and generate relationships between visual and textual information ( Peng et al. 2023a ; Grattafiori et al. 2024 ) .