Do Open-Vocabulary Detectors Transfer to Aerial Imagery? A Comparative Evaluation

Christos Tsourveloudis · 2026

Read Do Open-Vocabulary Detectors Transfer to Aerial Imagery? A Comparative Evaluation by Christos Tsourveloudis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22164.txt.

Introduction

Christos Tsourveloudis tests whether detectors that find anything in ordinary photos can cope with the view from above. Open-vocabulary detection uses vision-language models to recognize new categories from text alone, with no retraining, and it performs strongly on natural images. Aerial imagery is a different world of tiny objects and odd angles.

Tsourveloudis presents the first systematic benchmark of this transfer. Five state-of-the-art open-vocabulary models are evaluated on an aerial dataset of several thousand images across eighty categories. His 2026 preprint maps which models survive the move to drones and satellite views. That map matters for search and rescue, farming, and disaster mapping, where new object types appear without warning.

Catalog record

CREATORChristos Tsourveloudis
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22164

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Open-Vocabulary Detectors Transfer to Aerial Imagery? A Comparative Evaluation Christos Tsourveloudis Abstract Open-vocabulary object detection (OVD) enables zero-shot recognition of novel categories through vision-language models, achieving strong performance on natural images. However, transferability to aerial imagery remains unexplored. We present the first systematic benchmark evaluating five state-of-the-art OVD models on the LAE-80C aerial dataset (3,592 images, 80 categories) under strict zero-shot conditions. Our experimental protocol isolates semantic confusion from visual localization through Global, Oracle, and Single-Category inference modes. Results reveal severe domain transfer failure: the best model (OWLv2) achieves only 27.6% F1-score with 69% false positive rate. Critically, reducing vocabulary size from 80 to 3.2 classes yields 15× improvement, demonstrating that semantic confusion is the primary bottleneck. Prompt engineering strategies such as domain-specific prefixing and synonym expansion, fail to provide meaningful performance gains. Performance varies dramatically across datasets (F1: 0.53 on DIOR, 0.12 on FAIR1M), exposing brittleness to imaging conditions. These findings establish baseline expectations and highlight the need for domain-adaptive approaches in aerial OVD. I Introduction Unmanned Aerial Vehicles (UAVs) and satellite platforms are increasingly deployed for time-critical applications such as disaster response, infrastructure monitoring, and search-and-rescue operations. In these scenarios, robust object detection is essential—yet the objects of interest are often unpredictable. A disaster response team may need to locate ”overturned vehicles” or ”temporary shelters” that were never part of the detector’s training vocabulary. Traditional closed-set detectors, which recognize only predefined categories, fail catastrophically when confronted with such novel objects, limiting their utility in dynamic real-world deployments. [ 12 ] . At the same time, collecting and annotating large, diverse datasets to cover every possible category is expensive, time-consuming, and often impractical. These constraints have motivated the development of Open-Vocabulary Object Detection (OVD). Instead of learning only from labeled training categories, OVD methods incorporate semantic knowledge from vision-language models (VLMs) to relate visual features to natural language descriptions [ 26 ] . This allows them to detect objects defined solely by text prompts, enabling zero-shot recognition of categories not explicitly annotated during training [ 29 ] . By leveraging linguistic understanding, OVD systems offer a more scalable and flexible alternative that better reflects the complexity of the real world. However, a critical question remains unanswered: Do these OVD models, trained predominantly on ground-level imagery, transfer effectively to the aerial domain? Aerial imagery presents distinct challenges, including extreme scale variation, strict nadir viewpoints, and dense object clustering—conditions [ 3 ] . Moreover, semantic categories in aerial datasets often rely on fine-grained visual attributes or domain-specific terminology that may not align with the natural language priors encoded in VLMs. Despite the rapid growth of OVD methods, their zero-shot transferability to aerial platforms remains largely unexplored. While recent work has proposed aerial-specific OVD architectures trained on remote sensing data, no study has systematically evaluated whether general-purpose VLM-based detectors can provide a viable baseline for UAV applications, deployable without any aerial-specific fine-tuning. To better understand the feasibility of language-driven detection in these scenarios, this work investigates how existing vision–language models perform when applied directly to UAV imagery, without additional training. We also explore how different prompting strategies influence results, aiming to establish an initial reference point for applying open-vocabulary object detection techniques in aerial applications. The remainder of this paper is organized as follows: Section II reviews prior research Open-Vocabulary Object Detection. Section III details our methodology, including the taxonomic challenges of the LAE-80C benchmark, the architectures of the evaluated models, and our zero-shot experimental design. Section IV presents a quantitative and qualitative analysis of model performance, focusing on the precision-recall trade-offs and the impact of prompting strategies. Section V discusses the implications of our findings for real-world UAV deployment, and Section VI concludes the paper with directions for future work. II Related Work The emergence of large-scale vision-language models (VLMs) such as CLIP [ 20 ] and ALIGN [ 8 ] has fundamentally transformed object detection. By pretraining on hundreds of millions of image-text pairs, these models learn a shared embedding space where visual regions and natural language descriptions are semantically aligned. This enables a new detection paradigm: rather than training classifiers for fixed categories, detectors can match image regions to arbitrary text prompts at inference time. OVD methods generally fall into two categories: (i) two-stage approaches, such as ViLD, which use knowledge distillation from powerful VLMs but incur substantial computational cost [ 5 ] ; and (ii) end-to-end frameworks, such as GLIP [ 14 ] , Grounding DINO [ 16 ] , YOLO-World [ 2 ] and YOLOE [ 23 ] which integrate language grounding directly into the detection pipeline for streamlined inference. Recent studies also underscore the role of linguistic clarity; for instance, Hosoya et al. [ 7 ] show that OVD can underperform on ambiguous categories lacking precise descriptions—a key issue in remote sensing, where many targets defy specific naming. In the aerial domain, Teacher–student frameworks like CastDet employ VLMs pretrained on remote-sensing image–text pairs to create pseudo-labels for new aerial categories, facilitating detection without closed-set annotations [ 15 ] . Complementary techniques focus on strengthening semantic alignment through description regularization and similarity-aware losses, as in DescReg [ 28 ] . Specialized architectures such as OVA-DETR [ 25 ] and UAV-OVD [ 22 ] incorporate region–text contrastive learning and multi-level fusion to enhance small object detection while preserving real-time efficiency. Other efforts address cross-view mismatches by aligning ground- and aerial-view embeddings [ 10 ] . With regard to large foundation multimodal models, they have been tested for UAV tasks [ 6 ] but often lag in localizing small objects compared to tailored OVD detectors for aerial data. Progress in open-vocabulary aerial perception also relies on robust data resources. Traditional benchmarks such as DOTA [ 27 ] , DIOR [ 13 ] , VisDrone [ 30 ] , FAIR1M [ 21 ] and xView [ 11 ] were originally built for closed-set detection but have since been augmented with language annotations. More recent pipelines, such as LAE-1M [ 19 ] and OS-W2S [ 24 ] , automate text-aligned label generation using VLMs, enabling scalable supervision in remote sensing. The OVD paradigm can also be extended beyond bounding box detection to tasks such as segmentation and tracking. AeriaICLIP supports lightweight open-vocabulary segmentation tailored for UAV hardware [ 9 ] , while Follow Anything (FAn) combines open-set detection with tracking for dynamic navigation [ 17 ] . In summary, the field has advanced through language-aligned learning, adaptive architectures and automated data pipelines. Nonetheless, persistent hurdles remain, including unreliable zero-shot performance due to vague aerial category descriptions, the difficulty of managing dense small objects in high-altitude views, and strict UAV constraints on computation and power. III Methodology III-A Dataset: LAE-80C We evaluate zero-shot open-vocabulary detection on the LAE-80C benchmark, the evaluation subset of the Locate Anything on Earth (LAE-1M) dataset. LAE-80C aggregates validation splits from four established aerial object detection datasets: DOTA-v2.0, DIOR,FAIR1M and xView, resulting in 3,592 images with 86,558 annotated object instances across 80 categories [ 19 ] . We specifically selected LAE-80C for this study because it represents a rigorous stress test for zero-shot generalization. By aggregating data from diverse sources, it introduces high variance in image quality, sensor look-angle, and ground sampling distance—spanning from low-altitude drone imagery

Read the complete text