Scale-aware Recognition in Satellite Images under Resource Constraints

Shreelekha Revankar, Cheng Perng Phoo, Utkarsh Mall, Bharath Hariharan, Kavita Bala · 2024

Read Scale-aware Recognition in Satellite Images under Resource Constraints by Shreelekha Revankar, Cheng Perng Phoo, Utkarsh Mall, Bharath Hariharan, Kavita Bala (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00210.txt.

Description

Abstract Recognition of features in satellite imagery (forests, swimming pools, etc.) depends strongly on the spatial scale of the concept and therefore the resolution of the images. This poses two challenges: Which resolution is best suited for recognizing a given concept, and where and when should the costlier higher-resolution (HR) imagery be acquired?

Catalog record

CREATORShreelekha Revankar, Cheng Perng Phoo, Utkarsh Mall, Bharath Hariharan, Kavita Bala
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00210

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scale-aware Recognition in Satellite Images under Resource Constraints Shreelekha Revankar, Cheng Perng Phoo, Utkarsh Mall, Bharath Hariharan, Kavita Bala Abstract Recognition of features in satellite imagery (forests, swimming pools, etc.) depends strongly on the spatial scale of the concept and therefore the resolution of the images. This poses two challenges: Which resolution is best suited for recognizing a given concept, and where and when should the costlier higher-resolution (HR) imagery be acquired? We present a novel scheme to address these challenges by introducing three components: (1) A technique to distill knowledge from models trained on HR imagery to recognition models that operate on imagery of lower resolution (LR), (2) a sampling strategy for HR imagery based on model disagreement, and (3) an LLM-based approach for inferring concept “scale”. With these components we present a system to efficiently perform scale-aware recognition in satellite imagery, improving accuracy over single-scale inference while following budget constraints. Our novel approach offers up to a 26.3% improvement over entirely HR baselines, using 76.3 % fewer HR images. Figure 1: With these images we can see how concept scale is linked to spatial resolution. If we are seeking out a spatially large concept like forest , lower resolutions are favored (b) , as higher resolutions may lack the needed context to discern between a forest (a) and a park (c) . At the same time while seeking out finer concepts such as sports track , certain details can only be discerned well at higher resolutions (d) and are obscured at lower resolutions (e) . 1 Introduction Figure 2: System overview. First, we determine which resolution is best suited for the search concept based on its scale (sec. 3.3 ). Then, we analyze the search area to find which regions would benefit the most from higher resolution inference (sec. 3.5 ). We sample the best suited regions while staying within a user specified budget. Based on this guidance we perform inference using one of three models, a high resolution satellite model, a low resolution satellite model, and a low resolution satellite model with knowledge distilled from its high resolution counterpart (sec. 3.4 ). This knowledge distilled model allows us to infer finer details using low resolution satellite imagery alone. The ever-increasing number of earth observation satellites (now more than 1500) offer us a unique opportunity to understand changes at the planetary scale, be it tracking the ecological degradation of forests and coral reefs ( Wicaksono et al. 2021 ; Gao et al. 2020 ) , the destruction of cultural heritage ( Tapete et al. 2021 ) , or economic development and well-being ( Engstrom et al. 2022 ) . A key to all these downstream applications is the ability to recognize accurately a broad vocabulary of concepts: a major computer vision challenge. An important aspect of recognition in satellite imagery is the notion of scale : Satellite image resolution is dependent on the particular satellite/sensor and is characterized by the GSD, or ground sampling distance: the distance corresponding to a single pixel. For example, Sentinel-2 images are fairly low resolution (1 pixel = 100 m 2 ) ( ESA 2024 ) , while images captured by the NAIP program are fairly high resolution (1 pixel = 1 m 2 ) ( U.S. Geological Survey 2024 ) . The image scale matters because many concepts of interest also have a characteristic physical size/scale. An olympic-sized swimming pool, typically about 50 m long, would be barely a speck in Sentinel-2 images. In contrast, a lake may be hundreds of square kilometers, necessitating an exceptionally large image to capture its extent in terms of NAIP imagery, which current models would not be equipped to do. The right resolution may also depend on the underlying geography. Dense urban areas with many geographical features may require high resolution data to analyze. In contrast, large swathes of uninhabited deserts can probably be analyzed accurately even with low-resolution imagery. Accuracy notwithstanding, we must also consider cost and availability. Low-resolution (LR) imagery is free, abundant, and covers the planet densely over space and time ( ESA 2024 ; USGS Earth Resources Observation and Science 2024 ) . In contrast, high resolution (HR) imagery typically comes from drones or low flying satellites that are commissioned as needed, and therefore come with a high cost and inconsistent temporal and spatial coverage ( Planet Labs PBC 2018– ; U.S. Geological Survey 2024 ) . Scientists in many application domains, be it ecology, archaeology, and urban planning ( Wicaksono et al. 2021 ; Gao et al. 2020 ; Tapete et al. 2021 ; Engstrom et al. 2022 ) , today manually reason about these different aspects, namely, the scale of the concept, the nature of the underlying geography, and the cost of acquired imagery, to achieve the best recognition that they can afford with their budget. We need an approach that can automate this tradeoff and automatically identify when to acquire more expensive high resolution imagery, taking into account the cost, the scale of the concept and the geography. While there is past work on the impact of scale on satellite image recognition, much of it is focused on accuracy and disregards costs ( Reed et al. 2023 ; Mall et al. 2024 ; Bastani et al. 2023 ) . Prior work that has looked at costs focuses on the geography but ignores the scale of the concept itself Uzkent & Ermon 2020 . A holistic treatment of scale to achieve accurate recognition under a budget is missing. In this paper, we address this gap by proposing: • A knowledge distillation technique which allows LR models to improve significantly in recognizing finer concepts by learning from HR. • A novel approach that leverages the semantic understanding of LLMs to determine the scale of each concept. • A new approach for determining which geographical regions require higher resolution analysis by predicting when low and high resolution models might disagree, and • A unified framework that combines these ideas to yield the most accurate retrieval results for a range of concepts while adhering to a strict budget constraint. Finally, we note that while our experiments are on LR and HR satellite imagery, these techniques are more general and can generalize to any problem where there are two or more modalities with different costs and accuracy tradeoffs. We evaluate our approach with multiple recognition models (supervised and open-vocabulary) and multiple satellite modalities. We find that compared to simply using high resolution images always, our framework improves accuracy up to 13 points while reducing the number of HR images used by 5 \times . Our approach also significantly outperforms (by more than 25 points) other prior work that trades off between accuracy and cost. In sum, our results demonstrate that our holistic reasoning of scale leads to significantly higher accuracy with large reductions in cost. 2 Related Works 2.1 Multi-scale Recognition in Satellite Imagery Remotely sensed images or satellite images are inherently multi-scale. Depending on the sensor heights and properties, the physical distance between two adjacent pixels in remotely sensed images (known as Ground Sample Distance/GSD) could vary from 0.3m to 1km. These multi-scale images provide complementary information for various applications ( Wicaksono et

Read the complete text