Global and Dense Embeddings of Earth: Major TOM Floating in the Latent Space

\name Mikolaj Czerkawski* \email [email protected], \name Marcin Kluczek* \email [email protected], \name Jędrzej S. Bojanowski \email [email protected], \addr *Equal Contribution · 2024

Read Global and Dense Embeddings of Earth: Major TOM Floating in the Latent Space by \name Mikolaj Czerkawski* \email [email protected], \name Marcin Kluczek* \email [email protected], \name Jędrzej S. Bojanowski \email [email protected], \addr *Equal Contribution (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05600.txt.

Introduction

Czerkawski, Kluczek, and Bojanowski (2024) extend a community project that standardizes Earth observation data for AI use. Archives from programs like Copernicus hold more satellite imagery than anyone can label, so a practical route is to compress each image into a feature vector with a pretrained network. How to do that in a standard way for geospatial archives had not been defined.

The work plugs into the Major TOM project, which provides open, AI-ready Earth datasets. Alongside the paper, the team releases four global, dense embedding datasets for free. They describe the result as the most comprehensive open collection of geospatial visual embeddings by land surface covered. The authors aim to make satellite archives searchable and comparable at planetary scale.

Catalog record

CREATOR\name Mikolaj Czerkawski* \email [email protected], \name Marcin Kluczek* \email [email protected], \name Jędrzej S. Bojanowski \email [email protected], \addr *Equal Contribution
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-05600

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Global and Dense Embeddings of Earth: Major TOM Floating in the Latent Space \name Mikolaj Czerkawski* \email [email protected], \name Marcin Kluczek* \email [email protected], \name Jędrzej S. Bojanowski \email [email protected], \addr *Equal Contribution Abstract With the ever-increasing volumes of the Earth observation data present in the archives of large programmes such as Copernicus, there is a growing need for efficient vector representations of the underlying raw data. The approach of extracting feature representations from pretrained deep neural networks is a powerful approach that can provide semantic abstractions of the input data. However, the way this is done for imagery archives containing geospatial data has not yet been defined. In this work, an extension is proposed to an existing community project, Major TOM, focused on the provision and standardization of open and free AI-ready datasets for Earth observation. Furthermore, four global and dense embedding datasets are released openly and for free along with the publication of this manuscript, resulting in the most comprehensive global open dataset of geospatial visual embeddings in terms of covered Earth’s surface. Keywords: Earth observation, Copernicus Data, Image Embeddings, AI datasets 1 Introduction There is one trend in the domain of information technologies that does seem more consistent than anything else in recent decades, and it is the growth of produced data. Whether it is a piece of information generated online or a series of bytes transmitted by a satellite, the rate at which new data are produced keeps increasing every year. While every new piece of data can be considered valuable and beneficial, the increasing volume makes large-scale analyses more challenging and computationally expensive. This phenomenon is already being observed in the domain of Earth observation data, where every single year, petabytes of new data captured by the satellites enter the archives. For that reason, it is crucial to develop methods capable of navigating vast databases of imagery at speed and low computational costs. Figure 1: The pipeline building Major TOM embedding expansions according to the proposed standard. It begins with grid cell fragmenting, image preprocessing, and embedding and packing into the geoparquet archive format. A solution to this challenge can come in the form of extracting semantic features from the visual data at a large scale, in a process known as embedding ( Frome et al., 2013 ; Mikolov et al., 2013 ) . While manually engineered features could be used to compress the images into efficient vector representations, this approach has predominantly become effective with the development of large pre-trained deep neural networks, which can learn powerful representations of the input data. This includes methods, which rely solely on self-supervision. The process of embedding data into a compressed representation space is far from new and has been explored for over a decade ( Mikolov et al., 2013 ; Frome et al., 2013 ) , accelerated by the growth in popularity of deep learning pipelines and models ( Gordo et al., 2016 ) . Visual descriptors, which play a similar role to embeddings (and sometimes could even be synonymous) have also been a popular topic for many years in computer vision ( Tolias and Jégou, 2014 ; Philbin et al., 2007 ; Shen et al., 2014 ; Radenović et al., 2015 ) . In the field of Earth observation, several works have explored the use of vector representations for geospatial data. Most importantly, this includes work that attempted to learn meaningful representations of geospatial coordinates ( Yin et al., 2019 ; Yin et al., 2021 ; Tarasiou and Zafeiriou, 2022 ; Klemmer et al., 2023 ) . However, most of these efforts focus on learning a new latent space rather than applying an existing embedding function at a large scale. In this direction, LAION-EO ( Czerkawski and Francis, 2023 ) is a notable work, where 5 billion of images coming from the web (LAION-5B dataset ( Schuhmann et al., 2022 ) ) have been embedded with a CLIP model ( Radford et al., 2021 ) and used to retrieve instances of satellite imagery from billions of unorganized internet data. This work attempts to explore the direction of embedding Earth observation images at the global scale with several types of existing models and propose a standard for doing so. For that reason, it largely builds on top of an ongoing community project of Major TOM ( Francis and Czerkawski, 2024 ) , which aims to standardize the curation of large Earth observation datasets and set up paths for community-based dataset growth. This work is one example of such a growth, and just like the first Major TOM release, it aims at easy and free access to the datasets and accompanying tools for interacting with them. The definition of a standard for embedding expansions brings several benefits. A clear definition leads to reproducible work and embedding databases that can be analyzed, debugged, and fixed if needed. It also ensures that the embedding datasets produced by independent entities are compatible with each other, at least to some degree. It also makes the evaluation of the models used for producing the embeddings more accessible. By precomputing the latent features from a model, the embedding dataset producer removes the larger part of the computational burden, making it possible to evaluate the output representations for various models at a low cost. Finally, by building on top of the Major TOM grid system, embeddings generated according to the standard can be used as aligned embedding time-series data with no further adjustments since the sample footprints are already aligned. This work involves computation and analysis of global embeddings from 4 different pre-trained models. This is done in order to highlight the vastly different behaviors, strengths, and weaknesses. It is unlikely that the space of Earth observation will be dominated by a single general-purpose model. At the moment, the ecosystem of AI for satellite imagery already involves a large number of models to choose from, operating on various formatting (such as normalization, numbers of bands, processing level) of data, even within a domain of a single sensor. For that reason, an organized practice of inter-comparison between different embedding model candidates is going to be crucial. In the first release of Major TOM embeddings, over 169 million embeddings are released as a result of processing more than 62 TB of raw data, encompassing over 3.5 million unique images. The dataset distills information from approximately 9.368 trillion pixels of source imagery. At the time of release, this is the first open dataset of Copernicus embeddings built at this scale, with dense and global coverage across the full acquisition area of the sensor. The remainder of this paper describes the embedding methodology, the release standard, the fragmenting approach, the models selected for this release, and finally, some early visualizations of the global embedding data. 2 Embedding Methodology The global embedding dataset is built on top of the Major TOM Core datasets ( Francis and Czerkawski, 2024 ) , which enable fast and free access to more than 60 TB of AI-ready Copernicus data, densely covering the globe with multiple sensing modalities and processing levels. In this release, three core datasets were used: Major-TOM/Core-S2L1C , Major-TOM/Core-S2L2A , and Major-TOM/Core-S1RTC , corresponding to Sentinel-2 L1C and L2A, as well as

Read the complete text