Pixel-Wise Multimodal Contrastive Learning for Remote Sensing Images

Leandro Stival, Ricardo da Silva Torres, Helio Pedrini · 2026

Read Pixel-Wise Multimodal Contrastive Learning for Remote Sensing Images by Leandro Stival, Ricardo da Silva Torres, Helio Pedrini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04127.txt.

Introduction

Leandro Stival, Ricardo da Silva Torres, and Helio Pedrini propose learning from satellite image time series one pixel at a time. Satellites pour out Earth observation data, yet most deep models read whole images or whole series at once. The team instead turns each pixel's vegetation index history into a two-dimensional recurrence plot.

Their method, PIMC, pairs those plots with the satellite imagery in a multimodal contrastive setup that needs no labels. Plots are built from three vegetation indices that track plant growth over time. Tests cover pixel-level forecasting and sorting on the PASTIS data set, plus land cover sorting on EuroSAT. The three authors report that two-dimensional forms sharpen feature extraction and that contrastive training lifts quality for both views, beating current methods on the tasks tried.

Catalog record

CREATORLeandro Stival, Ricardo da Silva Torres, Helio Pedrini
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-04127

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pixel-Wise Multimodal Contrastive Learning for Remote Sensing Images Leandro Stival, Ricardo da Silva Torres, Helio Pedrini Abstract Satellites continuously generate massive volumes of data, particularly for Earth observation, including satellite image time series (SITS). However, most deep learning models are designed to process either entire images or complete time series sequences to extract meaningful features for downstream tasks. In this study, we propose a novel multimodal approach that leverages pixel-wise two-dimensional (2D) representations to encode visual property variations from SITS more effectively. Specifically, we generate recurrence plots from pixel-based vegetation index time series (NDVI, EVI, and SAVI) as an alternative to using raw pixel values, creating more informative representations. Additionally, we introduce PIxel-wise Multimodal Contrastive (PIMC), a new multimodal self-supervision approach that produces effective encoders based on two-dimensional pixel time series representations and remote sensing imagery (RSI). To validate our approach, we assess its performance on three downstream tasks: pixel-level forecasting and classification using the PASTIS dataset, and land cover classification on the EuroSAT dataset. Moreover, we compare our results to state-of-the-art (SOTA) methods on all downstream tasks. Our experimental results show that the use of 2D representations significantly enhances feature extraction from SITS, while contrastive learning improves the quality of representations for both pixel time series and RSI. These findings suggest that our multimodal method outperforms existing models in various Earth observation tasks, establishing it as a robust self-supervision framework for processing both SITS and RSI. K eywords Remote Sensing Images, Self-Supervised Learning, Pixel Time Series, Contrastive Learning 1 Introduction The use of remote sensing imagery (RSI) and multispectral remote sensing imagery (MSRSI) is crucial for Earth Observation (EO) tasks, such as forest monitoring ( 4 ; 30 ) , climate change analysis ( 12 ) , and crop estimation ( 52 ) . To effectively monitor changes and events on the Earth’s surface, it is essential to utilize RSI and incorporate temporal information about these locations. In this context, satellite image time series (SITS) have proven to be highly effective for supporting these tasks ( 51 ) . Nowadays, most of these images and time series are processed using machine learning models. However, obtaining high-quality labeled data remains a significant challenge in this field, as labeling SITS requires substantial time and effort from experts. Given the labor-intensive nature of this task, one of the most common approaches to training models on SITS and RSI is through self-supervised methods, such as contrastive learning ( 38 ; 36 ) . Most studies on using SSL for SITS concentrate on processing entire images to create robust feature representations. However, processing the entire image can often lead to higher computational costs and unnecessary overhead, especially in cases where only specific parts of the RSI are the focus of the analysis or monitoring. In this context, approaches that utilize only pixel information from SITS have shown promising results for SITS classification ( 17 ) . However, these methods rely solely on pixel values to represent the time series, overlooking the potential of using features extracted from RSI that offer richer spatial context information (e.g., geometric and morphological metrics, vegetation indices, and texture patterns). In this study, we use vegetation indices to process SITS pixels, allowing us to utilize more detailed information to train effective encoders. Pixel-based SITS representations typically consist of one-dimensional (1D) data, where pixel values at each timestamp are stored and processed. This approach tends to overlook internal relationships between the values within the series ( 27 ) . To enhance the representation of pixel data from SITS, we use the recurrence plot (RP) ( 14 ) method, which transforms a 1D time series into a 2D representation. The choice of the recurrence plot is motivated by its successful use in classification and retrieval tasks, including SITS and near-surface image sequences ( 10 ; 40 ; 15 ; 11 ) . This not only makes temporal correlations among time samples explicit but also allows us to explore the use of machine learning models well established for computer vision tasks. RP has also been successfully employed in time series analyses in multiple domains such as statistical analyses ( 8 ) , magnetic resonance ( 2 ) , electroencephalogram (EEG) ( 18 ) , and sports injury ( 59 ) . Nevertheless, the aforementioned studies merely focus on using transfer learning methods using pre-trained computer vision models in supervised learning scenarios. In contrast, our approach employs 2D representations more generically, enabling the utilization of unlabeled datasets for computing representations of multiple modalities using the same input sample (remote sensing image). In this paper, we introduce Pixel-Wise Image Multimodal Contrastive (PIMC), a multimodal contrastive method for training machine learning models using 2D representations of the pixel information from SITS alongside RSI. To validate the superiority of encoders trained with the 2D representation, we benchmarked models trained in a supervised manner on 1D time series and compared their performance with the self-supervised PIMC. This paper addresses the following research questions: • RQ1: How to create a multimodal framework that utilizes two-dimensional time series representations and MSRSI to train a machine learning model in a self-supervised multimodal contrastive way using vegetation index information from SITS? • RQ2: Are the machine learning models trained following the PIMC protocol effective in different downstream tasks (e.g., time series classification and forecasting, and image classification)? • RQ3: Would the PIMC approach yield results comparable to or better than state-of-the-art supervised and self-supervised models in downstream tasks? In short, this work introduces a new multimodal self-supervised contrastive learning to train effective encoders and investigates their use in downstream tasks involving time series and RSI data only. To the best of our knowledge, this is the first study concerning the use of such modalities in self-supervision learning based on contrastive learning. The main contributions of this paper are: • A new contrastive self-supervised method to align the latent spaces from pixel-wise time series and remote sensing images. • Demonstration of the effectiveness of encoders when trained using 2D representations of pixel-wise temporal information in SITS in various RSI-related downstream tasks. • Demonstration of the effectiveness of trained encoders when compared with SOTA methods in different downstream tasks involving both modalities. 2 Background and Related Work This section overviews background concepts and relevant studies to the topic under investigation in this paper. 2.1 Remote Sensing Images The widespread use of remote sensing imaging enables the daily production of vast amounts of data from the Earth’s surface. This abundance of data has fueled the growing application of machine learning models to a wide range of Earth observation (EO) tasks. In examining the current literature, we can highlight several studies ( 37 ; 38 ; 3 ) that utilize self-supervision methods to create effective feature extractors for various tasks, such as land use classification, semantic segmentation, and change detection. Following this idea of domain generalization for EO applications, foundational models have also led to an increase in the number of publications proposing models that can be used or fine-tuned for common EO tasks ( 25 ; 20 ; 56 ) . Nevertheless, all of these methods for creating RSI representations share one common aspect: they utilize the entire image. However, since we are discussing RSI,

Read the complete text