SAR-to-RGB Translation with Latent Diffusion for Earth Observation

Kaan Aydin, Joëlle Hanna, Damian Borth · 2025

Read SAR-to-RGB Translation with Latent Diffusion for Earth Observation by Kaan Aydin, Joëlle Hanna, Damian Borth (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11154.txt.

Introduction

Kaan Aydin, Joelle Hanna, and Damian Borth turn radar satellite images into optical-looking ones. Their paper came out in two thousand twenty-five. Sentinel-1 radar sees through clouds, while Sentinel-2 gives rich color images. Clouds and gaps often make the color record incomplete. A translator that fakes color views from radar could fill those holes.

The authors test three diffusion-based setups. Two add and remove noise in the standard way, one with class labels as a guide. The third blends radar and optical data, then strips the radar part away. They judge the fakes by real tasks, not just looks. Land cover sorting and cloud removal both stay useful. Class guidance lifts sorting accuracy most. The work helps Earth watchers keep steady records through cloudy spells.

Catalog record

CREATORKaan Aydin, Joëlle Hanna, Damian Borth
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-11154

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SAR-to-RGB Translation with Latent Diffusion for Earth Observation Kaan Aydin, Joëlle Hanna, Damian Borth Abstract Earth observation satellites like Sentinel-1 (S1) and Sentinel-2 (S2) provide complementary remote sensing (RS) data, but S2 images are often unavailable due to cloud cover or data gaps. To address this, we propose a diffusion model (DM)-based approach for SAR-to-RGB translation, generating synthetic optical images from SAR inputs. We explore three different setups: two using Standard Diffusion, which reconstruct S2 images by adding and removing noise (one without and one with class conditioning), and one using Cold Diffusion, which blends S2 with S1 before removing the SAR signal. We evaluate the generated images in downstream tasks, including land cover classification and cloud removal. While generated images may not perfectly replicate real S2 data, they still provide valuable information. Our results show that class conditioning improves classification accuracy, while cloud removal performance remains competitive despite our approach not being optimized for it. Interestingly, despite exhibiting lower perceptual quality, the Cold Diffusion setup performs well in land cover classification, suggesting that traditional quantitative evaluation metrics may not fully reflect the practical utility of generated images. Our findings highlight the potential of DMs for SAR-to-RGB translation in RS applications where RGB images are missing. 1 Introduction Earth observation plays an important role in environmental monitoring Hanna et al. 2023a , disaster management Adriano et al. 2020 , and land-use analysis Ramos and Sappa 2024 ; Hanna et al. 2023b ; Scheibenreif et al. 2022 . The Sentinel-1 (S1) Torres et al. 2012 and Sentinel-2 (S2) Drusch et al. 2012 missions, part of the European Space Agency’s Copernicus program, provide complementary remote sensing (RS) data. S1 uses Synthetic Aperture Radar (SAR) to capture detailed images of Earth’s surface, regardless of weather conditions and time of day. In contrast, S2 employs a multispectral instrument to capture optical images across 13 spectral bands. Combining these two modalities enables comprehensive surface monitoring, improving applications such as vegetation analysis, change detection, and disaster response Hafner et al. 2022 ; Prodromou et al. 2023 ; Notarnicola et al. 2017 . However, S2 images are often unavailable or difficult to use due to cloud cover, bad weather, or gaps in data collection, resulting in missing or incomplete information. This lack of paired S1-S2 observations poses challenges for multimodal applications that rely on both data sources. SAR-to-optical image translation aims to bridge this gap by generating S2 images from S1. This task is inherently challenging due to the fundamental differences between SAR and optical image modalities. SAR captures surface structure and texture but lacks spectral reflectance information, making the direct translation difficult. In this work, we focus on translating SAR images to the RGB bands of S2, as they are the most widely used for visual interpretation and downstream applications. Traditionally, Generative Adversarial Networks (GANs) Goodfellow et al. 2014 have been used for SAR-to-RGB translation, but they suffer from mode collapse and instability during training. To overcome these limitations, we propose a Diffusion Model (DM) for SAR-to-RGB translation to generate high-fidelity optical images. Compared to GANs, DMs offer more stable training and better preservation of structural details Dhariwal and Nichol 2021 . Our approach consists of three steps. First, we train a DM to learn the mapping from S1 to S2 images. Second, we generate synthetic S2 images from S1 inputs. Third, we evaluate the generated images in practical downstream tasks, such as land cover classification and cloud removal, to assess both their visual quality and usability in RS applications. In summary, the contributions of our work are as follows: 1. We analyze different diffusion strategies (Standard vs. Cold Diffusion) and conditioning strategies (with and without class conditioning) for SAR-to-RGB translation, exploring how to best leverage DMs for RS applications. 2. We systematically evaluate how well the synthetically generated RGB images perform in real-world RS tasks, such as land cover classification and cloud removal, beyond just their visual quality. 2 Related Works 2.1 SAR-to-RGB translation SAR-to-RGB translation is a specialized task within the broader field of Image-to-Image (I2I) translation that aims to convert SAR imagery into visible RGB images. Recent advances in deep learning, particularly GANs Goodfellow et al. 2014 , have shown promising results in this domain. For example, Fu et al . Fu et al. 2019 proposed a GAN-based approach for translating SAR images into optical images, utilizing multiscale discriminators to improve the quality of the generated images. Despite its advancements, this method faces challenges such as producing blurred boundaries and failing to accurately capture the geometric details of certain objects, particularly buildings. The remarkable success of DMs in image translation has led to a growing body of research focused on leveraging their capabilities. For instance, Bai et al . Bai et al. 2024 proposed a conditional diffusion approach for translating SAR images to RGB, utilizing a U-Net Ronneberger et al. 2015 architecture as the backbone. In parallel, more sophisticated approaches have explored broader applications of DMs. Berian et al . Berian et al. 2025 introduced CrossModalityDiffusion , a framework designed for multi-modal novel view synthesis. This method incorporates modality-specific encoders alongside DMs to generate consistent outputs across diverse modalities. Although not specifically focused on SAR-to-RGB translation, this approach highlights the broader potential of DMs in facilitating image translation across a wide range of modalities. 2.2 Applications of DMs in RS Since their introduction to RS in 2021, DMs have gained significant traction, demonstrating great potential in advancing image processing and analysis for RS Liu et al. 2024 . Beyond SAR-to-RGB translation, DMs have impacted various other areas of RS. For example, Khanna et al. Khanna et al. 2024 integrated additional numerical metadata, such as geolocation and sampling time, as prompts for a Stable Diffusion (SD) model. This approach enriched the model’s input, improving its ability to generate high-quality satellite images. Super-resolution is another domain benefiting from DMs. Han et al. Han et al. 2023 combined Transformer and CNN architectures to extract global and local features from multispectral low-resolution images. These fused feature representations were then used to guide a DM in generating high-resolution images in a single training step. Cloud removal is also a critical challenge in RS, as optical images are frequently obstructed by clouds. Zou et al. Zou et al. 2023 addressed this issue by developing DiffCR, a method that first extracts features from the cloudy image and its noise level. These extracted spatial and temporal features are then incorporated into the DM as control conditions, enhancing its ability to reconstruct cloud-free images. 2.3 Advancements in DMs DMs have emerged as the state-of-the-art approach for generative modeling and have overtaken GANs due to their ability to produce high-quality output with better diversity and a more stable training regime. DMs were first introduced by Sohl-Dickstein et al. Sohl-Dickstein et al. 2015 and started to garner more attention following the publication of the Denoising Diffusion Probabilistic Model (DDPM) by Ho et al. Ho et al. 2020 who formalized the framework as a generative process based on a forward diffusion process that gradually adds noise to data and a learned reverse process that denoises it step by step. Subsequent works introduced innovations that span multiple directions. Rombach et al . Rombach et al. 2022

Read the complete text