TerraGen: A Unified Multi-Task Layout Generation Framework for Remote Sensing Data Augmentation

Datao Tang, Hao Wang, Yudeng Xin, Hui Qiao, Dongsheng Jiang, Yin Li, Zhiheng Yu, Xiangyong Cao · 2025

Read TerraGen: A Unified Multi-Task Layout Generation Framework for Remote Sensing Data Augmentation by Datao Tang, Hao Wang, Yudeng Xin, Hui Qiao, Dongsheng Jiang, Yin Li, Zhiheng Yu, Xiangyong Cao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21391.txt.

Introduction

Datao Tang and seven co-authors build TerraGen, a generator that creates labeled satellite-style imagery for several vision tasks at once. Remote sensing work needs large labeled collections, yet older generators train a separate model per task and ignore geography, so their fake scenes can break real spatial rules.

TerraGen takes a layout, given as boxes or masks. Renders a matching image, guided by an encoder that unifies both layout kinds and by training signals that stress spatial fit at every scale. The team also built a shared data set of forty-five thousand images and a standard scoring protocol for this multi-task setting. Their generator earns the best image quality across tasks and, used to augment training data, lifts downstream results in both full-data and few-example regimes.

Catalog record

CREATORDatao Tang, Hao Wang, Yudeng Xin, Hui Qiao, Dongsheng Jiang, Yin Li, Zhiheng Yu, Xiangyong Cao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21391

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TerraGen: A Unified Multi-Task Layout Generation Framework for Remote Sensing Data Augmentation Datao Tang, Hao Wang, Yudeng Xin, Hui Qiao, Dongsheng Jiang, Yin Li, Zhiheng Yu, Xiangyong Cao Abstract Remote sensing vision tasks require extensive labeled data across multiple, interconnected domains. However, current generative data augmentation frameworks are task-isolated, i.e., each vision task requires training an independent generative model, and ignores the modeling of geographical information and spatial constraints. To address these issues, we propose TerraGen , a unified layout-to-image generation framework that enables flexible, spatially controllable synthesis of remote sensing imagery for various high-level vision tasks, e.g., detection, segmentation, and extraction. Specifically, TerraGen introduces a geographic-spatial layout encoder that unifies bounding box and segmentation mask inputs, combined with a multi-scale injection scheme and mask-weighted loss to explicitly encode spatial constraints, from global structures to fine details. Also, we construct the first large-scale multi-task remote sensing layout generation dataset containing 45k images and establish a standardized evaluation protocol for this task. Experimental results show that our TerraGen can achieve the best generation image quality across diverse tasks. Additionally, TerraGen can be used as a universal data-augmentation generator, enhancing downstream task performance significantly and demonstrating robust cross-task generalisation in both full-data and few-shot scenarios. Figure 1: Generation results of the TerraGen model and composition of the TerraGen dataset. The main visuals display high-fidelity remote sensing images synthesized by our TerraGen model from given layout conditions. The pie chart in the top-right corner illustrates the five-task distribution of our purpose-built TerraGen dataset. 1 Introduction Deep learning has revolutionized computer vision, yet its success is fundamentally predicated upon large-scale annotated datasets. This reliance poses a significant bottleneck in remote sensing, where data acquisition is costly and annotation demands specialized expertise across diverse tasks like object detection, segmentation, and change detection. Remote sensing tasks exhibit strong spatial and semantic correlations, sharing common geographic layouts. However, current research paradigms are hampered by redundant annotation and isolated task pipelines. A single scene often requires distinct labels for each task (e.g., bounding boxes and masks), while data augmentation techniques seldom exploit inter-task consistency. The advent of controllable generative models, such as GLIGEN ( Li et al. 2023 ) and InstanceDiffusion ( Wang et al. 2024 ) , has established powerful paradigms for image synthesis conditioned on text and layout. While effective for natural images, their application in remote sensing remains confined to single-task or single-condition generation, thus lacking flexible multimodal control and cross-task adaptability. More critically, remote sensing generative models ( Zhang et al. 2024b ; Zang et al. 2025 ; Toker et al. 2024 ; Tang et al. 2025 ) lack standardized evaluation benchmarks. Existing approaches are typically designed for specific tasks without unified multi-task evaluation frameworks, hindering both technical progress and practical adoption. This limitation is compounded by the unique geographic constraints in remote sensing imagery—such as road network connectivity, building arrangement patterns, and spatial relationships between land cover types—that are poorly captured by general-purpose generation methods. The central premise of this paper is that the challenge of task isolation can be overcome by identifying a universal medium. We argue that spatial layout information serves as this universal representation, bridging different remote sensing vision tasks, while multimodal conditional control enhances generation flexibility and precision. This premise is built on three key observations: First, spatial representations across tasks (bounding boxes in detection, pixel masks in segmentation, contours in instance segmentation) fundamentally describe the same geographic object distribution patterns with shared geometric and semantic foundations. Second, geographic object layouts follow specific spatial rules that can be effectively encoded through structured representations. Third, textual descriptions provide semantic details that complement layout conditions, enabling more precise and diverse generation control, as shown in Figure 1 . Based on these insights, we introduce TerraGen (from the Latin ’Terra’ for Earth), a multi-conditional generation framework for remote sensing imagery. Our contributions are threefold: • We construct the first large-scale multi-task remote sensing layout generation dataset and establish standardized evaluation protocols, addressing the critical lack of unified benchmarks in this field. • We propose a unified multi-conditional layout generation framework (TerraGen) that integrates spatial layout information (bounding boxes, segmentation masks) with semantic textual information through geographic spatial-aware conditional encoding mechanisms, achieving fine-grained generation control of complex remote sensing scenes. • Our TerraGen can serve as a universal data-augmentation engine, markedly boosting downstream-task accuracy and exhibiting strong generalisation across tasks under both full-data and few-shot scenarios. 2 Related Work 2.1 Text-to-Image Generation Text-to-image generation has evolved from GAN-based models ( Reed et al. 2016 ) to diffusion models ( Ramesh et al. 2021 ; Ramesh et al. 2022 ; Saharia et al. 2022 ; Nichol et al. 2021 ) , offering improved stability and visual quality. Recent methods ( Radford et al. 2021 ; Podell et al. 2023 ; Esser et al. 2024 ; Labs et al. 2025 ) adopt Multimodal Diffusion Transformers (MM-DiT), treating text as an independent modality and enhancing synthesis via multimodal attention. However, they underperform on remote sensing data due to its unique structures and patterns ( Tang et al. 2024 ) . 2.2 Layout-to-Image Generation Layout-to-image generation extends synthesis by introducing fine-grained spatial control, typically through structured inputs like bounding boxes or segmentation masks. In the natural image domain, models like GLIGEN ( Li et al. 2023 ) and LayoutDiffusion ( Zheng et al. 2023 ) achieve impressive results by injecting spatial guidance into the cross-attention layers of pre-trained diffusion models. Recent MM-DiT-based extensions ( Zhang et al. 2024a ; Zhang et al. 2025b ; Zhang et al. 2025a ) further advance this by integrating layout as another input modality, demonstrating strong layout fidelity. Nonetheless, these methods often rely on massive, web-scale datasets (e.g., COCO, LVIS) and still struggle with pixel-level precision for complex, overlapping instances. In the remote sensing domain, layout-conditioned generation is an emerging but critical area. Early methods like CC-Diff ( Zhang et al. 2024b ) and AeroGen ( Tang et al. 2025 ) have focused on bounding box-guided generation for object detection, while others like SatSynth ( Toker et al. 2024 ) have explored mask-guided synthesis for segmentation. Despite their progress, these approaches are often task-specific and tend to overlook crucial domain-specific constraints. For instance, they may generate objects that are spatially plausible in isolation but violate geographic rules, such as disconnected road networks, randomly oriented buildings, or illogical land-use adjacencies. Our framework moves beyond this by proposing a unified model that handles multiple layout types and is aware of these geospatial relationships. 2.3 Generative Augmentation in Remote Sensing The high cost of expert annotation and the inherent long-tail distribution of objects in remote sensing have motivated the use of generative models for data augmentation. Unlike traditional augmentation (e.g., rotation, scaling), diffusion-based methods can create novel, diverse, and highly realistic image-label pairs. Early successes like DiffuSat ( Khanna et al. 2023 ) and CRS-Diff ( Tang et al. 2024 ) demonstrated the potential of conditional generation to boost the performance of downstream segmentation models. SatSynth ( Toker et al. 2024 ) further validated this approach for both segmentation and detection tasks. However, a significant limitation of existing work is its fragmented, task-specific nature. Current pipelines require training separate generative models for detection, segmentation, or

Read the complete text