Top2Ground : A Height-Aware Dual Conditioning Diffusion Model for Robust Aerial-to-Ground View Generation
Jae Joong Lee, Bedrich Benes · 2025
Read Top2Ground : A Height-Aware Dual Conditioning Diffusion Model for Robust Aerial-to-Ground View Generation by Jae Joong Lee, Bedrich Benes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08258.txt.
Introduction
Jae Joong Lee and Bedrich Benes introduce Top2Ground, a diffusion model that generates ground level views from aerial images. The viewpoint gap is extreme, with heavy occlusion and a narrow field of view at street level. Rather than building depth maps or voxel models first, the method generates the ground image directly from the aerial input.
Denoising is conditioned jointly on spatial features from the aerial image and an estimated height map, plus semantic embeddings from CLIP. Geometry and content therefore guide the result together. On three data sets covering wide and narrow views, structural similarity improves by about seven percent on average over rivals.
Catalog record
| CREATOR | Jae Joong Lee, Bedrich Benes |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08258
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Top2Ground : A Height-Aware Dual Conditioning Diffusion Model for Robust Aerial-to-Ground View Generation Jae Joong Lee, Bedrich Benes Abstract Generating ground-level images from aerial views is a challenging task due to extreme viewpoint disparity, occlusions, and a limited field of view. We introduce Top2Ground , a novel diffusion-based method that directly generates photorealistic ground-view images from aerial input images without relying on intermediate representations such as depth maps or 3D voxels. Specifically, we condition the denoising process on a joint representation of VAE-encoded spatial features (derived from aerial RGB images and an estimated height map) and CLIP-based semantic embeddings. This design ensures the generation is both geometrically constrained by the scene’s 3D structure and semantically consistent with its content. We evaluate Top2Ground on three diverse datasets: CVUSA, CVACT, and the Auto Arborist. Our approach shows 7.3% average improvement in SSIM across three benchmark datasets, showing Top2Ground can robustly handle both wide and narrow fields of view, highlighting its strong generalization capabilities. 1 Introduction Aerial imagery captured by UAVs is increasingly used in applications such as urban planning, environmental monitoring, and 3D mapping, due to its scalability and low acquisition cost. With over one million UAVs registered in the U.S. 1 and a projected 6.4% annual growth rate 7 , the demand for extracting actionable insights from aerial views will likely continue to rise. Yet, many downstream tasks, such as infrastructure inspection, vegetation analysis, or geolocation, require fine-grained ground-level information that is not directly visible from above. Capturing such data in remote, restricted, or hazardous environments remains logistically challenging and costly. It is relatively simple to fly a UAV over a specific area, but retrieving ground-view images is complicated. This motivates the task of aerial-to-ground view synthesis , i.e., generating plausible, photorealistic ground-level images from aerial input images. The viewpoint is learned implicitly from the training data, which contains paired aerial and ground-level images, and our model learns this specific statistical mapping rather than performing a full 3D reconstruction. However, this task is inherently ill-posed due to extreme viewpoint changes, occlusions, and ambiguous ground semantics. Deep neural models can ease this task by learning the correspondence beforehand and estimating the ground views by conditioning on the aerial views. Early methods based on CNNs 35 ; 13 and GANs 24 ; 30 ; 18 learn direct mappings, but often struggle with spatial distortion and semantic inconsistency. More recent approaches introduce intermediate geometric reasoning to improve fidelity, such as density maps 21 or voxel-based projections 5 . The main limitation of these methods is that they require additional annotations or introduce significant computational overhead, which limits their scalability and deployment. We introduce Top2Ground , a novel diffusion-based framework that directly synthesizes ground-view images from aerial inputs. Our contribution is a new modeling principle for this task: the fusion of spatial (VAE), geometric (VAE on height map), and semantic (CLIP) information. This approach eliminates the need for complex 3D intermediate representations, such as voxels, a major limitation of prior work. Our approach introduces a height-aware dual conditioning that leverages two pre-trained embedding spaces. These modules play distinct and complementary roles: the VAE, using the aerial RGB and height map, acts as an “architect,” capturing fine-grained structural and geometric details. In contrast, CLIP acts as a “semantic director,” understanding the high-level context of the scene. Moreover, we condition the model on both the aerial RGB image and its estimated height map. This height prior acts as a geometric constraint, ensuring the synthesized ground view is geometrically consistent with the specific input aerial image, not just any plausible view. This dual conditioning is a more holistic approach than standard depth control, designed specifically for the extreme viewpoint shift between the aerial and ground domains. Unlike prior methods, Top2Ground is robust to both wide-field (e.g., CVUSA, CVACT) and narrow-field (e.g., AAD) aerial imagery. This generalization capability is essential for real-world UAV scenarios, where camera configurations vary and the field of view may be limited. By operating entirely in the latent space of a pretrained diffusion model and using classifier-free guidance, Top2Ground achieves high-quality synthesis while remaining efficient. We validate Top2Ground on three diverse datasets: CVUSA 33 , CVACT 28 , and the Auto Arborist Dataset (AAD) 2 . Our results demonstrate state-of-the-art performance across perceptual, semantic, and pixel-level metrics. Our model achieves an average improvement of 7.3% in SSIM, and 44.9% in KID across the three benchmark datasets. Our contributions are summarized as follows: 1. Height-aware diffusion without intermediate representations. We propose a diffusion-based architecture that synthesizes ground views directly from aerial imagery, eliminating the need for 3D voxel or density map intermediates. 2. Dual conditioning with semantic and spatial guidance. Our model jointly leverages CLIP (for semantic context) and VAE (for structural/geometric detail) embeddings extracted from aerial RGB and height maps, enhancing both structural fidelity and semantic alignment. 3. Robust generalization across wide and narrow aerial views. Top2Ground performs consistently across CVUSA, CVACT, and AAD, including challenging non-panoramic, narrow-FOV settings common in UAV-based deployments. 2 Related Work Diffusion-based image generation has become the state-of-the-art in generative modeling by synthesizing high-fidelity images through iterative denoising 8 . Score-based models 29 , guided sampling 6 , and latent diffusion 25 have improved both sample quality and training efficiency. Beyond unconditional generation, recent approaches incorporate various conditioning mechanisms. Text-guided models such as GLIDE 19 , Imagen 26 , and DiffusionCLIP 10 leverage language embeddings to control semantic outputs. InstructPix2Pix 3 refines text-to-image alignment through paired image editing. More structured conditioning has been explored via ControlNet 36 , which adds learnable branches for edge maps and segmentation masks, and Tree-D Fusion 11 , which integrates spatial priors for scene-aware control. These advances demonstrate the capacity of diffusion models for flexible and photorealistic image synthesis under various conditioning modalities. Aerial-to-ground image synthesis. Cross-view image synthesis aims to generate ground-level views from aerial inputs, a task complicated by extreme viewpoint changes, occlusions, and limited field-of-view overlap. Early approaches employed CNN-based 35 ; 13 or GAN-based 24 ; 30 ; 18 architectures to learn direct mappings, but they often fail to preserve spatial consistency and fine details. Geometry-guided models have introduced intermediate representations, such as height maps 27 , density fields 21 , or voxel reconstructions 5 , to improve fidelity. However, these methods require accurate geometric priors or computationally expensive estimation, and often struggle to model high-level semantics or generalize across diverse viewpoints. We also note that NeRF-based view synthesis methods are not directly comparable, as they typically require multiple input views, whereas our task is single-image synthesis. Top2Ground departs from prior work by removing the dependency on intermediate geometric representations. Instead, we propose a dual-conditioning latent diffusion framework that jointly uses a CLIP-based semantic encoder and a VAE-based visual encoder, which operates on both aerial RGB images and their estimated height maps. This enables our model to generate ground-level views that are both structurally coherent and semantically aligned, without requiring 3D voxels, density estimation, or depth supervision. Notably, prior works have focused mainly on panoramic imagery with wide aerial fields of view. To the best of our knowledge, Top2Ground is the first framework to demonstrate robust generalization across both wide-FOV (CVUSA, CVACT) and narrow-FOV (Auto Arborist Dataset) aerial imagery, within a