Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagery

Jie-Ying Lee, Yi-Ruei Liu, Shr-Ruei Tsai, Wei-Cheng Chang, Chung-Ho Wu, Jiewen Chan, Zhenjun Zhao, Chieh Hubert Lin, Yu-Lun Liu · 2025

Read Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagery by Jie-Ying Lee, Yi-Ruei Liu, Shr-Ruei Tsai, Wei-Cheng Chang, Chung-Ho Wu, Jiewen Chan, Zhenjun Zhao, Chieh Hubert Lin, Yu-Lun Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-15869.txt.

Introduction

Jie-Ying Lee leads eight co-authors in building city scenes in three dimensions from satellite pictures alone. Large, exact city models serve immersive and embodied uses. Yet real three-dimensional scans at that scale are scarce and costly. Their framework, Skyfall-GS, takes a new supply route for the raw material.

Satellite work supplies realistic coarse shape. Open diffusion models then add fine, close-up looks. A stepwise loop refines the scene in stages. Shape gets more complete and texture more photo-like at each stage. No costly three-dimensional labels are needed. The result supports real-time immersive exploration at city-block scale. In tests, Skyfall-GS gives shape that agrees better across views than leading prior methods. Its textures also look more real, the team reports.

Catalog record

CREATORJie-Ying Lee, Yi-Ruei Liu, Shr-Ruei Tsai, Wei-Cheng Chang, Chung-Ho Wu, Jiewen Chan, Zhenjun Zhao, Chieh Hubert Lin, Yu-Lun Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-15869

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagery Jie-Ying Lee, Yi-Ruei Liu, Shr-Ruei Tsai, Wei-Cheng Chang, Chung-Ho Wu, Jiewen Chan, Zhenjun Zhao, Chieh Hubert Lin, Yu-Lun Liu Abstract Synthesizing large-scale, explorable, and geometrically accurate 3D urban scenes is a challenging yet valuable task in providing immersive and embodied applications. The challenges lie in the lack of large-scale and high-quality real-world 3D scans for training generalizable generative models. In this paper, we take an alternative route to create large-scale 3D scenes by synergizing the readily available satellite imagery that supplies realistic coarse geometry and the open-domain diffusion model for creating high-quality close-up appearances. We propose Skyfall-GS , the first city-block scale 3D scene creation framework without costly 3D annotations, also featuring real-time, immersive 3D exploration. We tailor a curriculum-driven iterative refinement strategy to progressively enhance geometric completeness and photorealistic textures. Extensive experiments demonstrate that Skyfall-GS provides improved cross-view consistent geometry and more realistic textures compared to state-of-the-art approaches. Project page: https://skyfall-gs.jayinnn.dev/ . Figure 1: Our method synthesizes high-quality, immersive 3D urban scenes solely from multi-view satellite imagery, enabling realistic drone-view navigation without relying on additional 3D or street-level training data. Given multiple satellite images from diverse viewpoints and dates ( left ), our method leverages 3D Gaussian Splatting combined with pre-trained text-to-image diffusion models in an iterative refinement framework to generate realistic 3D block-scale city from limited satellite-view input ( right ). Our method significantly enhances visual fidelity, geometric sharpness, and semantic consistency, enabling real-time immersive exploration. 1 Introduction Synthetic high-quality, immersive, and semantically plausible 3D urban scenes have crucial applications in gaming, filmmaking, and robotics. The ability to create a large-scale and 3D-grounded environment supports realistic rendering and immersive experience for storytelling, demonstration, and embodied physics simulation. However, due to limited 3D-informed data, building a generative model for realistic and navigable 3D cities remains challenging. It is expensive and labor-intensive to acquire large-scale 3D and textured reconstructions of cities with detailed geometry, while using Internet image collections faces challenges in camera pose registration and excessive data noise (e.g., transient objects and different times of the day). These constraints set back existing 3D city generation frameworks from creating realistic and diverse appearances. With this observation, we propose an alternative route for virtual city creation with a two-stage pipeline: partial and coarse geometry reconstruction from multi-view satellite imagery, then close-up appearance completion and hallucination using an open-domain diffusion model. Satellite imagery offers a compelling alternative due to its extensive geographic coverage, automated collection, and high-resolution capabilities. For instance, Maxar’s WorldView-3 satellite captures approximately 680,000 \text{km}^{\text{2}} of imagery daily at resolutions up to 31 cm per pixel. Such data inherently encodes semantically plausible representations of real-world environments, enabling scalable 3D urban scene creation. However, in Figure 2 (a), we show that directly applying 3D reconstruction methods to satellite imagery is insufficient for creating navigable and immersive 3D cities. The significant amount of invisible regions (e.g., building facades) and limited satellite-view parallax create incorrect geometry and artifacts. Completing and enhancing the geometry and texture in the ground view requires a significant influx of extra information. In Figure 2 (b), we study a few state-of-the-art methods in city generation ( Xie et al. 2024 ; Xie et al. 2025b ) . These methods produce oversimplified building geometries and unrealistic appearances due to strong assumptions, particularly the reliance on semantic maps and height fields as the sole inputs, and overfitting to small-scale, domain-specific datasets. Such an observation motivates us to seek help from open-domain foundation vision models as an external information source, which provides better zero-shot generalization and diversity. Noticing that the ground-view novel-view renderings from the GS reconstructed scene exhibit noise-like patterns, we treat these renderings as intermediate results in a denoising diffusion process. Then, we complete the remaining denoising process to create hallucinated pseudo ground-truth for the GS scene optimization. To stabilize the convergence, we carefully design a curriculum-based view selection and iterative refinement process, where the sampled view angles gradually fall from the sky to the ground over time. Accordingly, we name our framework Skyfall-GS . In Figure 1 and Figure 2 , we show that Skyfall-GS yields significantly enhanced texture with 3D-justified geometry compared to the relevant baselines. Skyfall-GS is the first method synthesizing immersive, free-flight navigable 3D urban scenes without fixed-domain training on 3D data. Skyfall-GS operates on readily available satellite imagery as the only input, then hallucinates realistic aerial-view appearances and maintains a strong satellite-to-ground 3D consistency. Moreover, Skyfall-GS supports real-time and interactive rendering, as we design our framework to produce GS results without sophisticated data structures. Through experiments on diverse environments, we show that Skyfall-GS has better generalization and robustness compared to state-of-the-art methods. Our ablation shows that each of our designs improves the perceptual plausibility and semantic consistency. Skyfall-GS paves the way for scalable 3D urban virtual scene creation, enabling applications in virtual entertainment, simulation, and robotics. In summary, our contributions include: • We introduce Skyfall-GS, the first method to synthesize immersive, real-time free-flight navigable 3D urban scenes solely from multi-view satellite imagery using generative refinement. • An open-domain refinement approach leveraging pre-trained text-to-image diffusion models without domain-specific training. • A curriculum-learning-based iterative refinement strategy progressively enhances reconstruction quality from higher to lower viewpoints, significantly improving visual fidelity in occluded areas. Figure 2: Limitations of existing novel-view synthesis methods from satellite imagery. (a) Sat-NeRF ( Marí et al. 2022 ) and naive 3DGS ( Kerbl et al. 2023 ) yield blurred or distorted building facades due to insufficient geometric detail and limited parallax from satellite viewpoints. (b) City generation methods ( Xie et al. 2024 ; Xie et al. 2025b ) produce oversimplified building geometries and unrealistic appearances, primarily due to strong assumptions about the input data, and overfitting to small-scale, domain-specific datasets. In comparison, our method synthesizes more realistic appearances and geometries from aerial views. 2 Related Work Gaussian Splatting. 3D Gaussian Splatting (3DGS) ( Kerbl et al. 2023 ) offers real-time view synthesis rivaling NeRFs ( Mildenhall et al. 2021 ; Barron et al. 2021 ; Barron et al. 2022 ; Müller et al. 2022 ; Barron et al. 2023 ; Martin-Brualla et al. 2021 ) . Mip-Splatting ( Yu et al. 2024 ) fixes scale-change issues via on-the-fly resizing. Recent advances target satellite and aerial reconstruction: FusionRF ( Sprintson et al. 2024 ) achieves 17% depth improvement from multispectral acquisitions, while InstantSplat ( Fan et al. 2024 ) enables 40-second pose-free reconstruction. “In-the-wild” variants handle appearance and uncertainty ( Xu et al. 2024 ; Sabour et al. 2024 ; Wang et al. 2024b ; Dahmani et al. 2024 ; Zhang et al. 2024a ; Kulhanek et al. 2024 ; Hou et al. 2025 ) , including SpectroMotion ( Fan et al. 2025 ) for dynamic specular scenes, while large-scene methods use LOD and partitioning ( Kerbl et al. 2024 ; Liu et al. 2025c ; Liu et al. 2024 ; Lin et al. 2024 ; Turki et al. 2022 ; Tancik et al. 2022 ) . CAT-3DGS ( Zhan et al. 2025 ) achieves rate-distortion optimization via context-adaptive triplanes. For sparse-view satellite imagery, depth or co-regularization priors guide reconstruction ( Li et al. 2024b ; Zhang et al. 2024b ; Zhu et al. 2023

Read the complete text