Constructing a 3D Town from a Single Image

Kaizhi Zheng, Ruijian Zhang, Jing Gu, Jie Yang, Xin Eric Wang · 2025

Read Constructing a 3D Town from a Single Image by Kaizhi Zheng, Ruijian Zhang, Jing Gu, Jie Yang, Xin Eric Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15765.txt.

Introduction

Zheng and co-authors presented this method in twenty twenty-five for building a three-dimensional scene from one photo. Making detailed 3D scenes usually takes costly gear, many views, or long hours of hand modeling. Their system, SceneFuse-3D, starts from a single top-down image. It needs no extra training on 3D data.

The method cuts the image into overlapping regions. A ready-made object generator builds each region in 3D. A filling step then repairs gaps so the pieces join into one smooth whole. This design beats the blur and warped layouts of earlier scene generators. In tests it outscored four rival systems on shape quality, layout sense, and surface detail. The work shows that one photo can yield a tidy full scene.

Catalog record

CREATORKaizhi Zheng, Ruijian Zhang, Jing Gu, Jie Yang, Xin Eric Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15765

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Constructing a 3D Town from a Single Image Kaizhi Zheng, Ruijian Zhang, Jing Gu, Jie Yang, Xin Eric Wang Abstract Acquiring detailed 3D scenes typically demands costly equipment, multi-view data, or labor-intensive modeling. Therefore, a lightweight alternative, generating complex 3D scenes from a single top-down image, plays an essential role in real-world applications. While recent 3D generative models have achieved remarkable results at the object level, their extension to full-scene generation often leads to inconsistent geometry, layout hallucinations, and low-quality meshes. In this work, we introduce 3DTown , a training-free framework designed to synthesize realistic and coherent 3D scenes from a single top-down view. Our method is grounded in two principles: region-based generation to improve image-to-3D alignment and resolution, and spatial-aware 3D inpainting to ensure global scene coherence and high-quality geometry generation. Specifically, we decompose the input image into overlapping regions and generate each using a pretrained 3D object generator, followed by a masked rectified flow inpainting process that fills in missing geometry while maintaining structural continuity. This modular design allows us to overcome resolution bottlenecks and preserve spatial structure without requiring 3D supervision or fine-tuning. Extensive experiments across diverse scenes show that 3DTown outperforms state-of-the-art baselines, including Trellis, Hunyuan3D-2, and TripoSG, in terms of geometry quality, spatial coherence, and texture fidelity. Our results demonstrate that high-quality 3D town generation is achievable from a single image using a principled, training-free approach. Project website: https://eric-ai-lab.github.io/3dtown.github.io/ Figure 1: 3D Scene Generation from a Single Image. Given a top-down reference image (center), 3DTown generates coherent and realistic 3D scenes that preserves geometry, texture, and layout compared to other state-of-the-art image-to-3D generation models. Our method also generalizes across diverse styles (right), producing high-quality outputs without any 3D training. 1 Introduction Realistic 3D towns and structured environments serve as essential assets in simulation, robotics, digital content creation, and virtual world building. They enable scalable training for autonomous agents, immersive game environments, and rapid digital twin construction. However, constructing detailed and coherent 3D scenes typically requires either expensive 3D scanning equipment, multi-view data collection, or labor-intensive modeling. In contrast, generating 3D scenes from a single top-down image offers a lightweight and accessible alternative, making it possible to bootstrap realistic environments from minimal input. Despite its appeal, generating a coherent and realistic 3D scene from a single image presents several fundamental challenges. First, the synthesized scene must exhibit consistent geometry across novel views, which is difficult for pure volumetric rendering methods such as Neural Radiance Fields (NeRF) [ 35 ] or 3D Gaussian Splatting (3DGS) [ 18 ] . While these techniques excel at photorealistic appearance modeling, they often suffer from geometry artifacts, especially in occluded or sparsely visible regions, leading to multiview inconsistencies and structural implausibility [ 25 , 5 , 66 , 64 ] . Second, the global layout of the generated scene must remain faithful to the input image. This is particularly difficult when considering the whole scene as one asset and using image-to-3D asset generators [ 59 , 52 , 24 ] , which fail to preserve spatial relationships between elements, resulting in distorted or semantically misaligned arrangements. Third, the local fidelity of individual objects should align closely with the visual evidence in the input. Due to the resolution constraints of 3D representations and the domain shift from object-level training to scene-level inference, previous image-to-3D generators are prone to creating low-quality meshes and misaligning the texture. To address these challenges, we propose 3DTown , a training-free framework for generating complex 3D scenes from a single top-down image, by enhancing the capability of image-to-3D object generators. Our method combines two core components: region-based generation and spatial-aware 3D inpainting. Each targets specific challenges in existing pipelines. We divide the scene into overlapping regions and synthesize each independently. This modular approach enables spatial upscaling and improves local alignment by grounding generation on localized image crops. To maintain global coherence and object continuity across regions, we estimate a coarse 3D structure from monocular depth and landmark detection, forming a spatial prior. A masked rectified flow mechanism then completes missing parts while preserving known content, enhancing structural consistency and object-level fidelity throughout the scene. Through extensive experiments, we demonstrate that 3DTown generates realistic, diverse, and geometrically consistent 3D scenes from a single top-down image. Our method significantly outperforms strong baselines, including Trellis [ 59 ] , Hunyuan3D-2 [ 52 ] , and TripoSG [ 24 ] , across both human preference and GPT-based evaluations. Quantitative results show notable gains in geometry quality, layout coherence, and texture fidelity, while qualitative comparisons highlight 3DTown’s ability to preserve spatial structure and fine-grained detail. These results underscore the effectiveness of our modular, training-free approach to 3D scene synthesis. Our main contributions are summarized as follows: • We propose 3DTown , a training-free framework for generating structured 3D scenes from a single top-down image, leveraging pretrained object-centric generators for zero-shot scene asset synthesis. • We develop a modular generation strategy that combines region-wise latent synthesis with spatial-aware 3D inpainting, effectively addressing resolution bottlenecks, image-geometry misalignment, and inter-region inconsistency. • We conduct comprehensive evaluations on diverse scenes and show that 3DTown outperforms state-of-the-art baselines (Trellis, Hunyuan3D-2, and TripoSG) in geometry quality, layout coherence, and texture realism under both human and GPT-4o-based assessments. 2 Related Work 3D Scene Generation with 2D Generative Models Recent progress in 2D generative modeling [ 42 , 13 , 4 , 39 ] has inspired a growing body of work that combines image synthesis with external 3D reconstruction techniques to generate 3D scenes. A common strategy is to use pretrained image generation models to outpaint a given input view according to known or sampled camera trajectories, followed by reconstruction via depth fusion or volumetric representations such as Neural Radiance Fields (NeRF) [ 35 ] and 3D Gaussian Splatting (3DGS) [ 18 ] . Early approaches primarily targeted indoor scenes [ 56 , 20 , 14 , 19 ] , while more recent efforts have extended to arbitrary natural scenes [ 25 , 1 , 10 , 5 ] , achieving improved reconstruction quality through advanced depth fusion pipelines [ 64 , 68 , 66 , 62 , 45 , 8 , 63 ] . Despite producing visually compelling results, these methods often suffer from geometric inconsistencies caused by hallucinations during image generation, particularly in occluded or ambiguous regions. To mitigate this, recent approaches explore panoramic [ 46 , 23 , 57 , 44 , 26 ] and multiview generation [ 29 , 51 , 11 ] to improve spatial coverage and structure. However, these approaches typically restrict the freedom of movement or require the camera motion for close views, and still struggle with accurate 3D geometry synthesis. The concurrent work Syncity [ 7 ] proposes block-wise generation via a 3D generator and assembles them to scenes, but all generated scenes have compact layouts and cannot handle the entire scene images. In contrast, we aim to generate geometry-consistent 3D assets with arbitrary layout directly from a single top-down image. 3D Scene Generative Model Beyond 2D-to-3D approaches, recent research has explored the direct generation of 3D scenes using native 3D representations. One prominent line of work leverages large language models [ 9 , 69 , 2 , 15 , 47 ] or

Read the complete text