HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation

Wenqi Dong 1,2 Bangbang Yang 2 Zesong Yang 1,2 Yuan Li 1 Tao Hu 2 Hujun Bao 1 Yuewen Ma 2 Zhaopeng Cui 1 1 Zhejiang University 2 ByteDance · 2025

Read HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation by Wenqi Dong 1,2 Bangbang Yang 2 Zesong Yang 1,2 Yuan Li 1 Tao Hu 2 Hujun Bao 1 Yuewen Ma 2 Zhaopeng Cui 1 1 Zhejiang University 2 ByteDance (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13072.txt.

Introduction

Generating a whole room in 3D that users can rearrange is harder than generating one object, since scenes tangle many items with occlusions and shadows. Wenqi Dong and co-authors present HiScene in this 2025 paper, a framework that treats a scene as a hierarchy: the room is one big object, and it breaks down into movable items. The bridge is an isometric view, drawn like a 2D image yet structured like the 3D scene.

A video diffusion step fills in the parts of objects hidden behind others, and shape priors keep placements coherent. The authors report more natural arrangements and complete, editable objects than rival methods, with physical plausibility kept intact. The target is interactive media and games.

Catalog record

CREATORWenqi Dong 1,2 Bangbang Yang 2 Zesong Yang 1,2 Yuan Li 1 Tao Hu 2 Hujun Bao 1 Yuewen Ma 2 Zhaopeng Cui 1 1 Zhejiang University 2 ByteDance
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13072

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation Wenqi Dong 1,2 Bangbang Yang 2 Zesong Yang 1,2 Yuan Li 1 Tao Hu 2 Hujun Bao 1 Yuewen Ma 2 Zhaopeng Cui 1 1 Zhejiang University 2 ByteDance Abstract Scene-level 3D generation represents a critical frontier in multimedia and computer graphics, yet existing approaches either suffer from limited object categories or lack editing flexibility for interactive applications. In this paper, we present HiScene, a novel hierarchical framework that bridges the gap between 2D image generation and 3D object generation and delivers high-fidelity scenes with compositional identities and aesthetic scene content. Our key insight is treating scenes as hierarchical ”objects” under isometric views, where a room functions as a complex object that can be further decomposed into manipulatable items. This hierarchical approach enables us to generate 3D content that aligns with 2D representations while maintaining compositional structure. To ensure completeness and spatial alignment of each decomposed instance, we develop a video-diffusion-based amodal completion technique that effectively handles occlusions and shadows between objects, and introduce shape prior injection to ensure spatial coherence within the scene. Experimental results demonstrate that our method produces more natural object arrangements and complete object instances suitable for interactive applications, while maintaining physical plausibility and alignment with user inputs. More details at https://zju3dv.github.io/hiscene/ . Figure 1 : HiScene allows users to generate scene-level 3D assets with natural layout and appealing looking, while delivering compositional items for versatile applications such as interactive editing and simulation. 1 Introduction Recently, we have witnessed remarkable breakthroughs of generative techniques in both 2D and 3D content creation, which empowers users to create stunning images and intricate 3D objects with simple text prompts. However, extending such generation capabilities to scene-level 3D content—for instance, generating complete 3D room models with coherent geometry and appearance guided by user-provided text or images—remains a significant challenge. Recent approaches typically rely on large language models (LLMs) with handcrafted rules to generate 3D scene layouts and place objects accordingly, yet these results often lack realism (e.g., exhibiting constrained object diversity and simplistic arrangements) due to LLMs’ limited spatial understanding capabilities [ 66 ] . Others attempt to lift 2D images into 3D scenes using depth-based mesh deformation [ 16 , 13 ] . While these approaches offer category-free generation capabilities, they typically produce scenes as inseparable wholes, limiting interactive applications such as scene editing, object manipulation, and data curation for robotic semantic understanding. Based on these observations, we believe an ideal scene generation method should have the following properties: 1) Realistic layout & assets : it should generate scenes with natural object arrangements and diverse content beyond just a few simple objects from limited categories. The scenes should reflect real-world spatial relationships and object interactions. 2) Compositional & complete instances : for interactive applications, each object in the scene should be a complete, intact 3D entity that can be individually manipulated, edited, or replaced without disrupting the entire scene. 3) Spatial alignment & plausibility : the generated scene should faithfully represent the user’s text or image prompt while maintaining physical coherence and plausibility in the 3D space (e.g., without distorted shapes, unrealistic proportions, or floating placement). In this paper, we propose a novel hierarchical scene generation framework, named HiScene. Rather than determining how scenes are built in 3D space with handcrafted rules, we leverage the complementary knowledge embedded in image generation models about how scenes should appear with aesthetic appeal and reasonable layout, and instantiate the concrete 3D representation that aligns with the image in a top-down manner. Our key insight is that we can treat a scene as hierarchical level “objects” under the isometric view . From the generator’s perspective, a room can be seen as a complex object itself, while each individual item within the room can also be separately generated and manipulated. By leveraging this hierarchical approach, HiScene bridges the gap between object-level and scene-level generation, producing complete scenes that benefit from pre-trained object-centric generation priors while maintaining compositional structure. While the hierarchical scene generation approach is technically plausible, we identify several challenges in creating high-fidelity and compositional 3D scenes that we address in this paper. Hierarchical Scene Parsing. To bootstrap the hierarchical scene generation, we first initialize the entire scene with a pre-trained 3D generation model [ 70 ] from the given isometric view image. Once obtaining the complete 3D Gaussian representation [ 23 ] of the scene, a key challenge is to accurately isolate individual objects from the scene structure. To address this, we implement a hierarchical scene parsing approach based on “analysis by synthesis” to identify distinct objects and prepare them for subsequent generative refinement. Specifically, we render multi-view images and perform 3D identity segmentation using 2D segmentation priors enhanced with contrastive learning techniques [ 78 ] . Then, for each identified object, we render object-centric circular views and carefully identify occlusion regions, enabling us to understand the spatial relationships between objects and more effectively reconstruct complete object identities in the following steps. Instance Refinement with Video-diffusion-based Amodal Completion. A key challenge during identity refinement is that the rendered instance views often exhibit significant occlusions. Despite advances in 3D object generation, reconstructing complete objects from occluded views remains ill-posed, while directly applying standard inpainting methods often produces implausible results due to limited object understanding (see Fig. 7 ). Moreover, the target instance might also include ambient shadow caused by the foreground occluder, which cannot be addressed with conventional inpainting frameworks. To tackle this problem, we reformulate the instance refinement as a 2D amodal completion and 3D regeneration task, and propose a novel video-diffusion-based completion framework to handle it. Specifically, our approach treats the amodal completion process as a temporal transition video effect, where occlusions gradually dissolve to reveal the complete object. To enable this capability, we construct a specialized dataset for training video diffusion model to perform such completion transitions. The temporal nature of our video-based completion effectively handles challenging cases including occlusion shadow removal, outperforming static image-based inpainting or completion methods by preserving structural coherence and producing more plausible results even in complex scenarios. Spatial Aligned Generation. Even with advanced 3D generation models to refine each segmented identity, ensuring spatial alignment between the refined object and its original placement is non-trivial due to the unposed nature of compressed latent [ 70 ] . As a result, naïvely applying refinement with amodally completed object views might produce objects with variant shapes, making them incompatible with the original scene layout. To address this issue, we propose a shape prior injection mechanism that conditions the refinement stage of each identity. Specifically, we first extract a geometric shape prior from a view-aligned generation method [ 72 ] , and use this aligned shape prior as the latent initialization for our refinement pipeline rather than starting from random noise. This approach significantly reduces geometric ambiguity during refinement and ensures proper spatial alignment between the generated objects and the original scene context. The contribution of the paper can be summarized as follows. 1) We propose HiScene, a novel scene-level generation method that produces high-fidelity 3D scenes with compositional identities, natural scene arrangement and diverse content. HiScene bridges the gap between 2D and 3D generation by leveraging isometric views, enabling

Read the complete text