ReplicateAnyScene: Zero-Shot Video-to-3D Composition via Textual-Visual-Spatial Alignment
Mingyu Dong *, Chong Xia *, Mingyuan Jia, Weichen Lyu, Long Xu, Zheng Zhu, Yueqi Duan †, 1 Tsinghua University 2 Zhejiang University, Project Page: https://xiac20.github.io/ReplicateAnyScene/ · 2026
Read ReplicateAnyScene: Zero-Shot Video-to-3D Composition via Textual-Visual-Spatial Alignment by Mingyu Dong *, Chong Xia *, Mingyuan Jia, Weichen Lyu, Long Xu, Zheng Zhu, Yueqi Duan †, 1 Tsinghua University 2 Zhejiang University, Project Page: https://xiac20.github.io/ReplicateAnyScene/ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10789.txt.
Introduction
Dong and six colleagues propose ReplicateAnyScene, a tool turning casual videos into compositional 3D scenes without manual prompting or extra visual inputs. A five-stage cascade extracts and aligns generic priors from vision models across textual, visual, and spatial dimensions, grounding them into structured 3D scenes that stay coherent and physically plausible. To test the task more fully, they introduce the C3DR benchmark for reconstruction quality from diverse angles. Tests show the method beats existing baselines in generating high-quality compositional 3D scenes.
Catalog record
| CREATOR | Mingyu Dong *, Chong Xia *, Mingyuan Jia, Weichen Lyu, Long Xu, Zheng Zhu, Yueqi Duan †, 1 Tsinghua University 2 Zhejiang University, Project Page: https://xiac20.github.io/ReplicateAnyScene/ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36674 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10789
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReplicateAnyScene: Zero-Shot Video-to-3D Composition via Textual-Visual-Spatial Alignment Mingyu Dong *, Chong Xia *, Mingyuan Jia, Weichen Lyu, Long Xu, Zheng Zhu, Yueqi Duan †, 1 Tsinghua University 2 Zhejiang University, Project Page: https://xiac20.github.io/ReplicateAnyScene/ Abstract Humans exhibit an innate capacity to rapidly perceive and segment objects from video observations, and even mentally assemble them into structured 3D scenes. Replicating such capability, termed compositional 3D reconstruction, is pivotal for the advancement of Spatial Intelligence and Embodied AI. However, existing methods struggle to achieve practical deployment due to the insufficient integration of cross-modal information, leaving them dependent on manual object prompting, reliant on auxiliary visual inputs, and restricted to overly simplistic scenes by training biases. To address these limitations, we propose ReplicateAnyScene , a framework capable of fully automated and zero-shot transformation of casually captured videos into compositional 3D scenes. Specifically, our pipeline incorporates a five-stage cascade to extract and structurally align generic priors from vision foundation models across textual, visual, and spatial dimensions, grounding them into structured 3D representations and ensuring semantic coherence and physical plausibility of the constructed scenes. To facilitate a more comprehensive evaluation of this task, we further introduce the C3DR benchmark to assess reconstruction quality from diverse aspects. Extensive experiments demonstrate the superiority of our method over existing baselines in generating high-quality compositional 3D scenes. † † footnotetext: * Equal contribution. † Corresponding author. Figure 1 : We propose ReplicateAnyScene , a framework for fully automated and zero-shot compositional 3D reconstruction from casually captured videos. Our method extracts and aligns cross-modal priors from vision foundation models to generate semantically coherent and physically plausible 3D scenes. 1 Introduction Compositional 3D scene reconstruction, unlike holistic approaches like NeRF [ 36 ] and 3DGS [ 25 ] that prioritize overall visual fidelity, focuses on extracting complete object geometry and assembling them globally. This makes it inherently more suitable for interactive applications in Spatial Intelligence and Embodied AI. Early compositional 3D scene reconstruction methods [ 48 ] typically relied on time-consuming manual annotation and hand-crafted rules. Recently, the rapid development of advanced 3D reconstruction and generation techniques has catalyzed a new paradigm. Consequently, several modern approaches [ 70 , 37 , 66 , 62 ] have been proposed. These methods typically conceptualize compositional reconstruction as a sequential pipeline: reconstructing 3D scene layouts with semantics, generating independent 3D assets and composing the final scenes. However, existing methods struggle to achieve robust and automated deployment in real-world scenarios. Their specialized designs are brittle, manifesting in a heavy dependence on manual intervention, strict requirements for auxiliary depth or semantic inputs, a tendency to generate low-quality objects alongside physically implausible arrangements and a severe restriction to overly simplistic scenes stemming from inherent training data biases. Specifically, DPRecon [ 37 ] utilizes SDF [ 38 ] and SDS [ 41 ] for scene decomposition, and necessitates highly accurate depth priors, severely restricting its applicability to simplistic scenes. InstaScene [ 66 ] relies on semantic 3DGS [ 25 ] , which inherently limits its semantic richness. Moreover, lacking explicit relational constraints, it frequently produces object collisions and floating artifacts. Alternatively, MetaScenes [ 70 ] retrieves assets from complete point clouds, leading to poor consistency and requiring substantial manual annotation. Recently, SimRecon [ 62 ] employs view optimization and explicit scene graphs to enhance asset quality and physical plausibility. However, built upon semantic 3DGS, it faces similar bottlenecks, and its view optimization lacks robustness in complex scenarios. Driven by the rapid advancement of vision foundation models (VFMs), reliable solutions have emerged for specialized tasks across various dimensions, including VGGT [ 56 ] for geometric reconstruction, SAM3D [ 9 ] for 3D asset generation, and VLMs [ 3 , 34 , 19 ] for spatial relation reasoning. However, directly assembling these isolated models yields suboptimal results, because they operate independently without cross-dimensional awareness. Their naive integration leads to severe information fragmentation, manifesting as scale ambiguities, identity mismatches, and disjointed spatial layouts. To overcome these bottlenecks, we propose ReplicateAnyScene , a fully automated and zero-shot framework designed to synergize these isolated VFMs. Rather than treating them as simple plug-and-play modules, we progressively ground 2D semantic and perceptual priors into a coherent 3D physical space through uniquely crafted multimodal alignment mechanisms. Our pipeline is structured as a five-stage cascade, enabling the incremental fusion of rich semantic, perceptual, and geometric cues. Specifically, we begin with Progressive Object Discovery stage, employing a vision-language model (VLM) to scan video frames and extract a non-redundant set of open-vocabulary object categories. Guided by these semantics, the second stage, Spatial-Guided Visual Deduplication , resolves 2D tracking fragmentation by lifting masks into 3D and merging overlapping geometric fragments for consistent instance identities. With unique instances tracked, Optimal-View 3D Asset Generation stage selects the viewpoint maximizing visible surface area to guide the synthesis of high-fidelity 3D meshes. To accurately ground these assets, Iterative Visual-Spatial Alignment stage establishes dense correspondences between rendered and real views, refining object poses through an iterative render-match-optimize cycle. Finally, the last stage, Semantic-Aware Scene Refinement , utilizes VLM-inferred relationships to correct spatial anomalies, such as unnatural tilting or floating, assembling the objects into a physically plausible 3D environment. Furthermore, to address the lack of standardized evaluation protocols in this task, we introduce the C3DR benchmark to assess compositional reconstruction quality across diverse semantic, visual and geometric dimensions. Extensive experiments demonstrate that our method significantly outperforms existing baselines, setting a new standard for high-quality, fully automated compositional scene generation as shown in Fig. 1 . 2 Related Works 2.1 3D Indoor Datasets Rapid advancements in 3D computer vision have driven the creation of numerous indoor scene datasets [ 55 , 4 , 69 , 11 , 7 , 21 ] . Typically, these datasets provide scanned meshes from real scenes along with semantic and instance segmentation. However, their holistic representation inherently lacks compositional structure. This fundamental limitation hinders the further advancement of Embodied AI tasks that require object-level manipulation, such as robotic navigation [ 10 , 44 ] and interactive tasks [ 26 , 47 ] . To enable more structured and interactive tasks, recent efforts have turned to construct compositional indoor scene datasets. Existing approaches generally rely on human-intensive methods, ranging from manual assembly [ 42 , 17 , 48 ] to semi-automated pipelines [ 2 , 70 ] . These automated generation methods procedurally synthesize scenes using rule-based commonsense priors [ 14 , 39 , 15 ] or learned layout priors [ 49 , 65 ] . Unfortunately, these synthesized environments often suffer from oversimplified layouts and exhibit a significant domain gap from real-world distributions, acting merely as weak augmentations of existing 3D assets. Our approach, by contrast, enables the rapid and automatic generation of compositional indoor scenes directly from monocular video sequences. This eliminates the need for laborious manual labeling while preserving the richness and authenticity of real-world environments, showing great potential for large-scale data generation. 2.2 Image-Based 3D Asset Generation With the rapid advancement of diffusion models [ 31 , 46 , 20 ] and large-scale 3D asset datasets [ 16 , 13 , 8 , 12 ] , 3D generation has witnessed significant progress. Early image-conditioned approaches [ 64 , 58 , 50 , 33 , 32 , 23 , 51 ]