Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation
Sherwin Bahmani Tianchang Shen Jiawei Ren Jiahui Huang Yifeng Jiang, [2pt] Haithem Turki Andrea Tagliasacchi David B. Lindell Zan Gojcic, [2pt] Sanja Fidler Huan Ling Jun Gao Xuanchi Ren, [2pt] NVIDIA University of Toronto Vector Institute Simon Fraser University*equal contribution · 2025
Read Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation by Sherwin Bahmani Tianchang Shen Jiawei Ren Jiahui Huang Yifeng Jiang, [2pt] Haithem Turki Andrea Tagliasacchi David B. Lindell Zan Gojcic, [2pt] Sanja Fidler Huan Ling Jun Gao Xuanchi Ren, [2pt] NVIDIA University of Toronto Vector Institute Simon Fraser University*equal contribution (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19296.txt.
Introduction
Sherwin Bahmani and twelve co-authors teach a video generation model to build true 3D scenes, a system they call Lyra. Learning-based 3D reconstruction normally needs photos of real places from many angles, which are not always available. Video diffusion models, by contrast, have absorbed rich spatial knowledge from watching video, but they output flat frames. The team's self-distillation framework transfers that hidden 3D knowledge into an explicit 3D Gaussian Splatting representation.
A second decoder, trained to emit the 3D representation, is supervised by the ordinary image decoder's output. Training needs only synthetic data the video model generates itself. At run time the system creates 3D scenes from a text prompt or a single image, ready for real-time rendering. It also extends to dynamic scenes from one ordinary video. Experiments report state-of-the-art results for both still and moving scene generation.
Catalog record
| CREATOR | Sherwin Bahmani Tianchang Shen Jiawei Ren Jiahui Huang Yifeng Jiang, [2pt] Haithem Turki Andrea Tagliasacchi David B. Lindell Zan Gojcic, [2pt] Sanja Fidler Huan Ling Jun Gao Xuanchi Ren, [2pt] NVIDIA University of Toronto Vector Institute Simon Fraser University*equal contribution |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-19296
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Sherwin Bahmani Tianchang Shen Jiawei Ren Jiahui Huang Yifeng Jiang, [2pt] Haithem Turki Andrea Tagliasacchi David B. Lindell Zan Gojcic, [2pt] Sanja Fidler Huan Ling Jun Gao Xuanchi Ren, [2pt] NVIDIA University of Toronto Vector Institute Simon Fraser University*equal contribution Abstract The ability to generate virtual environments is crucial for applications ranging from gaming to physical AI domains such as robotics, autonomous driving, and industrial AI. Current learning-based 3D reconstruction methods rely on the availability of captured real-world multi-view data, which is not always readily available. Recent advancements in video diffusion models have shown remarkable imagination capabilities, yet their 2D nature limits the applications to simulation where a robot needs to navigate and interact with the environment. In this paper, we propose a self-distillation framework that aims to distill the implicit 3D knowledge in the video diffusion models into an explicit 3D Gaussian Splatting (3DGS) representation, eliminating the need for multi-view training data. Specifically, we augment the typical RGB decoder with a 3DGS decoder, which is supervised by the output of the RGB decoder. In this approach, the 3DGS decoder can be purely trained with synthetic data generated by video diffusion models. At inference time, our model can synthesize 3D scenes from either a text prompt or a single image for real-time rendering. Our framework further extends to dynamic 3D scene generation from a monocular input video. Experimental results show that our framework achieves state-of-the-art performance in static and dynamic 3D scene generation. https://research.nvidia.com/labs/toronto-ai/lyra Figure 1: Feed-Forward 3D and 4D Scene Generation. From a single image (top), Lyra infers a 3D Gaussian Splatting (3DGS) representation in a feed-forward fashion, through self-distilling a video diffusion model without requiring real-world multi-view data. With a video input (bottom), Lyra infers a dynamic 3DGS that offers interactive control in both time (rows) and viewpoint (columns). 1 Introduction Creating high-quality 3D environments at scale is a long-standing challenge in computer vision and graphics, empowering applications across film-making, VR/AR, and physical AI with closed-loop simulation. These applications require explicit 3D representations that support real-time rendering, physical interaction, and consistent multi-view synthesis. Recent advances in neural 3D reconstruction ( Mildenhall et al. 2020 ; Kerbl et al. 2023 ; Moenne-Loccoz et al. 2024 ) enable recovering such representations from posed images. However, the reliance on accurate camera poses and high-quality images significantly limits their scalability. The challenge is even greater for dynamic scenes, which typically require synchronized multi-camera setups ( Li et al. 2022 ) . Reconstruction methods are also inherently constrained to the observed content and cannot extrapolate beyond the input views. Recent works aim to mitigate these limitations by using feed-forward reconstruction models ( Zhang et al. 2024e ; Ren et al. 2024c ) , but these approaches face their own bottleneck: the scarcity of diverse, large-scale 3D training data, which leads to poor out-of-domain generalization. Video diffusion models ( Cosmos 2025 ; Wan 2025 ) offer a promising alternative. Trained on massive internet-scale video corpora, they achieve impressive fidelity and generalization across diverse environments. By learning from real-world videos with varied camera trajectories, ranging from handheld motion to cinematic panning and drone footage, these models implicitly encode cues about the underlying 3D world without requiring multi-view training data. Unlike reconstruction approaches ( Charatan et al. 2024 ) , generative models can also hallucinate plausible content beyond what is visible in the input frames. Yet, video diffusion models generate only 2D frames, lacking explicit 3D representations. This limits their use in simulation and downstream tasks that demand geometric consistency, long-term coherence, and physical interaction. Encouragingly, recent work ( Wang et al. 2024e ; Ren et al. 2025 ) has shown that video models can be adapted for explicit camera control using relatively small-scale posed datasets. This ability to generate posed image sequences transforms them into a powerful tool that can serve as both input and supervision for 3D reconstruction models. Motivated by this insight, we bridge these two paradigms, reconstruction and generation, through what we call generative 3D scene reconstruction . We introduce Lyra , a novel method for generating explicit 3D environments from the latent representations of video diffusion models in a single forward pass. Cruically, Lyra is trained in a self-distillation framework (Fig. 2 ), with a 3D Gaussian Splatting (3DGS) decoder operating directly in the latent space of a video diffusion model. Given a single input image or video, we sample a camera trajectory as conditioning input, denoise the resulting video latent, and decode it along two parallel branches. The first branch uses the standard RGB decoder to synthesize a video sequence, while the second is our 3DGS decoder, which produces an explicit 3D representation. Together, these branches form a self-distillation framework in which the RGB branch (teacher) supervises the 3DGS branch (student). To extend viewpoint coverage, we sample multiple camera trajectories, generate multiple video latents, and train the 3DGS decoder to fuse information across them while mitigating long-term and multi-view inconsistencies. This generative reconstruction framework provides three key benefits: (i) it enables generation of large-scale synthetic environments spanning diverse scenarios directly from video diffusion models, removing the need for real-world multi-view captures; (ii) by operating in latent space of a video model, it allows efficient processing of multiple views without the heavy memory overhead of pixel-space feed-forward reconstruction methods; and (iii) its explicit 3DGS output guarantees geometric consistency, providing representations that are directly applicable to downstream tasks such as physical simulation. Consequently, at inference time, Lyra generates high-quality 3D Gaussian scenes from monocular input that support real-time rendering, without requiring any additional optimization or post-processing . We further extend this self-distillation framework to dynamic 4D generation from monocular video. In this setting, the video model (teacher) provides space–time supervision, while the student learns to produce time-conditioned 3DGS representations that enable novel-view synthesis of dynamic scenes. Overall, our work makes the following contributions: • We introduce a self-distillation framework that trains a 3DGS student decoder using a pre-trained camera-controlled video diffusion model as the teacher, eliminating the need for captured multi-view real-world data. • Our framework extends to dynamic scenes to support 4D reconstruction from a monocular video. • Our model generalizes across diverse scenes, achieving state-of-the-art results in single-image 3D scene generation and single-video 4D scene generation. Figure 2: Self-distillation framework of Lyra . Previous work (left) ( Szymanowicz et al. 2025b ) trains multi-view reconstruction blocks using real-world datasets with limited diversity ( Zhou et al. 2018 ; Ling et al. 2024b ) . In contrast, we propose a self-distillation framework (right) for generative 3D scene reconstruction. Precisely, a pre-trained camera-controlled video diffusion model with its RGB decoder output (teacher) supervises the rendering of the 3DGS decoder (student). Using a video model pre-trained on diverse 2D video data allows us to provide diverse multi-view supervision. 2 Related work Multi-view image generation. Early works on multi-view image generation ( Watson et al. 2023 ; Liu et al. 2023 ; Shi et al. 2024 ; Wang & Shi 2023 ) mainly focus on object-centric scenes without background. Follow-up works ( Sargent et al. 2024 ; Tang et al. 2023b ; Schneider et al. 2025 ) extend multi-view image generators to the scene