VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting Generation with Flexible Pose and Multi-View Joint Modeling
Hyojun Go, Byeongjun Park, Hyelin Nam, Byung-Hoon Kim, Hyungjin Chung, Changick Kim · 2025
Read VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting Generation with Flexible Pose and Multi-View Joint Modeling by Hyojun Go, Byeongjun Park, Hyelin Nam, Byung-Hoon Kim, Hyungjin Chung, Changick Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15855.txt.
Introduction
Hyojun Go and five co-authors present a system that builds three-dimensional scenes from a line of text, in a 2025 paper. Their method, VideoRFSplat, retrains a video generation model so it directly outputs a point-cloud style scene made of soft three-dimensional blobs. Older ways of doing this often wobbled, since one model had to invent both camera views and camera positions at once.
The design splits the work into two streams, one for images and one for camera poses, which talk through shared blocks. Poses are also cleaned up faster than images during generation, so settled poses can guide the views and cut confusion between them. Trained on several large real-world video sets, the system beat earlier direct generators that lean on heavy after-the-fact refinement.
Catalog record
| CREATOR | Hyojun Go, Byeongjun Park, Hyelin Nam, Byung-Hoon Kim, Hyungjin Chung, Changick Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15855
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting Generation with Flexible Pose and Multi-View Joint Modeling Hyojun Go, Byeongjun Park, Hyelin Nam, Byung-Hoon Kim, Hyungjin Chung, Changick Kim Abstract We propose VideoRFSplat, a direct text-to-3D model leveraging a video generation model to generate realistic 3D Gaussian Splatting (3DGS) for unbounded real-world scenes. To generate diverse camera poses and unbounded spatial extent of real-world scenes, while ensuring generalization to arbitrary text prompts, previous methods fine-tune 2D generative models to jointly model camera poses and multi-view images. However, these methods suffer from instability when extending 2D generative models to joint modeling due to the modality gap, which necessitates additional models to stabilize training and inference. In this work, we propose an architecture and a sampling strategy to jointly model multi-view images and camera poses when fine-tuning a video generation model. Our core idea is a dual-stream architecture that attaches a dedicated pose generation model alongside a pre-trained video generation model via communication blocks, generating multi-view images and camera poses through separate streams. This design reduces interference between the pose and image modalities. Additionally, we propose an asynchronous sampling strategy that denoises camera poses faster than multi-view images, allowing rapidly denoised poses to condition multi-view generation, reducing mutual ambiguity and enhancing cross-modal consistency. Trained on multiple large-scale real-world datasets (RealEstate10K, MVImgNet, DL3DV-10K, ACID), VideoRFSplat outperforms existing text-to-3D direct generation methods that heavily depend on post-hoc refinement via score distillation sampling, achieving superior results without such refinement. 1 Introduction Generating realistic 3D scenes from text has garnered increasing attention in AR, gaming, and robotics. Early works [ 92 , 41 ] primarily relied on Neural Radiance Fields (NeRF) [ 50 , 54 , 60 ] to model 3D scenes, but their computationally intensive volumetric rendering poses limitations for real-time rendering. Recently, 3D Gaussian Splatting (3DGS) [ 31 ] has emerged as a promising alternative, enabling real-time rendering while preserving high-fidelity details. Consequently, recent 3D generation approaches [ 12 , 38 , 85 , 42 , 70 , 81 , 94 ] have increasingly adopted 3DGS for its efficiency and high-quality results. To generate 3DGS, several approaches [ 12 , 38 , 48 , 71 , 75 , 85 , 42 ] use Score Distillation Sampling (SDS) [ 60 ] with 2D diffusion models [ 63 ] , but this requires time-expensive per-scene optimization. Another line of work adopts pipeline-based methods, where generated images are reconstructed into 3DGS using separate reconstruction models [ 70 , 81 , 94 ] or multi-stage frameworks [ 86 , 15 , 65 , 87 , 98 ] . While pipeline-based methods can produce high-quality results, their reliance on sequential processing increases the risk of failure and adds unnecessary complexity [ 40 ] . To overcome these limitations, recent works have explored direct 3DGS generation [ 40 , 23 , 91 , 21 , 37 , 82 ] , which enables end-to-end inference, eliminating per-scene optimization and complex multi-stage pipelines. A key approach in this direction is fine-tuning 2D generative models [ 63 , 72 , 34 , 33 ] , which possess strong text-to-2D generation capabilities, to extend their generalizability to 3DGS from arbitrary texts. Despite the growing number of works targeting 3D generation, the majority focuses on object-level synthesis, leaving real-world scene generation comparatively underexplored. Object-level generation typically operates in controlled, predefined settings as seen in Objaverse datasets [ 17 , 16 ] , where objects are captured with fixed or simple camera trajectories. In contrast, real-world scenes present additional challenges due to their complex and diverse scene-specific camera motions, as well as their unbounded nature with diverse backgrounds and large-scale spatial variations [ 37 , 21 ] . These pose fundamental challenges to developing generative models for direct 3DGS generation, introducing difficulties distinct from object-level generation. To address these challenges, previous works [ 21 , 37 ] have adopted generative models that learn the joint distribution of multi-view images and camera trajectories. By incorporating the camera pose into the generative formulation, the model (1) implicitly infers the scene’s pose from the text prompt, allowing it to handle diverse and complex camera poses without requiring user inputs. (2) The model generates consistent multi-view images aligned with these inferred scene-specific viewpoints, effectively capturing unbounded environments with diverse backgrounds and spatial variations. However, prior works [ 21 , 36 , 37 ] have suffered from instability in extending 2D generative models to joint modeling due to the modality gap, hindering high-quality generation and alignment between generated images and camera poses. To address this, several stabilization techniques leveraging external models have been proposed. For instance, NVComposer [ 36 ] leverages Dust3R [ 77 ] distillation to improve consistency, while SplatFlow [ 21 ] and director3D [ 37 ] rely on other 2D models and refinements during sampling. While these help reduce instability, dependency on external models hinders seamless integration into a single model. In this paper, to eliminate external dependency, we present VideoRFSplat , a direct 3DGS generation model that introduces an architecture and sampling strategy for jointly generating camera poses and multi-view images when leveraging a video generation model. Our core idea is a dual-stream architecture, side-attaching a dedicated rectified flow-based pose generation model alongside a pre-trained video generation model, jointly trained to generate multi-view image latents and camera poses simultaneously. This side-attached pose diffusion model runs parallel to the video model’s forward stream, interacting at specific layers while maintaining separate forward paths. This design minimizes interference between the two modalities, allowing each to specialize independently and ensuring consistency between poses and multi-view images. Similar to previous approaches [ 82 , 21 , 37 ] , a Gaussian splat decoder decodes 3DGS from the generated poses and image latents in a feed-forward fashion. Then, we decouple the timesteps of the pose and multi-view generation models, allowing each to operate at different noise levels independently. Unlike standard approaches that synchronize timesteps and noise levels across modalities, our design permits flexible asynchronous sampling. This approach is motivated by our observation that synchronized denoising of multi-view images and camera poses, particularly at early timesteps, leads to mutual ambiguity, increasing uncertainty and causing unstable generation. To mitigate this issue, we design the pose modality—found to be more robust to faster denoising—to undergo a more rapid denoising process than the images. By doing so, the clearer pose information effectively reduces the ambiguity in the pose modality, stabilizing the sampling. Furthermore, we propose an asynchronous adaptation of Classifier-Free Guidance (CFG) that enables the clearer pose to better guide multi-view image generation. Moreover, the proposed asynchronous sampling strategy with decoupled timesteps naturally extends to the camera conditional generation task. We train VideoRFSplat on RealEstate10K [ 99 ] , MVImgNet [ 90 ] , DL3DV-10K [ 43 ] , and ACID [ 45 ] datasets. Notably, VideoRFSplat achieves superior performance without relying on SDS++ refinement [ 37 ] , surpassing existing text-to-3D direct generation methods that depend on SDS++ [ 37 , 21 ] , demonstrating effectiveness of our approach and eliminating dependencies on external models. 2 Related Works 2.1 Diffusion Models and Rectified Flow Diffusion models [ 28 , 67 , 68 ] have recently achieved state-of-the-art performance in high-fidelity generation across diverse modalities, including images [ 18 ,