Novel View Synthesis as Video Completion
Qi Wu, Khiem Vuong, Minsik Jeon, Srinivasa Narasimhan, Deva Ramanan · 2026
Read Novel View Synthesis as Video Completion by Qi Wu, Khiem Vuong, Minsik Jeon, Srinivasa Narasimhan, Deva Ramanan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08500.txt.
Introduction
Wu and four colleagues recast sparse novel view synthesis, predicting a scene from a new camera angle, as low frame rate video completion. Their FrameCrafter adapts video models, which already hold implicit multi view knowledge, by adding per frame latent codes and dropping time position codes so input order stops mattering. With about five input views, video models trained to forget time rival dedicated methods on sparse view benchmarks.
Catalog record
| CREATOR | Qi Wu, Khiem Vuong, Minsik Jeon, Srinivasa Narasimhan, Deva Ramanan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46834 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08500
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Novel View Synthesis as Video Completion Qi Wu, Khiem Vuong, Minsik Jeon, Srinivasa Narasimhan, Deva Ramanan Abstract We tackle the problem of sparse novel view synthesis (NVS) using video diffusion models; given K K ( ≈ 5 \approx 5 ) multi-view images of a scene and their camera poses, we predict the view from a target camera pose. Many prior approaches leverage generative image priors encoded via diffusion models. However, models trained on single images lack multi-view knowledge. We instead argue that video models already contain implicit multi-view knowledge and so should be easier to adapt for NVS. Our key insight is to formulate sparse NVS as a low frame-rate video completion task. However, one challenge is that sparse NVS is defined over an unordered set of inputs, often too sparse to admit a meaningful order, so the models should be invariant to permutations of that input set. To this end, we present FrameCrafter, which adapts video models (naturally trained with coherent frame orderings) to permutation-invariant NVS through several architectural modifications, including per-frame latent encodings and removal of temporal positional embeddings. Our results suggest that video models can be easily trained to “forget” about time with minimal supervision, producing competitive performance on sparse-view NVS benchmarks. Figure 1 : Video Diffusion as Multi-view Prior. While prior generative NVS methods typically initialize from image diffusion models and rely on large pose-annotated multi-view datasets, videos naturally capture viewpoint changes and cross-view consistency, making them a more scalable source of supervision for NVS. We present FrameCrafter , which adapts pretrained video diffusion models with lightweight modifications using LoRA on only 1K DL3DV scenes, achieving competitive performance and demonstrating a more data-efficient path to high-fidelity novel view synthesis. 1 Introduction Video diffusion models have advanced rapidly in recent years [ 4 , 39 , 46 , 19 , 3 , 6 ] , demonstrating remarkable capabilities in high-fidelity video generation and strong spatio-temporal consistency. Beyond visual realism, these models exhibit increasingly sophisticated world understanding, suggesting an implicit awareness of 3D structure and viewpoint transformation [ 14 ] . Motivated by this geometric consistency, prior works have attempted to repurpose video diffusion models for 3D/4D reconstruction [ 7 , 16 ] and view-conditioned generation [ 31 , 1 , 57 , 45 ] . However, their potential for sparse-view novel view synthesis (NVS) — particularly under large viewpoint changes — remains largely underexplored. Sparse-view novel view synthesis requires a model to understand multi-view relationships and infer geometry that is only partially-observed. Regression methods [ 26 , 18 ] learn deterministic mappings under geometric constraints from input views, but often struggle when geometry is sparsely observed or unseen. Image diffusion [ 24 , 10 , 55 , 20 ] , in contrast, benefits from large-scale generative pretraining and is better equipped to synthesize plausible content under large viewpoint changes. However, videos, as another scalable source to train diffusion models, are inherently more aligned with the NVS objective: they naturally capture multi-view consistency and viewpoint transitions over time. Despite this, most prior diffusion-based methods for sparse-view NVS rely on image diffusion backbones and require substantial multi-view supervision to induce 3D awareness. Our key thesis is as follows: video diffusion models provide a stronger prior for sparse-view NVS than image diffusion models. We leverage geometric reasoning that has already emerged from large-scale video pretraining, instead of learning multi-view geometry “from scratch” on multi-view training data. While perhaps obvious in retrospect, our approach is nontrivial to operationalize because one must modify a video model to accept unordered collections of views (and cameras) as inputs. To do so, we introduce FrameCrafter with three key architectural modifications. The first and most important is to modify the video VAE to encode a K -frame video as a collection of K independent 1-frame videos, allowing multi-view inputs to be treated as an unordered set. This is actually a small architectural change, as current image-to-video diffusion models already treat the input image as a 1-frame video during encoding. Secondly, many NVS solutions break permutation invariance by defining camera poses in a world coordinate system aligned to the first input image. Instead, we define world coordinates aligned to the novel query view (to be synthesized). In practice, we find that video diffusion models can be easily taught to condition on cameras by channel-concatenating latent (1-frame) video encodings with Plücker raymaps. Thirdly, we remove the temporal component of 3D Rotary Positional Embeddings (RoPE) [ 37 ] so that the model cannot rely on frame indices when processing input views. We demonstrate that our lightweight architectural modifications can turn state-of-the-art video models into state-of-the-art multi-view models that are invariant to permutations of their input. Our results show that, with only a small number of (LoRA [ 13 ] ) trainable parameters and minimal training data (1K scenes from DL3DV [ 22 ] ), we can finetune video diffusion models to outperform prior image diffusion-based models. Moreover, we observe a consistent scaling trend: as video diffusion backbones advance, their improvements directly translate into stronger novel view synthesis performance. Our framework can be seamlessly adapted to different video diffusion models, and progress in large-scale video foundation models can be transferred to NVS without requiring massive curated multi-view datasets. 2 Related Work Novel View Synthesis. Early NVS methods are primarily regression-based, learning radiance fields or renderable scene representations from multi-view inputs using frameworks such as NeRF [ 26 ] and its variants [ 47 , 41 , 5 , 2 ] , as well as explicit representations such as 3D Gaussian Splatting [ 18 ] . Subsequent works improve robustness via regularization [ 27 ] or feed-forward scene prediction [ 50 , 17 , 15 , 52 ] . More recently, diffusion-based approaches have emerged as a powerful alternative by leveraging the geometric and appearance priors of pretrained diffusion backbones [ 24 , 34 , 25 , 20 , 10 , 44 , 55 ] . By incorporating large-scale generative priors, these methods synthesize more realistic novel views and generalize better under large viewpoint changes. Nevertheless, achieving reliable multi-view consistency often relies on substantial pose-annotated multi-view datasets [ 56 , 30 , 22 , 8 ] . This reliance becomes particularly important in sparse-view settings, where limited observations increase geometric ambiguity and place greater demands on cross-view reasoning. Recent state-of-the-art models such as SEVA [ 55 ] demonstrate strong cross-domain performance by combining large pretrained image diffusion backbones with extensive multi-view supervision. In contrast, we explore video diffusion models as scalable priors for sparse-view NVS, leveraging cross-frame consistency from web-scale video pretraining and directly repurposing pretrained video backbones for geometric reasoning without relying on large curated multi-view datasets. Video Diffusion Models and 3D-Aware Video Generation. Modern video diffusion architectures [ 39 , 46 ] typically adopt latent diffusion frameworks [ 32 ] with spatio-temporal VAEs [ 4 ] and transformer-based denoisers [ 28 ] , achieving strong temporal consistency and scalability through web-scale video pretraining. Beyond generic video generation, several works incorporate camera control or 3D-aware conditioning into video models. One line of research [ 48 , 31 , 7 , 54 ] first reconstructs a scene and then applies video diffusion to inpaint missing regions during rendering. These approaches provide