Reconstruct, Inpaint, Finetune: Dynamic Novel-view Synthesis from Monocular Videos
Kaihua Chen, Tarasha Khurana, Deva Ramanan · 2025
Read Reconstruct, Inpaint, Finetune: Dynamic Novel-view Synthesis from Monocular Videos by Kaihua Chen, Tarasha Khurana, Deva Ramanan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12646.txt.
Introduction
Kaihua Chen, Tarasha Khurana, and Deva Ramanan synthesize new camera views of moving scenes from a single video, in a 2025 paper. Filming a dance from one phone leaves the far side unseen. Rebuilding that view is hard: optimization-heavy methods are slow, and pure feed-forward ones warp the geometry.
Their recipe has three steps. First reconstruct the dynamic 3D scene and render the parts visible from the new angle. Then let a video diffusion model, CogNVS, paint in the hidden parts; it teaches itself from ordinary 2D videos, so wild footage can train it. Last, fine-tune it briefly on the test video itself. Across benchmarks the method beat nearly all prior art for dynamic novel views.
Catalog record
| CREATOR | Kaihua Chen, Tarasha Khurana, Deva Ramanan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-12646
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reconstruct, Inpaint, Finetune: Dynamic Novel-view Synthesis from Monocular Videos Kaihua Chen, Tarasha Khurana, Deva Ramanan Abstract We explore novel-view synthesis for dynamic scenes from monocular videos. Prior approaches rely on costly test-time optimization of 4D representations or do not preserve scene geometry when trained in a feed-forward manner. Our approach is based on three key insights: (1) covisible pixels (that are visible in both the input and target views) can be rendered by first reconstructing the dynamic 3D scene and rendering the reconstruction from the novel-views and (2) hidden pixels in novel views can be “inpainted" with feed-forward 2D video diffusion models. Notably, our video inpainting diffusion model (CogNVS) can be self-supervised from 2D videos, allowing us to train it on a large corpus of in-the-wild videos. This in turn allows for (3) CogNVS to be applied zero-shot to novel test videos via test-time finetuning . We empirically verify that CogNVS outperforms almost all prior art for novel-view synthesis of dynamic scenes from monocular videos. Figure 1: We present CogNVS, a video diffusion model that enables novel-view synthesis of dynamic scenes. Given an in-the-wild monocular video of a dynamic scene, we first reconstruct the scene, render it from the target novel-view and inpaint any unobserved regions. Because CogNVS can be pre-trained via self-supervision, it can also be test-time-finetuned on a given target video, enabling it to zero-shot generalize to novel domains. Our simple pipeline outperforms almost all prior state-of-the-art for dynamic novel-view synthesis. We show outputs from CogNVS from two unseen videos; a generated video above, and a real-world video below. 1 Introduction Rapid advances in static 3D scene representations [ 1 , 2 ] have paved the way for spacetime understanding of the dynamic world. This has enabled photorealistic content creation and immersive virtual reality applications. In this work, we focus on the problem of novel-view synthesis from casually-captured monocular videos of dynamic scenes. Why is this hard? Prior work on dynamic view synthesis addresses this task from two extremes. The first class of methods “test-time” optimize a new 4D representation from scratch for every new test video. While this ensures physically-plausible scene geometry, careful choices in modeling scene motion – in the form of an independent deformation field, or learnable temporal offsets – have to be made [ 3 , 4 , 5 ] . More importantly, it can take on the order of hours to optimize and render a novel-view video. An attractive alternative is to train large feed-forward video models directly for view synthesis [ 6 , 7 ] . While inference on such models is dramatically faster (on the order of milliseconds), the resulting renderings often are not as accurate as their test-time optimized counterparts. From a pragmatic perspective, such models need to be trained on mega-scale multi-view training data, which is difficult to obtain for dynamic scenes. Our method addresses the above challenges by decomposing the problem of dynamic view-synthesis into three distinct stages. First, we lean on the success of non-rigid structure from motion [ 8 , 9 , 10 ] approaches that produce reconstructions of visible scene regions, sometimes known as "2.5D" reconstructions (since occluded regions are not reconstructed). We point out that such reconstructions can be trivially produced for casual mobile videos captured with depth sensors and egomotion [ 11 ] . When such reconstructions are rendered from a target novel view, previously-hidden regions will not not be rendered. To “inpaint” these regions, we train a 2D video-inpainter – CogNVS – by fine-tuning a video diffusion model (CogVideoX [ 12 ] ) to condition on the partially-observable novel-view pixels. Importantly, we allow CogNVS to also update the appearance of previously-visible pixels, allowing our pipeline to model view-dependent (dynamic) scene effects. The key insight of our work is that CogNVS can be trained on any 2D videos via self-supervision. However, rather than training our inpainter with random 2D masks, we make use of 3D multi-view supervision that better captures 3D scene visiblity, similar to prior art [ 13 ] . Specifically, given a 2D training video, we first reconstruct it (with an off-the-shelf method such as MegaSAM) and then render the reconstruction from a random camera trajectory. This rendering is used to identify co-visible pixels from the source video that remain visible in the novel views. This original source video and its co-visible-only masked variant can now form a training pair for 3D-consistent video inpainting. Importantly, because such a training pair does not require ground-truth 3D supervision, CogNVS can be trained on diverse in-the-wild 2D videos. We use dynamic scenes from TAO [ 14 ] , SA-V [ 15 ] , Youtube-VOS [ 16 ] , and DAVIS [ 17 ] . Equally as important, we use the same paradigm to test-time finetune CogNVS on the test video-of-interest. We show that this allows our pipeline to “zero-shot” generalize to test videos that were never seen during training. We argue that our test-time finetuning of 2D diffusion models can be seen as the “best-of-both-worlds”, by leveraging large-scale training data (for data-driven robustness) and test-time optimization (for accuracy). In summary, our contributions are as follows: (1) We decompose dynamic view synthesis into three stages of reconstruction, inpainting and test-time finetuning, (2) we use a large corpus of only 2D videos for training CogNVS, and (3) we do extensive zero-shot benchmarking on three evaluation datasets against state-of-the-art methods and show improvements on dynamic view synthesis. 2 Related Work Novel-view synthesis has seen recent advancements with the rise of implicit scene representations like NeRFs [ 1 ] and Gaussian primitives [ 2 , 18 , 19 ] . We have seen widespread efforts in scaling these representations to model larger scenes [ 20 , 21 , 22 ] , making them faster to fit [ 23 , 24 , 25 , 2 , 26 , 27 , 28 , 29 ] , anti-aliased [ 30 , 31 , 32 , 33 , 34 ] , and extend to representing dynamic scenes [ 35 , 36 , 37 , 23 ] . The most popular paradigms have been the adoption of dynamic NeRFs [ 1 , 37 , 38 , 39 ] and deformable Gaussian primitives [ 2 , 5 , 40 , 41 ] for modeling scene dynamics, apart from using voxel grids [ 42 , 43 ] or learnable tokenzation [ 44 ] . Most approaches need multi-view posed videos as input, and only recently monocular view synthesis has gained traction [ 45 , 46 , 3 , 47 ] . However, each of the aforementioned approaches have to be test-time optimized separately for every new test video, are slow to optimize and yet fail to recover highly-detailed dynamic scene content [ 46 ] . Moreover, there is no focus on predicting the unobservable scene content, which is exacerbated by benchmarking metrics that only evaluate co-visible pixels [ 11 ] in training and inference views and therefore encourage benchmarking on novel views that are not too far apart from the training views. Our approach instead reformulates dynamic view synthesis as an inpainting task, which specifically focuses on generating parts of the scene that were occluded from the training views, thereby facilitating extreme novel view synthesis