Dynamic View Synthesis as an Inverse Problem
Hidir Yesiltepe, Pinar Yanardag, [email protected] · 2025
Read Dynamic View Synthesis as an Inverse Problem by Hidir Yesiltepe, Pinar Yanardag, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08004.txt.
Introduction
Hidir Yesiltepe and Pinar Yanardag recast dynamic view synthesis, making new camera angles from one video, as an inverse problem solved on a frozen video model. No retraining is allowed. The trick sits in how the starting noise is built: a snag in standard noise schedules blocks clean inversion, so they design a recursive noise form with a closed solution.
A second step fills regions the camera move reveals, sampling hidden areas with awareness of what should be visible. Experiments show sharp, faithful new views from the latent tweaks alone. The work hints that much video generation power hides in the starting noise.
Catalog record
| CREATOR | Hidir Yesiltepe, Pinar Yanardag, [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08004
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dynamic View Synthesis as an Inverse Problem Hidir Yesiltepe, Pinar Yanardag, [email protected] Abstract In this work, we address dynamic view synthesis from monocular videos as an inverse problem in a training-free setting. By redesigning the noise initialization phase of a pre-trained video diffusion model, we enable high-fidelity dynamic view synthesis without any weight updates or auxiliary modules. We begin by identifying a fundamental obstacle to deterministic inversion arising from zero-terminal signal-to-noise ratio (SNR) schedules and resolve it by introducing a novel noise representation, termed K-order Recursive Noise Representation. We derive a closed form expression for this representation, enabling precise and efficient alignment between the VAE-encoded and the DDIM inverted latents. To synthesize newly visible regions resulting from camera motion, we introduce Stochastic Latent Modulation, which performs visibility aware sampling over the latent space to complete occluded regions. Comprehensive experiments demonstrate that dynamic view synthesis can be effectively performed through structured latent manipulation in the noise initialization phase. Figure 1: From real-world complex scenes to AI-generated videos, our method preserves identity fidelity and synthesizes plausible novel views by operating entirely in noise initialization phase. 1 Introduction Dynamic view synthesis (DVS) [ 13 , 35 , 43 , 53 , 59 ] from monocular videos [ 10 , 11 , 26 , 60 , 48 , 2 , 15 , 58 , 55 ] is a computer vision task that aims to generate new, dynamic perspectives of a scene using only a single video as input. This process involves predicting how a scene would appear from angles not captured in the original footage, requiring the inference of depth, occluded regions, and unseen details. In the film industry, DVS can revolutionize post-production by enabling virtual camera sweeps through a scene or producing additional shots from different angles, eliminating the need for expensive reshoots. In robotics, it supports advanced perception systems by generating synthetic viewpoints that train algorithms for navigation, manipulation, and active perception tasks in complex environments. Historically, DVS has relied on explicit 3D reconstruction methods, such as Neural Radiance Fields (NeRF) [ 34 ] , its dynamic extension D-NeRF [ 40 ] , K-Planes [ 9 ] , and 3D/4D Gaussian Splatting [ 22 , 52 ] . These seminal volumetric and point-based approaches model scenes as continuous representations or point configurations. However, they impose strict prerequisites: multi-view supervision, computationally intensive per-scene optimization, and precise camera calibration. Recently, a paradigm shift has emerged, leveraging video-diffusion [ 5 , 57 , 49 , 4 , 29 ] priors to address these limitations. Diffusion models appeal for DVS because they implicitly capture geometry and appearance in their latent space, inherently provide temporal consistency, and bypass the need for explicit 3D modeling. Under this diffusion paradigm, two dominant approaches have surfaced. The first involves attention-sharing architectures, as seen in Generative Camera Dolly [ 48 ] , TrajectoryAttention [ 55 ] , TrajectoryCrafter [ 58 ] , and ReCamMaster [ 2 ] . These methods integrate camera-aware branches such as pixel-trajectory attention, dual streams, or 3D attention layers to enable fine-grained camera control. However, they require additional architectural modules and extensive retraining on large synthetic datasets like Unreal Engine 5 [ 8 ] or Kubric [ 14 ] , leading to domain-gap issues when applied to natural settings. The second recipe employs LoRA-based fine-tuning, exemplified by ReCapture [ 60 ] and Reangle-A-Video [ 20 ] . These approaches attach spatial and temporal Low-Rank Adaptations (LoRAs) [ 18 , 7 , 42 ] , and perform per-video fine-tuning leveraging masked losses. Across both strategies, shared limitations persist: they require updating backbone parameters or adding layers, depend on curated synthetic data or video-specific fine-tuning, and suffer from pitfalls when the inversion process misaligns with the model’s forward noise schedule. These constraints underscore a critical open question: Can we achieve 6-DoF monocular DVS without any weight updates, auxiliary modules, or synthetic pre-training purely by manipulating the initial noise fed into a video-diffusion model? In this work, we pioneer a fundamentally different approach to DVS from monocular videos. We demonstrate that by solely manipulating the initial noise fed into a video diffusion model, we can achieve state-of-the-art performance without any weight updates or auxiliary modules. This novel perspective shifts the focus from architectural redesign or resource-intensive retraining to efficient noise design, distinguishing our method from existing approaches. Our approach is centered around two key innovations. First, we identify and formalize the Zero-Terminal SNR Collapse Problem , which arises when training schedules enforce zero signal-to-noise ratio at the terminal timestep, causing a collapse in information content and obstructing deterministic inversion. To resolve this, we propose the K-order Recursive Noise Representation (K-RNR) , which recursively refines the initial noise in alignment with the model’s forward schedule, enabling stable and faithful reconstruction of the original scene. We derive closed-form expressions for this refinement process and stabilize generation with an adaptive variant that prevents scale explosion. Second, to address the synthesis of newly visible content due to camera motion, we introduce Stochastic Latent Modulation , a visibility-aware sampling mechanism that directly completes occluded latent regions using context-aware latent permutations. This enables plausible scene completion in the noise initialization phase. Together, these components form a unified framework that achieves high-fidelity reconstruction and physically consistent view synthesis from monocular input. Our contributions can be summarized as follows: • We identify and formalize the Zero-Terminal SNR Collapse Problem, showing that while zero terminal SNR schedules improve generation quality, they inherently break injectivity, preventing deterministic inversion and hindering faithful reconstruction. • We propose K-order Recursive Noise Representation (K-RNR) to resolve the obstruction caused by the Zero-Terminal SNR Problem. By defining a recursive refinement relation between the VAE-encoded latent and the positive-SNR DDIM-inverted latent, we derive closed-form noise expression, enabling high-fidelity reconstructions of original scenes. • We introduce Stochastic Latent Modulation (SLM), a novel latent-space completion mechanism that infers content for newly visible regions by performing visibility-aware sampling and contextual latent permutation, enabling physically plausible synthesis in occluded areas without modifying the model. 2 Related Work This section reviews prior research in two closely related areas relevant to our work. The first is novel view synthesis for dynamic scenes, and the second is video-to-video translation with camera control. Figure 2: Approaches to Zero-Terminal SNR Collapse Problem. (b) Low strength preserves source content but renders unseen regions as black. (c) High strength improves propagation into unseen areas but causes identity drift. (d) DDIM inverted latent as initial noise leads to washed-out , high saturation generation. (f) Our K-RNR ( k=6 ) with Stochastic Latent Modulation preserves identity and completes newly visible regions with plausible content. Novel View Synthesis for Dynamic Scenes. Novel view synthesis seeks to generate unseen perspectives from available visual data, with substantial advancements driven by neural rendering. For static scenes, Neural Radiance Fields (NeRF) [ 34 ] and 3D Gaussian Splatting [ 22 ] provide detailed 3D reconstructions. Dynamic scene extensions, such as D-NeRF [ 40 ] , K-Planes [ 9 ] , HexPlane [ 6 ] , and HyperReel [ 1 ] , depend on synchronized multi-view inputs, which are often impractical for casual settings. Monocular video methods, including Neural Scene Flow Fields [ 27 ] , DynIBaR [ 28 ] , Robust Dynamic Radiance Fields