VDNeRF: Vision-only Dynamic Neural Radiance Field for Urban Scenes
Zhengyu Zou, Jingfeng Li, Hao Li, Xiaolei Hou, Jinwen Hu, Jingkun Chen, Lechao Cheng, Dingwen Zhang · 2025
Read VDNeRF: Vision-only Dynamic Neural Radiance Field for Urban Scenes by Zhengyu Zou, Jingfeng Li, Hao Li, Xiaolei Hou, Jinwen Hu, Jingkun Chen, Lechao Cheng, Dingwen Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06408.txt.
Introduction
Zhengyu Zou and colleagues reconstruct moving city scenes in three dimensions without being handed camera poses. Neural radiance fields render realistic new views of a scene from photos, but they usually need exact camera positions and balk at large dynamic streets. Their method, VDNeRF, recovers the camera path and the scene motion from images alone.
Two radiance field models split the job: one handles the static background and camera poses, the other tracks dynamic objects through three-dimensional scene flow. A training scheme separates camera motion from object motion without extra sensors. On major urban driving datasets, VDNeRF beat leading pose-free methods at both pose estimation and dynamic view synthesis.
Catalog record
| CREATOR | Zhengyu Zou, Jingfeng Li, Hao Li, Xiaolei Hou, Jinwen Hu, Jingkun Chen, Lechao Cheng, Dingwen Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06408
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VDNeRF: Vision-only Dynamic Neural Radiance Field for Urban Scenes Zhengyu Zou, Jingfeng Li, Hao Li, Xiaolei Hou, Jinwen Hu, Jingkun Chen, Lechao Cheng, Dingwen Zhang Abstract Neural Radiance Fields (NeRFs) implicitly model continuous three-dimensional scenes using a set of images with known camera poses, enabling the rendering of photorealistic novel views. However, existing NeRF-based methods encounter challenges in applications such as autonomous driving and robotic perception, primarily due to the difficulty of capturing accurate camera poses and limitations in handling large-scale dynamic environments. To address these issues, we propose Vision-only Dynamic NeRF (VDNeRF), a method that accurately recovers camera trajectories and learns spatiotemporal representations for dynamic urban scenes without requiring additional camera pose information or expensive sensor data. VDNeRF employs two separate NeRF models to jointly reconstruct the scene. The static NeRF model optimizes camera poses and static background, while the dynamic NeRF model incorporates the 3D scene flow to ensure accurate and consistent reconstruction of dynamic objects. To address the ambiguity between camera motion and independent object motion, we design an effective and powerful training framework to achieve robust camera pose estimation and self-supervised decomposition of static and dynamic elements in a scene. Extensive evaluations on mainstream urban driving datasets demonstrate that VDNeRF surpasses state-of-the-art NeRF-based pose-free methods in both camera pose estimation and dynamic novel view synthesis. I Introduction Dynamic urban scene reconstruction using Neural Radiance Field (NeRF) [ 26 ] has become a pivotal research area due to its ability to synthesize photorealistic images from arbitrary viewpoints. This capability is essential for applications such as robotic perception [ 30 ] , augmented reality (AR) [ 4 ] , virtual reality (VR) [ 3 ] , scene understanding [ 18 , 56 ] , and autonomous driving [ 17 ] . However, traditional NeRF methods typically require precise camera pose information for each image, which is often challenging to obtain accurately in real-world scenarios [ 14 ] . To estimate camera poses from images, Structure from Motion (SfM) [ 37 ] techniques are commonly employed. Unfortunately, these methods can be time-consuming and may lack robustness, particularly in urban environments where dynamic elements like vehicles and pedestrians introduce variability. This variability can lead to multi-view inconsistencies across input images, making SfM less effective. To eliminate the dependency on SfM, methods such as Nope-NeRF [ 2 ] and LocalRF [ 25 ] have been developed to jointly optimize both camera poses and NeRF parameters by parameterizing camera poses. However, these approaches assume strictly static scenes, an assumption that is often violated in dynamic urban environments. The presence of moving objects introduces challenges in distinguishing between camera-induced motion and object-induced motion, complicating the optimization process and limiting the applicability of these methods in real-world urban scenarios. RoDynRF [ 22 ] extends NeRF-based pose-free reconstruction to dynamic scenes by employing object segmentation to generate 2D motion masks, thereby decoupling static and dynamic components. It utilizes a static radiance field to estimate camera poses and reconstruct the static background, while a dynamic radiance field, combined with a deformation field, reconstructs dynamic objects. However, RoDynRF faces limitations in effectively representing complex urban scenes, often resulting in visual artifacts and low-resolution outputs. Moreover, its reliance on precise 2D motion masks, which can be unreliable in urban settings, may lead to incorrect identification of stationary objects as dynamic, resulting in inaccurate scene representations. Beyond these limitations, two key challenges must be addressed to achieve NeRF-based vision-only urban scene reconstruction and perception: (I) Robust and Accurate Camera Pose Estimation: In long and narrow urban scenes with dynamic objects, achieving precise camera pose estimation is crucial for modeling object motion and representing spatiotemporal scenes. However, existing methods often fall into local minima in such complex settings, leading to failures in recovering extensive camera trajectories. (II) Self-Supervised Decomposition of Static and Dynamic Elements: Effectively separating static backgrounds from dynamic foregrounds without precise annotations is vital for applications like identifying moving vehicles in autonomous driving. The joint optimization of camera poses and NeRF introduces challenges, as uncertainties in camera poses and the presence of dynamic objects can interfere with the static-dynamic decomposition process. In this paper, we propose VD-NeRF (Vision-only Dynamic NeRF), a method that achieves novel view synthesis and spatiotemporal reconstruction for urban scenes using only a sequence of input images. VD-NeRF utilizes two separate NeRF models to represent dynamic scenes: (a) Static NeRF Model : Responsible for optimizing camera poses and reconstructing the static background; (b) Dynamic NeRF Model : Reconstructs dynamic objects by introducing temporal encoding and employs a flow field to predict 3D scene flow, ensuring consistency and accuracy in modeling dynamic objects. Additionally, VD-NeRF incorporates a carefully designed training framework that enables robust camera pose estimation and self-supervised static-dynamic decomposition, allowing for more accurate reconstruction and perception of urban spatiotemporal scenes. The scene is divided into overlapping sub-scenes based on spatial dimensions, and training begins with a limited number of images, progressively incorporating more as training advances. Motion masks are utilized to minimize the interference of dynamic objects during camera pose estimation. Once accurate camera poses are established, the dynamic NeRF model reconstructs dynamic foregrounds and performs self-supervised decoupling of dynamic objects without relying on motion masks. By leveraging 3D scene flow, the model aggregates information across multiple time steps, enhancing the distinction between dynamic and static objects and reducing the likelihood of incorrect decoupling. In summary, our core contributions are: • We present a method that uses only image sequences from dynamic urban scenes to recover accurate camera trajectories, learn spatiotemporal scene representations, and achieve photorealistic dynamic novel view synthesis. • We propose a novel training framework that enhances the robustness and accuracy of camera pose estimation and urban scene reconstruction, enabling self-supervised static-dynamic decomposition without requiring precise annotations. • Our method is extensively validated on mainstream urban scene datasets, demonstrating state-of-the-art performance in both camera pose estimation and dynamic novel view synthesis. II Related Work II-A NeRF With Camera Pose Estimation. Eliminating camera pre-processing to achieve joint optimization of camera poses and NeRF is a hot research area. The first to utilize NeRF for optimizing camera poses is iNeRF [ 54 ] which formulated camera pose estimation as an inverse problem. Leveraging a pre-trained NeRF model, it estimates poses by sampling key points. Since then, many NeRF-based pose-free methods have been proposed. NeRF -- [ 46 ] treats camera poses as learnable parameters, proposing a framework for joint optimization of camera poses and NeRF. GNeRF [ 24 ] introduces Generative Adversarial Networks (GAN) to optimize camera poses. BARF [ 21 ] extends the classical image alignment theory to NeRF, proposing a coarse-to-fine registration strategy for camera pose recovery. While recovering the camera poses, SC-NeRF [ 13 ] also takes camera intrinsic and distortion as optimization parameters. L2G-NeRF [ 5 ] introduces a local-to-global registration method to avoid falling into local minima. Nope-NeRF [ 2 ] utilizes monocular depth estimation to provide a prior to constrain the relative pose between consecutive images, recovering the large camera motion trajectory. LocalRF [ 25 ] progressively jointly optimizes camera poses and NeRF, improving robustness in long camera trajectories and large-scale scenes. CF-NeRF [ 52 ] and CT-NeRF [ 34 ] use an incremental optimization approach to handle