Joint Learning of Depth, Pose, and Local Radiance Field for Large Scale Monocular 3D Reconstruction

Shahram Najam Syed, Yitian Hu, Yuchao Yao · 2025

Read Joint Learning of Depth, Pose, and Local Radiance Field for Large Scale Monocular 3D Reconstruction by Shahram Najam Syed, Yitian Hu, Yuchao Yao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18237.txt.

Introduction

Shahram Najam Syed, Yitian Hu, and Yuchao Yao at Carnegie Mellon University rebuild large scenes in 3D from a single moving camera. Solving depth, camera pose, and radiance separately breaks down at scale. Ambiguous depth creates ghost geometry, pose drift corrupts alignment, and one global neural radiance field cannot hold hundreds of meters of content. Their 2025 framework learns all three jointly, so each part corrects the others.

A vision transformer depth network, trained with metric scale supervision, gives consistent depths across wide views. A feature space bundle adjustment layer refines poses from learned descriptors instead of brittle keypoints, suppressing drift. Local radiance fields are added and frozen on the fly as view overlap drops, covering city block scale on one GPU. On the Tanks and Temples benchmark, trajectory error falls to between 0.001 and 0.021 meters across eight sequences. That is up to 18 times lower than BARF, with no calibration needed.

Catalog record

CREATORShahram Najam Syed, Yitian Hu, Yuchao Yao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-18237

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Joint Learning of Depth, Pose, and Local Radiance Field for Large Scale Monocular 3D Reconstruction Shahram Najam Syed, Yitian Hu, Yuchao Yao Abstract Photorealistic 3-D reconstruction from monocular video collapses in large-scale scenes when depth, pose, and radiance are solved in isolation: scale-ambiguous depth yields ghost geometry, long-horizon pose drift corrupts alignment, and a single global NeRF cannot model hundreds-of-metres of content. We introduce a joint learning framework that couples all three factors and demonstrably overcomes each failure case. Our system begins with a Vision-Transformer (ViT) depth network trained with metric-scale supervision, giving globally consistent depths despite wide field-of-view variations. A multi-scale feature bundle-adjustment (BA) layer refines camera poses directly in feature space—leveraging learned pyramidal descriptors instead of brittle keypoints—to suppress drift on unconstrained trajectories. For scene representation, we deploy an incremental local-radiance-field hierarchy: new hash-grid NeRFs are allocated and frozen on-the-fly when view overlap falls below a threshold, enabling city-block-scale coverage on a single GPU. All modules are integrated in a progressive windowed optimiser that alternates depth warm-up, BA pose refinement, and radiance fine-tuning, enforcing mutual geometric consistency without any external calibration. Evaluated on the Tanks & Temples benchmark, our method reduces Absolute Trajectory Error to 0.001–0.021 m across eight indoor–outdoor sequences—up to 18 × \times lower than BARF and 2 × \times lower than NoPe-NeRF—while maintaining sub-pixel Relative Pose Error. On the large-scale Static Hikes (Indoor) set, we achieve PSNR 20.19 dB, SSIM 0.704, and LPIPS 0.62, matching or surpassing state-of-the-art view-synthesis baselines. Qualitative reconstructions show drift-free, meter-accurate geometry over trajectories exceeding 100 m. These results demonstrate that metric-scale, drift-free 3-D reconstruction and high-fidelity novel-view synthesis are achievable from a single uncalibrated RGB camera, paving the way for practical AR/VR mapping, autonomous-robot perception, and digital-twin generation in unstructured environments. 1 Introduction Dense 3-D scene reconstruction underpins a wide spectrum of emerging applications—from AR navigation on commodity smartphones to autonomous robot Exploration, telepresence, and digital-twin generation. Practical deployments, however, often demand metric-scale accuracy over hundreds of metres while relying on nothing more than an uncalibrated, hand-held monocular camera. Meeting all three requirements simultaneously remains an open challenge. Why monocular pipelines still fail. Self-supervised depth–pose networks (e.g., [ 2 , 3 ] ) learn from photometric warping without ground truth, yet their depth is ambiguous up to a single, unknown scale, and accumulated pose error drifts over long trajectories. Classic SfM pipelines such as COLMAP [ 10 ] recover scale but require textured scenes, extensive feature matches, and pre-calibrated intrinsics. Neural radiance fields (NeRF) [ 1 ] achieve photo- realistic novel-view synthesis, yet standard practice assumes known intrinsics and short camera paths; even calibration-relaxing extensions like BARF [ 5 ] and NoPe-NeRF [ 6 ] struggle to stay drift-free at city-block scale and invariably rely on a single global NeRF, leading to prohibitively large memory footprints. Our method Our paper highlights three empirically verified failure modes— scale drift, pose drift, and limited NeRF extent —and introduced a single framework that neutralises all three by coupling depth, pose, and radiance optimisation . Concretely: • Metric-scale depth. A Vision-Transformer (ViT) backbone, trained with object-size priors, yields scale-consistent depth even under varied field-of-view. • Drift-free poses. Multi-scale feature bundle adjustment refines SE(3) trajectories by directly back-propagating reprojection residuals through learned pyramidal descriptors, simultaneously updating camera intrinsics. • Incremental radiance fields. Lightweight hash-grid NeRFs are spawned and frozen on-the-fly when the viewing baseline to existing fields exceeds a threshold, enabling seamless coverage of hundred-metre paths. All modules are trained in a progressive windowed schedule that alternates depth warm-up, feature-space BA, and radiance fine-tuning, thereby maintaining mutual geometric consistency without any external calibration data. Contributions. Building on the above insight, this paper makes three contributions: 1. A ViT-based depth estimator coupled with feature-level bundle adjustment that jointly refines depth, intrinsics, and pose in a single differentiable loop. 2. An incremental local-radiance-field representation that scales NeRF to city-block trajectories while keeping training time and memory bounded. 3. An extensive evaluation on Tanks & Temples , Static Hikes , and custom CMU smartphone footage, demonstrating up to 18 \times lower ATE than BARF and competitive view-synthesis quality at one-third the runtime. The remainder of the paper details our method (§ 3 ), experimental setup (§ 4 ), and ablations, culminating in a discussion of limitations and future directions. 2 Related Work Monocular 3-D reconstruction sits at the intersection of three research threads: self-supervised depth–pose learning, neural implicit scene representations, and joint optimisation of camera parameters and geometry. We briefly review each line of work and underline the limitations our framework addresses. 2.1 Self-Supervised Monocular Depth and Pose The seminal work of Zhou et al. [ 2 ] showed that depth and ego-motion can be learned without ground truth by minimising photometric reprojection error between consecutive frames. Follow-up efforts improved robustness with edge-aware smoothness [ 3 ] , feature-metric consistency [ 11 ] , and scale-consistent training [ 4 ] . Despite these advances, predictions remain scale-ambiguous and degrade in low-texture regions or long trajectories—issues we mitigate through metric-scale priors and feature bundle adjustment. 2.2 Neural Radiance Fields NeRF [ 1 ] revolutionised view synthesis but assumes known intrinsics and short camera paths. Speed-ups such as Instant-NGP [ 7 ] , TensoRF [ 12 ] , and Plenoxels [ 13 ] reduce training time, yet still rely on calibrated inputs. BARF [ 5 ] jointly refines poses during training, while Self-Calibrating-NeRF (SC-NeRF) [ 8 ] optimises intrinsics, but both require coarse SfM initialisation . NoPe-NeRF [ 6 ] dispenses with external poses but exhibits significant drift on unstructured data. Moreover, all above methods employ a single global field , which scales poorly beyond room-sized scenes. 2.3 Large-Scale and Local Radiance Fields To extend NeRF to outdoor or city-scale environments, Mega-NeRF [ 14 ] and Block-NeRF [ 15 ] partition scenes into static spatial tiles, assuming accurate LIDAR or SfM priors. LocalRF [ 9 ] dynamically spawns local fields but still depends on ground-truth poses. Our incremental hash-grid representation inherits the locality principle yet is, to our knowledge, the first to operate with un-calibrated, pose-unknown monocular video. 2.4 Joint Optimisation of Calibration and Geometry Early differentiable SfM frameworks [ 16 , 17 ] combine feature extraction and bundle adjustment, but they output sparse point clouds rather than photorealistic renderings. Hybrid approaches couple depth networks with NeRF losses—e.g. iNeRF [ 18 ] or NeuralBundler [ 19 ] —yet still inherit scale or calibration dependencies. Our method advances this line by fusing a ViT-based metric depth, feature-level pose BA, and incremental local NeRFs in a single, fully differentiable pipeline. Prior work solves at most two of the three challenges—scale ambiguity, pose drift, or limited field extent—often assuming external calibration or dense features. By jointly learning metric depth, drift-free pose, and incremental radiance fields, our framework is the first to deliver photorealistic, metre-accurate reconstructions from raw, uncalibrated handheld video over city-block trajectories. 3 Methodology Given an uncalibrated RGB stream \{I_{t}\}_{t=0}^{T} , our pipeline jointly estimates (i) metre-scale depth maps D_{t} , (ii) drift-free camera poses \mathbf{T}_{t}\!\in\!\mathrm{SE}(3) , and (iii) an incremental set of hash-grid local radiance fields \{\mathcal{R}_{k}\} . Figure 1 : Overview of our joint depth–pose–radiance pipeline. A ViT-based depth module predicts metric depth (left), a coarse-to-fine feature-metric bundle-adjustment (center) refines

Read the complete text