UVRM: A Scalable 3D Reconstruction Model from Unposed Videos
Shiu-hong Kao, Xiao Li, Jinglu Wang, Yang Li, Chi-Keung Tang, Yu-Wing Tai, Yan Lu · 2025
Read UVRM: A Scalable 3D Reconstruction Model from Unposed Videos by Shiu-hong Kao, Xiao Li, Jinglu Wang, Yang Li, Chi-Keung Tang, Yu-Wing Tai, Yan Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09347.txt.
Introduction
Shiu-hong Kao and co-authors present UVRM in a 2025 paper on rebuilding 3D objects from video. Classic training for such models needs the camera pose for each frame, a slow step prone to error. That need has kept training to fake data or small sets with poses marked by hand.
UVRM learns from plain single camera video with no pose data at all. A transformer folds the frames into a code that ignores pose, then decodes a three plane form of the object. Training leans on a guide method plus fake new views made by a diffusion model. Tests on two data sets show strong rebuilds across many object types. The model works fast as well as well, the team reports.
Catalog record
| CREATOR | Shiu-hong Kao, Xiao Li, Jinglu Wang, Yang Li, Chi-Keung Tang, Yu-Wing Tai, Yan Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-09347
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UVRM: A Scalable 3D Reconstruction Model from Unposed Videos Shiu-hong Kao, Xiao Li, Jinglu Wang, Yang Li, Chi-Keung Tang, Yu-Wing Tai, Yan Lu Abstract Large Reconstruction Models (LRMs) have recently become a popular method for creating 3D foundational models. Training 3D reconstruction models with 2D visual data traditionally requires prior knowledge of camera poses for the training samples, a process that is both time-consuming and prone to errors. Consequently, 3D reconstruction training has been confined to either synthetic 3D datasets or small-scale datasets with annotated poses. In this study, we investigate the feasibility of 3D reconstruction using unposed video data of various objects. We introduce UVRM , a novel 3D reconstruction model capable of being trained and evaluated on monocular videos without requiring any information about the pose. UVRM uses a transformer network to implicitly aggregate video frames into a pose-invariant latent feature space, which is then decoded into a tri-plane 3D representation. To obviate the need for ground-truth pose annotations during training, UVRM employs a combination of the score distillation sampling (SDS) method and an analysis-by-synthesis approach, progressively synthesizing pseudo novel-views using a pre-trained diffusion model. We qualitatively and quantitatively evaluate UVRM’s performance on the G-Objaverse and CO3D datasets without relying on pose information. Extensive experiments show that UVRM is capable of effectively and efficiently reconstructing a wide range of 3D objects from unposed videos. 1 Introduction The task of digitally reproducing, modifying, and photo-realistically rendering 3D scenes and objects stands as a core research area in computer vision, with wide-ranging applications. As digital landscapes and interactive technologies increasingly pervade sectors such as entertainment, robotics, and design, the demand for scalable and adaptable 3D models is at an all-time high. Recent breakthroughs in neural 3D representations, initiated by NeRF [ 19 ] and expanded upon by subsequent methods like 3DGS [ 13 ] have achieved unprecedented results, paving the way for the creation of generalized 3D foundation models. These models, analogous to the progress in natural language processing and text-to-image models in computer vision, will enable more efficient and flexible 3D content generation, manipulation, and interaction at scale. Figure 1 : Different from previous methods, which either focus on (a) per-scene pose-free training or (b) 3D reconstruction model trained with known camera poses, we propose UVRM (c) aims for a fully pose-free training of 3D reconstruction model from 2D observations. Large Reconstruction Models (LRMs) have contributed notable development on 3D foundation models, offering the ability to reconstruct 3D representations from various objects or scenes in a single forward pass. Currently, LRMs are trained on synthetic 3D datasets, like Objaverse [ 7 ] and its derivatives. Although effective for now, the reliance on synthetic 3D data is increasingly seen as a bottleneck due to the upper-bound quantity and quality of such data, especially in comparison to the abundant data available in image, video, and speech domains. To overcome this, alternative approaches for 3D reconstruction training use large video and image datasets. However, these approaches require precise camera poses to align 3D representations with their corresponding 2D observations for training using reconstruction losses such as render loss. The scarcity of multi-view datasets with accurate pose annotations, combined with the challenges of estimating camera poses from arbitrary images or videos, due to issues like homogeneous regions or view-dependent appearances, poses a significant barrier. Thus, creating a pose-free training and inference framework for 3D reconstruction is crucial for scaling 3D reconstruction efficiently and advancing toward the goal of building generalized 3D foundation models. In this paper, we take an important step toward harnessing large-scale 2D datasets for 3D reconstruction. Our primary objective is to explore whether it is feasible to train a 3D reconstruction model for various objects, using only monocular videos without any pose annotations. We tackle this inquiry from two angles: • Pose-free alignment - Is it possible to achieve 3D reconstruction from an arbitrary number of input views without explicit pose alignment? • Pose-free training - Can we train a reconstruction model solely with 2D data, devoid of pose annotations? We affirmatively answer these questions by introducing UVRM, a R econstruction M odel for U nposed V ideos. To tackle the first question, we propose encoding 3D scenes using viewpoint-invariant features. This differs from traditional approaches, which typically focus on estimating view-specific or pixel-aligned features with explicit pose calibration. Our method encodes all input views into a unified latent space, utilizing a transformer-based model to implicitly aggregate information from multiple views. The result is a latent feature with viewpoint-invariant tokens. This strategy allows for the scaling of inputs to dense, pose-free multi-view images without compromising on memory or computational efficiency. These viewpoint-invariant tokens can then be decoded into a neural 3D representation suitable for rendering. To address the second question, we integrate the Score Distillation Sampling (SDS) method with an analysis-by-synthesis strategy. This involves incrementally augmenting view-consistent pseudo-views using pre-trained diffusion models throughout the training process. Our method circumvents the need to calculate render loss between input and reconstructed views during training, thereby obviating the necessity for ground truth pose annotations for input videos. We evaluate our model’s performance on the synthetic G-Objaverse dataset [ 44 ] without using of pose information, as well as CO3D [ 24 ] dataset with real-world videos. We demonstrate that the proposed UVRM can reconstruct various 3D objects from pose-free monocular videos. Notably, UVRM outperforms prior pose-free NeRF methods that rely on per-object optimization, showcasing superior results. Our contributions can be summarized as: • A new research problem of training 3D reconstruction model from 2D datasets without explicit pose calibration. • A new method that takes pose-free monocular videos as input for 3D object reconstruction. • A novel training pipeline that eliminates pose annotations for training 3D reconstruction models. 2 Related Work Neural 3D Representations. The neural field representation of 3D scenes has attracted significant attention from the literature since the pioneer work of NeRF [ 19 ] . NeRF has demonstrated its effectiveness on the task of view synthesis from multi-view posed images, leading to a number of follow-up works that extend its capabilities. Some representative works including NeUS [ 32 ] , Tri-plane [ 3 ] , and Gaussian splatting [ 13 ] . These techniques utilize multi-layer perceptrons to generate implicit or hybrid fields such as sign distance functions or volume radiance fields. Our method leverage the expressiveness of neural representations, aims to reconstruct 3D objects with minimal requirement of inputs (i.e., pose-free monocular video). Multi-view 3D Reconstruction. Vanilla multi-view reconstruction method, regardless of its 3D representations, requires accurate camera poses. Camera poses are often obtained from Structure-from-Motion (SfM) methods such as COLMAP [ 28 ] , which significantly increases the time cost and risk of failure, due to the sensitivity of traditional feature matching strategy. Some recent works incorporates this pipeline with neural representations to jointly improve the reconstruction quality and camera estimation robustness [ 39 , 34 , 37 , 9 , 8 ] . Another stream of works rely on additional information such as depth [ 2 , 29 ] or optical flow [ 18 ] . Our method focus on 3D reconstruction from pose-free videos without explicit pose calibration. Figure 2 : UVRM architecture. We propose UVRM,