Learning 3D Reconstruction with Priors in Test Time
Lei Zhou, Haoyu Wu, Akshat Dave, Dimitris Samaras · 2026
Read Learning 3D Reconstruction with Priors in Test Time by Lei Zhou, Haoyu Wu, Akshat Dave, Dimitris Samaras (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03878.txt.
Introduction
Lei Zhou and colleagues improved 3D reconstruction from multiple views by treating known camera poses, intrinsics, and depth as constraints at test time. Rather than retraining image-only transformers, they optimized the network during inference with a self-supervised loss plus prior penalties. The method cut point-map distance error by more than half on ETH3D, 7-Scenes, and NRGBD sets. It also beat retrained prior-aware feed-forward methods, showing test-time optimization can carry prior knowledge.
Catalog record
| CREATOR | Lei Zhou, Haoyu Wu, Akshat Dave, Dimitris Samaras |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38961 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.03878
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning 3D Reconstruction with Priors in Test Time Lei Zhou, Haoyu Wu, Akshat Dave, Dimitris Samaras Abstract We introduce a test-time framework for multiview Transformers (MVTs) that incorporates priors (e.g., camera poses, intrinsics, and depth) to improve 3D tasks without retraining or modifying pre-trained image-only networks. Rather than feeding priors into the architecture, we cast them as constraints on the predictions and optimize the network at inference time. The optimization loss consists of a self-supervised objective and prior penalty terms. The self-supervised objective captures the compatibility among multi-view predictions and is implemented using photometric or geometric loss between renderings from other views and each view itself. Any available priors are converted into penalty terms on the corresponding output modalities. Across a series of 3D vision benchmarks, including point map estimation and camera pose estimation, our method consistently improves performance over base MVTs by a large margin. On the ETH3D, 7-Scenes, and NRGBD datasets, our method reduces the point-map distance error by more than half compared with the base image-only models. Our method also outperforms retrained prior-aware feed-forward methods, demonstrating the effectiveness of our test-time constrained optimization (TCO) framework for incorporating priors into 3D vision tasks. The code is available at https://github.com/cvlab-stonybrook/TCO . 1 Introduction Reconstructing 3D geometry from images is a fundamental problem in computer vision, with potential applications in autonomous driving 9 , augmented reality 10 , and robotics 34 . Traditional methods, such as Structure from Motion (SfM) 15 ; 40 ; 36 ; 31 , approach the problem with multi-stage pipelines, including feature matching 28 ; 5 ; 8 ; 39 ; 35 ; 26 and bundle adjustment 41 ; 36 , that are optimized separately. To address this, feed-forward multiview transformers (MVTs) 45 ; 25 ; 3 ; 43 ; 46 ; 22 ; 19 are proposed to perform 3D reconstruction from a set of images with just one forward pass. Thanks to large-scale end-to-end training, MVTs achieve impressive results. However, MVTs 43 ; 45 ; 25 ; 3 ; 46 are natively designed to only accept RGB images as input. This limits their applications in real-world scenarios where extra information is available. For example, camera poses can be obtained from a camera rig system 20 or sophisticated SfM methods, such as COLMAP 36 , and can be used to improve geometry reconstruction quality. In another scenario, depth maps can be obtained from depth sensors, such as LiDAR or stereo cameras, which complement RGB images for improving camera pose estimation 55 ; 54 . More recent works 19 ; 22 modify MVTs to accept additional priors as input, including camera poses and depth maps. However, these methods are bound to specific architectures and types of priors. They must be retrained whenever the backbone or the available priors change, which is inflexible and computationally expensive. In this paper, we propose a simple test-time constrained optimization (TCO) framework that incorporates priors without changing or retraining the original MVTs. Rather than feeding priors as input, we treat them as constraints on the model’s outputs and optimize the network to fit them at inference. The idea is to leverage the ground truths (e.g., priors) of some prediction modalities to improve the other prediction modalities as well. To prevent the optimization from overfitting to the priors without improving the overall 3D task quality, the optimization loss consists of prior penalty terms and a self-supervised objective. The self-supervised objective is defined as the compatibility among MVTs’ predictions, implemented by the photometric or geometric loss between the renderings projected from other views and each view itself. To exploit the synergy between the different prediction modalities, we adopt a fine-tuning strategy that freezes all the prediction heads but fine-tunes only the shared decoder network with LoRA 16 . We evaluate our method across standard 3D vision benchmarks, including point-map estimation with camera poses and intrinsics and camera pose estimation with depth maps. We observe consistent improvements over base MVTs across all the benchmarks. On ETH3D 37 , 7-Scenes 38 , and NRGBD 2 , we reduce the error by more than half compared with the base image-only models. When compared with other prior-aware methods that retrain a new model by feeding priors as input, our method also shows superior performance. These results prove the effectiveness of our TCO framework in incorporating priors for 3D tasks. In summary, our contributions are: • A test-time constrained optimization (TCO) framework that incorporates scene and camera priors without retraining or architectural changes. • A TCO loss design that includes a self-supervised objective to enforce multi-view prediction compatibility and prior penalty terms. • Extensive evaluations on 3D reconstruction benchmarks showing consistent large-margin improvements over base MVTs, surpassing prior-aware feed-forward methods while remaining plug-and-play. 2 Related Work Feed-forward 3D Reconstruction (No Priors). Reconstructing scenes from images has long relied on classical Structure-from-Motion (SfM) pipelines that estimate camera parameters and triangulate structure through multi-stage optimization 36 ; 31 ; 15 ; 40 . Recent feed-forward models bypass iterative optimization by directly regressing 3D structure from one or more views. DUSt3R predicts pairwise pointmaps anchored to a reference frame and recovers relative poses post hoc 45 , while MASt3R augments the representation with dense features for robust matching 25 . Multi-view variants reduce pairwise redundancy: MUSt3R and VGGT process many images jointly 3 ; 43 , and \pi^{3} further improves scalability with permutation-equivariant training that avoids fixing a reference frame 46 . Feed-forward 3D Reconstruction with Priors. Several feed-forward systems optionally leverage geometric priors at inference to improve 3D reconstruction or enable metric predictions. Such guidance includes intrinsics/extrinsics, sparse 3D points, or monocular depth priors. Pow3R demonstrates that conditioning on any subset of camera parameters and sparse points improves two-view pointmap prediction 19 . MapAnything supports flexible inputs, seamlessly incorporating priors to predict metric geometry 22 . Related lines of work show how intrinsics can enhance monocular depth estimation (e.g., UniDepth 33 ). Geometry, e.g., depth maps, has also been used as conditioning for camera pose estimation (e.g., Align3R 29 , Madpose 50 ). In this spirit, we design a model that benefits from priors when available, yet remains well-posed without them. Test-time Compute. Scaling test-time compute 47 ; 18 ; 13 has emerged as an effective way to trade latency for accuracy in LLMs 1 ; 27 . Inspired by theories of fast and slow thinking 21 , a strong pre-trained model acts as fast ”System‑1,” while iterative reasoning at inference serves as ”System‑2.” With techniques such as chain‑of‑thought and reflection, models revisit intermediate steps to refine predictions. Beyond language, recent work brings this reasoning into continuous latent spaces 14 . Another line of work views reasoning as an energy minimization process 6 ; 7 ; 12 . Energy‑based Transformers (EBTs) 12 , for example, score the compatibility between inputs and predictions and use gradient‑guided sampling at test time to improve consistency. Our approach shares this spirit: we define a test‑time objective that checks mutual consistency among outputs and refine all predictions to better satisfy it. In vision, related efforts 11 apply self‑supervised reconstruction losses at inference—e.g., masked‑autoencoder style objectives—to boost downstream performance. More specifically in 3D vision, Test3R 51 fine-tunes DUSt3R 45 at test time to generalize it from two-view