Regist3R: Incremental Registration with Stereo Foundation Model
Sidun Liu, Wenyu Li, Peng Qiao, Yong Dou · 2025
Read Regist3R: Incremental Registration with Stereo Foundation Model by Sidun Liu, Wenyu Li, Peng Qiao, Yong Dou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12356.txt.
Introduction
Sidun Liu, Wenyu Li, Peng Qiao, and Yong Dou wrote this computer vision paper, published in two thousand twenty-five, on rebuilding large 3D scenes from many photos. Recent foundation models do this from unposed images, but they slow down and drift as photo counts climb.
Their model, Regist3R, adds views one at a time in an incremental chain instead of aligning everything at once. That keeps costs down and stops errors from piling up across the whole scene. On public data sets it matches optimization-based methods on accuracy while running far faster. The team also built a hard test set from slanted aerial photos with hundreds of views over long spans. Regist3R handled scenes with thousands of views, a first for this class of model. The authors aim it at city modeling and aerial mapping.
Catalog record
| CREATOR | Sidun Liu, Wenyu Li, Peng Qiao, Yong Dou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-12356
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Regist3R: Incremental Registration with Stereo Foundation Model Sidun Liu, Wenyu Li, Peng Qiao, Yong Dou Abstract Multi-view 3D reconstruction has remained an essential yet challenging problem in the field of computer vision. While DUSt3R and its successors have achieved breakthroughs in 3D reconstruction from unposed images, these methods exhibit significant limitations when scaling to multi-view scenarios, including high computational cost and cumulative error induced by global alignment. To address these challenges, we propose Regist3R, a novel stereo foundation model tailored for efficient and scalable incremental reconstruction. Regist3R leverages an incremental reconstruction paradigm, enabling large-scale 3D reconstructions from unordered and many-view image collections. We evaluate Regist3R on public datasets for camera pose estimation and 3D reconstruction. Our experiments demonstrate that Regist3R achieves comparable performance with optimization-based methods while significantly improving computational efficiency, and outperforms existing multi-view reconstruction models. Furthermore, to assess its performance in real-world applications, we introduce a challenging oblique aerial dataset which has long spatial spans and hundreds of views. The results highlight the effectiveness of Regist3R. We also demonstrate the first attempt to reconstruct large-scale scenes encompassing over thousands of views through pointmap-based foundation models, showcasing its potential for practical applications in large-scale 3D reconstruction tasks, including urban modeling, aerial mapping, and beyond. 1 Introduction Multi-view 3D reconstruction is a fundamental task in computer vision, enabling the reconstruction of 3D models from multiple 2D images. The challenge lies in accurately estimating camera poses and reconstructing 3D structures from these images, particularly in complex environments with large spatial spans. As the cornerstone of multi-view 3D reconstruction, Structure from Motion (SfM) [ 1 ] reconstructs 3D structures and camera poses from unordered 2D images, but struggles with feature matching noise, scale drift, and scalability. Although global SfM [ 2 , 3 , 4 ] optimizes all camera parameters simultaneously, its dependency on reliable initial pairwise geometries limits its effectiveness in scenes with sparse features. Incremental SfM [ 5 , 6 , 7 ] mitigates these issues through sequential image registration and iterative refinement, proving to be more resilient in unstructured environments where global methods often fail. This adaptability explains its prevalent use in real-world applications with imperfect data conditions. Traditional SfM pipelines rely on sequential modules—feature extraction, matching, relative pose estimation, and triangulation [ 8 , 9 ] —a fragmented paradigm susceptible to cumulative error propagation. Recent learning-based methods [ 10 , 11 , 12 ] attempt to replace these handcrafted components with end-to-end formulations. While VGGSfM [ 10 ] introduces differentiable submodules and Detector-free SfM [ 13 ] employs learned feature matching, both retain the classical pipeline structure. More radical approaches like FlowMap [ 14 ] and Ace-Zero [ 11 ] abandon modular designs entirely, directly optimizing geometry via per-scene gradient descent with geometry regressor networks. However, such methods remain constrained to scenarios with high image overlap and stable illumination. In this landscape, DUSt3R [ 12 ] emerges as a breakthrough by predicting globally consistent pointmaps in a unified coordinate system from two views, demonstrating exceptional geometric coherence without incremental error accumulation. Building on this success, recent extensions [ 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 ] explore multi-view generalizations of DUSt3R’s framework, which demonstrate divergent optimization paradigms. Post-optimization approaches, exemplified by MASt3R-SfM [ 16 ] and Light3R-SfM [ 20 ] , iteratively reconstruct scenes from pairwise matchings: MASt3R-SfM leverages pointmaps and matches from MASt3R for bundle adjustment, while Light3R-SfM accelerates convergence via direct Procrustes alignment [ 23 ] . However, these methods fail to establish fully differentiable reconstruction pipelines driven by Transformer inference. Alternatively, another research direction [ 17 , 18 ] explores direct multi-view geometry prediction through scaled Transformer architectures. By integrating cross-attention mechanisms, these models enforce coordinate consistency across views during single-forward passes. Yet their scalability remains fundamentally constrained by the quadratic complexity growth of cross-attention layers with increasing view numbers. These limitations expose a critical gap in current learning-based SfM systems—achieving both optimization-free and scalable multi-view generalization. Multi-view 3D reconstruction models often suffer from limited generalization when extrapolated beyond their training distribution in viewpoint diversity or scene scale, whereas stereo models achieve stronger robustness by leveraging low-level geometric primitives that inherently constrain spatial reasoning across extended viewpoints and environments. In this paper, we try to answer a challenging question: Can we build an inference-only and scalable SfM system with just Stereo Foundation Models? To address this challenge, we present Regist3R, a stereo foundational model for incremental registration. Regist3R is a transformer-based [ 24 ] network that regresses the pointmap of an unregistered view in the world coordinate system by leveraging a given reference view along with its world coordinate pointmap. An intuitive formulation and illustration compared to DUSt3R is shown as Eq. 1 and Fig. 1 , \displaystyle I^{1}+I^{2} \displaystyle\xrightarrow{DUSt3R}X^{1,1}+X^{2,1} (1) \displaystyle I^{i}+I^{j}+X^{i,1} \displaystyle\xrightarrow{Regist3R}X^{j,1}. Analogous to DUSt3R, Regist3R uses a two-stream architecture with cross-attention to encode reference pointmap and target image respectively. Only one regression head is kept for the estimation of target pointmap. Although the model only accepts stereo input, it can be scaled to any size of viewpoints via autoregression. Drifting is one of the most critical problems for incremental reconstruction [ 25 ] , and time-consuming bundle adjustment (BA) is adopted by most of traditional pipelines [ 5 ] . In the absence of BA for inference-based models, we enhance Regist3R’s noise resistance to mitigate drift. Specifically, in addition to feeding the model ground truth pointmap as input, we use a chain training strategy to estimate target structure based on inaccurate pointmap. For efficient inference, we build a minimum spanning tree (MST) based on the view similarity, and perform pairwise inference only between the parents and children. This takes only N-1 times inference for a collection containing N images. As the drifting happens when the tree goes deep, we adopt a tree compression trick to reduce the length of reconstruction chain. We summarize the key contributions of this work as follows: • We propose Regist3R, a stereo foundation model for inference-only and scalable incremental registration. Based on our Regist3R, we further build a feed-forward incremental SfM pipeline for efficient reconstruction. • The experiments show that the performance of Regist3R is on par with or even better than optimization-based approaches and multi-view models. Furthermore, Regist3R demonstrates significant superiority over comparable methods when the scene has more views and larger spatial span. 2 Related Works 2.1 Structure-from-Motion Structure-from-Motion (SfM) [ 1 ] is a pivotal technique in computer vision for reconstructing 3D scenes from 2D images. Traditional SfM pipelines predominantly employ two approaches: global and incremental methods. Global SfM [ 26 , 2 , 4 , 27 , 28 ] addresses the entire dataset simultaneously, estimating camera positions and 3D structures in a unified optimization process. While this approach offers efficiency, it often faces challenges related to scalability and precision. The computational demands can be substantial, especially with large datasets, and maintaining accuracy across extensive reconstructions remains a concern. Recent advancements, such as the GLOMAP [ 29 ] and XM [ 30 ] , aim to enhance global SfM by balancing efficiency with improved accuracy, making it more viable for complex modeling tasks. Incremental SfM [ 31 , 7 , 5 , 32 ] , on